It's not too late for access control

[David M. Lloyd]

The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear. The
word "public" literally means "accessible to all" after all; that's why
the term was selected in the first place, and it is nearly certain that
this was the intent of the thing up to this date.

What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.

The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.

What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list and
in this email, but that even now, it's not too late to change the approach.

I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. Users would make public any type or member which is
*meant* to be public, i.e. accessible by all. Rather than (at best)
changing their expectations as to the behavior of "public" only to
immediately betray that expectation by forcing them through a backdoor
in order to meet practical needs, we ensure that their expectations
remain: public members are public, and things that are secret are not
public.

Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?

[Remi Forax]

----- Mail original -----

Envoyé: Lundi 11 Juillet 2016 16:21:46
Objet: It's not too late for access control
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear. The
word "public" literally means "accessible to all" after all; that's why
the term was selected in the first place, and it is nearly certain that
this was the intent of the thing up to this date.
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list and
in this email, but that even now, it's not too late to change the approach.
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. Users would make public any type or member which is
*meant* to be public, i.e. accessible by all. Rather than (at best)
changing their expectations as to the behavior of "public" only to
immediately betray that expectation by forcing them through a backdoor
in order to meet practical needs, we ensure that their expectations
remain: public members are public, and things that are secret are not
public.
Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?
--
- DML

Hi David,
package private means package private :)

One early design idea of jigsaw was to introduce a new modifier "module" with a visibility in between public and package private.
It's a bad idea !

First, there is already in the Java ecosystem a notion of non-exported package, packages startings with com.sun or packages containing internal, it was just a convention and not something enforced by the VM. What the JPMS spec does is just to normalize how to declare an exported package and mandate that the VM check this new rule.

Sure it means that public classes are not accessible/visible by everybody anymore, but a class like sun.misc.Unsafe was never really accessible by everybody despite being declared public.

Furthermore, declaring if something is exported or not at class level instead of at package level seems wrong to me, usually, several classes works together for a purpose and you want these classes to be exported or not, so it's not something that should be decided at class level.

So i see the JPMS spec conept of non-exported package as a standardization of an existing practice not something new that people will have trouble to understand and reason about.

regards,
Rémi

[David M. Lloyd]

Post by Remi Forax
----- Mail original -----

Envoyé: Lundi 11 Juillet 2016 16:21:46
Objet: It's not too late for access control
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear. The
word "public" literally means "accessible to all" after all; that's why
the term was selected in the first place, and it is nearly certain that
this was the intent of the thing up to this date.
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list and
in this email, but that even now, it's not too late to change the approach.
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. Users would make public any type or member which is
*meant* to be public, i.e. accessible by all. Rather than (at best)
changing their expectations as to the behavior of "public" only to
immediately betray that expectation by forcing them through a backdoor
in order to meet practical needs, we ensure that their expectations
remain: public members are public, and things that are secret are not
public.
Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?
--
- DML

Hi David,
package private means package private :)
One early design idea of jigsaw was to introduce a new modifier "module" with a visibility in between public and package private.
It's a bad idea !

I am not suggesting that; we definitely should not do that for reasons
discussed in the past.

Post by Remi Forax
First, there is already in the Java ecosystem a notion of non-exported package, packages startings with com.sun or packages containing internal, it was just a convention and not something enforced by the VM. What the JPMS spec does is just to normalize how to declare an exported package and mandate that the VM check this new rule.
Sure it means that public classes are not accessible/visible by everybody anymore, but a class like sun.misc.Unsafe was never really accessible by everybody despite being declared public.

The key advantage of using package-private is that you can drop the
access of shared internal classes out of public, and just make that
package accessible directly to its internal consumers. From the JDK
perspective: imagine getting rid of all those "shared secret" classes,
for example. In fact many JDK internal classes and APIs could just be
hidden, closing innumerable security holes.

For many, many existing projects, this approach promises exact
compatibility with the status quo in terms of accessibility, while also
opening an avenue forward to improve module security.

Post by Remi Forax
Furthermore, declaring if something is exported or not at class level instead of at package level seems wrong to me, usually, several classes works together for a purpose and you want these classes to be exported or not, so it's not something that should be decided at class level.

I did not propose deciding at a class level; I think that would be a bad
idea.

Post by Remi Forax
So i see the JPMS spec conept of non-exported package as a standardization of an existing practice not something new that people will have trouble to understand and reason about.

Except that there is no existing practice here. Today, if I can access
a Class or a ClassLoader, I can access all its public members, and this
is the crux of the whole thing. There is no valid compatibility-based
argument for changing this behavior. The idea is to take the status
quo, and add a new capability that can optionally be utilized for better
security, rather than to break the status quo, call it "compatible", and
completely break many, many users instantly.

[Paul Benedict]

I agree with David in what his email says. I find everything he has said in
his email to be reasonable. It's not useful for me to repeat what he has
stated. However, I will note that I, too, in my Jigsaw testing have found
myself having difficulty retraining my mind that "public" does not mean
public anymore. During the course of my experimentation, I, at times, found
myself forgetting to export my packages because I am still adjusting my
habits. I have said to myself, "It's public so why can't I see it? Oh,
that's right, the package is not exported." Point being that i was focusing
on the visibility modifier of the class and forgetting how the package
export plays into things.

It's a difficult habit for me to break (evidently) because I am still
making that mistake from time-to-time. :-) With that said, I believe David
is absolutely correct on the downside of having changed the meaning of
"public" since it is now (possibly) bounded by module boundaries. I can't
offer myself up as the prime example of tripping over the change, but I did
want to offer my experience as an example, nevertheless, to emphasis that
David is on the mark with his analysis.

All things being equal with Jigsaw features today, I'd rather have "public"
retain it's global visibility and extend "package private" to the module.
This makes great sense to me. Why? Because when I develop a library, I
really don't see any great benefit in hiding a type from another package.
Let me qualify that.... I really don't see any great benefit in hiding a
type from another package **that I own!** (double emphasis). I have
inspected all my uses of "package private" and it always comes down to one
thing: preventing static linkage to the type from packages **outside my
library**.

That's how I really use it. The wisdom in David's suggestion is that the
change from "package private" to "module private" aligns how the visibility
modifier is actually used to protect types. I think it would be a shame to
pass on David's suggestion since it is so elegant, so please seriously
consider it.

Cheers,
Paul

Post by David M. Lloyd

Post by Remi Forax
----- Mail original -----

Envoyé: Lundi 11 Juillet 2016 16:21:46
Objet: It's not too late for access control
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear. The
word "public" literally means "accessible to all" after all; that's why
the term was selected in the first place, and it is nearly certain that
this was the intent of the thing up to this date.
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list and
in this email, but that even now, it's not too late to change the approach.
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. Users would make public any type or member which is
*meant* to be public, i.e. accessible by all. Rather than (at best)
changing their expectations as to the behavior of "public" only to
immediately betray that expectation by forcing them through a backdoor
in order to meet practical needs, we ensure that their expectations
remain: public members are public, and things that are secret are not
public.
Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?
--
- DML

Hi David,
package private means package private :)
One early design idea of jigsaw was to introduce a new modifier "module"
with a visibility in between public and package private.
It's a bad idea !

I am not suggesting that; we definitely should not do that for reasons
discussed in the past.
First, there is already in the Java ecosystem a notion of non-exported

Post by Remi Forax
package, packages startings with com.sun or packages containing internal,
it was just a convention and not something enforced by the VM. What the
JPMS spec does is just to normalize how to declare an exported package and
mandate that the VM check this new rule.
Sure it means that public classes are not accessible/visible by everybody
anymore, but a class like sun.misc.Unsafe was never really accessible by
everybody despite being declared public.

The key advantage of using package-private is that you can drop the access
of shared internal classes out of public, and just make that package
imagine getting rid of all those "shared secret" classes, for example. In
fact many JDK internal classes and APIs could just be hidden, closing
innumerable security holes.
For many, many existing projects, this approach promises exact
compatibility with the status quo in terms of accessibility, while also
opening an avenue forward to improve module security.
Furthermore, declaring if something is exported or not at class level

Post by Remi Forax
instead of at package level seems wrong to me, usually, several classes
works together for a purpose and you want these classes to be exported or
not, so it's not something that should be decided at class level.

I did not propose deciding at a class level; I think that would be a bad
idea.
So i see the JPMS spec conept of non-exported package as a standardization

Post by Remi Forax
of an existing practice not something new that people will have trouble to
understand and reason about.

Except that there is no existing practice here. Today, if I can access a
Class or a ClassLoader, I can access all its public members, and this is
the crux of the whole thing. There is no valid compatibility-based
argument for changing this behavior. The idea is to take the status quo,
and add a new capability that can optionally be utilized for better
security, rather than to break the status quo, call it "compatible", and
completely break many, many users instantly.
--
- DML

[John Rose]

Post by Paul Benedict
All things being equal with Jigsaw features today, I'd rather have "public"
retain it's global visibility and extend "package private" to the module.

Again, $0.02, with apologies in advance for opining on a subject I'm not
an expert in.

From a pure language POV I find this appealing, but in our world there
are myriads of existing public classes already coded in libraries which need
large-scale encapsulation. Surely we need to be able to tell some libraries,
"Hey, not so public, there; we're restricting your visibility." If we can't do that
we don't have modules at all. If Java had modules from the start, maybe
we'd say "public is globally public, so any class can punch through all layers of
encapsulation by using the super-strong keyword 'public', and the default
is 'module'." (Advance apology if this is a straw-man.)

Post by Paul Benedict
This makes great sense to me. Why? Because when I develop a library, I
really don't see any great benefit in hiding a type from another package.
Let me qualify that.... I really don't see any great benefit in hiding a
type from another package **that I own!** (double emphasis). I have
inspected all my uses of "package private" and it always comes down to one
thing: preventing static linkage to the type from packages **outside my
library**.

To me this looks like a simple power struggle between the class author
expecting a platform for full publicity, and the system assembler expecting
to be able to control inter-library visibility. Both can't have the final word.
Shouldn't the assembler have it? And if the assembler is given absolute
control over some unit of access control, it seems likely to me that this
unit has to look a lot like a Jigsaw module.

Note that system assemblers cannot be expected to edit individual classes.
That would be another scale error. So in the end, the author of a class and
of a package has to provide a provisional, local spec. of what's visible, while
the assembler (who works with unmodifiable libraries) owns the global spec.

(HTH. I'm probably missing a couple of Third Way options; please point them
out. I'll go back to lurk mode now.)

— John

[John Rose]

Post by John Rose
lurk mode

P.S. I forgot one more point: As JVM/JIT guy I find the prospect of sealing
packages very appealing. Maybe it's a false promise, but it would be a
home run if we could reliably seal sub-assemblies for tree-shaking AOT
compilation. Doing this requires pretty strong assurances that the AOT
engine can enumerate all uses of a "public" class. Doesn't this use case
interfere with the proposed principle that "public always means global"?
Again, it's a tug-of-war between the class author and an assembly-time
operation.

[David M. Lloyd]

Post by John Rose

Post by John Rose
lurk mode

P.S. I forgot one more point: As JVM/JIT guy I find the prospect of sealing
packages very appealing. Maybe it's a false promise, but it would be a
home run if we could reliably seal sub-assemblies for tree-shaking AOT
compilation. Doing this requires pretty strong assurances that the AOT
engine can enumerate all uses of a "public" class. Doesn't this use case
interfere with the proposed principle that "public always means global"?
Again, it's a tug-of-war between the class author and an assembly-time
operation.

I'm definitely sympathetic to the AOT argument. However there is
absolutely no difference in the guarantees that can be derived from the
Jigsaw proposal versus my proposal. Consider these cases:

1) A package is intended to be completely internal and is not exported;
in Jigsaw, the types in this package would be public, and in my proposal
they would not be.
2) A package is intended to be shared to certain targets; same as above.
3) A package is meant to be accessible to all; in both cases the types
would be public; in the Jigsaw case it would have to be additionally
exported to all.

In all three of these cases, AOT compilation would have to assume that
this linkage-oriented information would not change between compile and
assembly, and in all three of these cases, this assumption can be
trivially defeated by an even mildly determined user. So in the end, it
seems to me that AOT is something that would have to be entered into
with open eyes by the users, and all phases would have to be conscious
of additional restrictions required to execute a stable run-time image.

[Andrew Dinn]

Post by John Rose

Post by John Rose
lurk mode

P.S. I forgot one more point: As JVM/JIT guy I find the prospect of sealing
packages very appealing. Maybe it's a false promise, but it would be a
home run if we could reliably seal sub-assemblies for tree-shaking AOT
compilation. Doing this requires pretty strong assurances that the AOT
engine can enumerate all uses of a "public" class. Doesn't this use case
interfere with the proposed principle that "public always means global"?
Again, it's a tug-of-war between the class author and an assembly-time
operation.

I've been waiting for someone from the JVM side to make a proposal about
using modules to optimize compilation -- AOT being the obvious candidate
that stands to gain the most -- and I'm very glad it is John who finally
proposed it :-). Those assurances really do need to be very strong. I
wonder if they can be made strong enough, especially granted the current
proposal to provide class Instrumentation with method redefineModule
that, inter alia, accepts a set of exports to add to those already in
place for some module being redefined.

My agent wants to use this extended API to provide fairly tightly
constrained incremental exposure of non-exported packages. Essentially,
any currently unexported package in any module may potentially be made
visible to a single, privileged module provided as part of my agent.
This is to allow the agent to obtain specific reflective access to
non-public members of the classes so exported. Other agents may not be
quite so careful to limit the degree to which they expose module
packages to other modules.

I'm not sure how AOT compiled code is supposed to protect itself against
the potential consequences of my agent tweaking accessibility in this
way. Would it be not require -- at the least -- wholesale deoptimization
of compiled methods in the module to which the package is exported? How
would one arrange for JITted code to trap out and invalidate any
speculative optimizations that might be invalidated by this incremental
export?

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

[John Rose]

Post by Andrew Dinn
I'm not sure how AOT compiled code is supposed to protect itself against
the potential consequences of my agent tweaking accessibility in this
way. Would it be not require -- at the least -- wholesale deoptimization
of compiled methods in the module to which the package is exported? How
would one arrange for JITted code to trap out and invalidate any
speculative optimizations that might be invalidated by this incremental
export?

Either wholesale de-opt, or a fast-fail error when the injection is attempted.
(It's another example of the tug-of-war between parties with different views on a class.)

Either outcome is bad enough that we need speed bumps to prevent people
from just cruising heedlessly into the failure zone. By "speed bump" I mean
a need to assert extra-specific intention, and/or obtain an access right,
in order to increase the visibility of (or inject code into) a module that
has been consolidated into an AOT sub-assembly. (N.B. I'm using these
terms in a general sense, not referring to any particular tech.)

BTW, if you contrive to inject a MethodHandles.Lookup donor into
a class, the donee can get everything (99.9%) the class can do by
materializing and calling appropriate method handles. With a little
care (bind the MHs to static finals) you get identical performance.
This is easier than spinning adapters and injecting those one by one.
Is that how you are doing your injection?

— John

[Andrew Dinn]

Post by John Rose

Post by Andrew Dinn
I'm not sure how AOT compiled code is supposed to protect itself against
the potential consequences of my agent tweaking accessibility in this
way. Would it be not require -- at the least -- wholesale deoptimization
of compiled methods in the module to which the package is exported? How
would one arrange for JITted code to trap out and invalidate any
speculative optimizations that might be invalidated by this incremental
export?

Either wholesale de-opt, or a fast-fail error when the injection is attempted.
(It's another example of the tug-of-war between parties with different views on a class.)

Ok, so I suspect this may be some while coming to fruition :-)

Post by John Rose
Either outcome is bad enough that we need speed bumps to prevent people
from just cruising heedlessly into the failure zone. By "speed bump" I mean
a need to assert extra-specific intention, and/or obtain an access right,
in order to increase the visibility of (or inject code into) a module that
has been consolidated into an AOT sub-assembly. (N.B. I'm using these
terms in a general sense, not referring to any particular tech.)

I think this is probably necessary but it's certainly not in place right
now. Currently the 'access right' is baked into class Instrumentation,
an instance of which is simply handed out to agents - to do with what
they will. So if you want some sort of switch to drive AOT I think it
has to be as crude as 'an agent has been loaded'.

Post by John Rose
BTW, if you contrive to inject a MethodHandles.Lookup donor into
a class, the donee can get everything (99.9%) the class can do by
materializing and calling appropriate method handles. With a little
care (bind the MHs to static finals) you get identical performance.
This is easier than spinning adapters and injecting those one by one.
Is that how you are doing your injection?

Well, this is still being implemented so I might be able to profit form
using a MethodHandle. As it stands I'm relying on reflection so I'm
really not sure whether/how my agent might imperil any speculative AOT
compilation.

I'm not currently injecting member accesses into existing bytecode. My
agent only ever injects a callout to agent code located in the unnamed
module. That agent code may determine that it needs to use reflection to
access some specific member (field or method) that is not normally
visible to the unnamed module and in those circumstances it uses
addExports to provide the option of reflective accessibility.

Of course, it doesn't simply export the relevant package to the unnamed
module -- that would be highly insecure. Instead it exports the package
to a module provided along with the agent and employs a private channel
to acquire an enabled reflective handle.

It's hard to know for sure whether this might affect speculative AOT
compilation and, if so, how to respond. The code into which I inject the
call out may lie in any class. The members the agent accesses may lie in
any class. The reflective access is legitimised under the aegis of the
module system from a module granted the requisite permission. However,
the handle is exercised by code located in the unnamed module.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

[David M. Lloyd]

Not commenting on the entire piece here, just one bit...

Post by John Rose

Post by Paul Benedict
This makes great sense to me. Why? Because when I develop a library, I
really don't see any great benefit in hiding a type from another package.
Let me qualify that.... I really don't see any great benefit in hiding a
type from another package **that I own!** (double emphasis). I have
inspected all my uses of "package private" and it always comes down to one
thing: preventing static linkage to the type from packages **outside my
library**.

To me this looks like a simple power struggle between the class author
expecting a platform for full publicity, and the system assembler expecting
to be able to control inter-library visibility. Both can't have the final word.
Shouldn't the assembler have it? And if the assembler is given absolute
control over some unit of access control, it seems likely to me that this
unit has to look a lot like a Jigsaw module.

In the current model though, the assembler doesn't have the control at
all. The module author is the one who decides who can access what.
Before Java 9, it was just by making some things public and some things
non-public; now it's a question of ensuring that certain packages (maybe
all) are accessible to all using explicit mechanisms, but the end result
is basically the same.

If the assembler wants to add nuance to accessibility, they have to
modify those lovely binary descriptors at assembly time. I don't think
Jigsaw's restriction of "public" adds anything over what I propose in
this regard (note that what I propose does not involve editing classes
for any reason though); my proposal doesn't really change this one way
or another, except that it retains the current well-understood
difference between visibility and accessibility, with the former acting
not as a security mechanism but a linking mechanism, just like today.

Post by John Rose
Note that system assemblers cannot be expected to edit individual classes.
That would be another scale error. So in the end, the author of a class and
of a package has to provide a provisional, local spec. of what's visible, while
the assembler (who works with unmodifiable libraries) owns the global spec.

Yeah but think about it: how different is the need to edit classes from
the need to edit binary module descriptors, encoded *as* class files?
Really, the assembler should have to do neither thing, because that's an
undue burden.

[Sanne Grinovero]

Post by John Rose

Post by Paul Benedict
All things being equal with Jigsaw features today, I'd rather have "public"
retain it's global visibility and extend "package private" to the module.

Again, $0.02, with apologies in advance for opining on a subject I'm not
an expert in.
From a pure language POV I find this appealing, but in our world there
are myriads of existing public classes already coded in libraries which need
large-scale encapsulation. Surely we need to be able to tell some libraries,
"Hey, not so public, there; we're restricting your visibility." If we can't do that
we don't have modules at all. If Java had modules from the start, maybe
we'd say "public is globally public, so any class can punch through all layers of
encapsulation by using the super-strong keyword 'public', and the default
is 'module'." (Advance apology if this is a straw-man.)

As a maintainer and contributor to several popular Java open source
libraries my experience is that in practice very few existing
libraries will "just work" in Java 9 out of the box: people will have
to update their code. This is based on my experience, as some of these
projects have dozens of dependencies and it's taking a long time to
identify each problematic point, discuss patches, getting other
communities to release timely, an often there is need for "recursive
releases" and various iterations for each problem as they get
identified, for each dependency.

The priority for most OSS communities and developers at this stage is
to be able to compile and run basic tests; some started early and are
in a good shape today (e.g. Apache Lucene, Hibernate), but several
others have not been sensible to this matter: so improving modularity
and adjusting visibility is "only" a secondary need for many (not
defending this position, just my impression), but I'm confident that
we'll eventually get all library maintainers to actively update their
API as well.

No doubt there will be a next phase in which existing APIs will be
reviewed to take advantage of Jigsaw, for example I look forward to
publish our Hibernate libraries as Jigsaw modules, but this is work
that needs to be done by explicitly by updating the libraries and
review the APIs with a focus on visibility.

I'm generally grateful that the OpenJDK developers are working hard to
not break existing libraries, but in this case I don't think it's
worth it as we'd merely have outdated libraries half-working at best.

I'd rather see the modules system be truly great, so that it will be
worth our time to fix all the libraries in the coming years; in the
mean time, I suspect that exposing too much as "public" shouldn't be a
problem as it worked well enough so far.

Post by John Rose

Post by Paul Benedict
This makes great sense to me. Why? Because when I develop a library, I
really don't see any great benefit in hiding a type from another package.
Let me qualify that.... I really don't see any great benefit in hiding a
type from another package **that I own!** (double emphasis). I have
inspected all my uses of "package private" and it always comes down to one
thing: preventing static linkage to the type from packages **outside my
library**.

To me this looks like a simple power struggle between the class author
expecting a platform for full publicity, and the system assembler expecting
to be able to control inter-library visibility. Both can't have the final word.
Shouldn't the assembler have it? And if the assembler is given absolute
control over some unit of access control, it seems likely to me that this
unit has to look a lot like a Jigsaw module.

Great points. In fact in my team we also have experience "packaging"
lots of these OSS Java libraries into JBoss Modules, and have always
appreciated that - as an assembler - with JBoss Modules I can define
the dependencies via external metadata, without having to recompile or
have to reassemble the jar.
As you suggest, it is indeed useful to be able to override the intent
of the library authors, especially as different libraries are
developed by independent teams / communities / companies.

This was also raised by David in the past; I'm a bit concerned that
Jigsaw's more strict ways have not been tested enough "on the field"
on the aspect of strict metadata being packaged in the module itself.
In particular, I appreciated the flexibility of JBoss Modules most
after having used it for a long time: some libraries started to evolve
over time and you need to start thinking about backwards/forward
compatibility among many libraries which are released at different
points in time. Often even this "timeline" of releases isn't a
linearizable sequence, as different libraries will release
independently on their own timeline and not necessarily upgrading all
of their dependencies consistently with other projects in the world.

Please don't underestimate the importance of this flexibility for the
"assembler": its importance might not be very clear now when drawing
dependency graphs in a "clean room" scenario, but it becomes essential
when you consider that the rich ecosystem of libraries is actually in
constant evolution.

Regards,
Sanne

Post by John Rose
Note that system assemblers cannot be expected to edit individual classes.
That would be another scale error. So in the end, the author of a class and
of a package has to provide a provisional, local spec. of what's visible, while
the assembler (who works with unmodifiable libraries) owns the global spec.
(HTH. I'm probably missing a couple of Third Way options; please point them
out. I'll go back to lurk mode now.)
— John

[Alan Bateman]

Post by Sanne Grinovero
As a maintainer and contributor to several popular Java open source
libraries my experience is that in practice very few existing
libraries will "just work" in Java 9 out of the box: people will have
to update their code. This is based on my experience, as some of these
projects have dozens of dependencies and it's taking a long time to
identify each problematic point, discuss patches, getting other
communities to release timely, an often there is need for "recursive
releases" and various iterations for each problem as they get
identified, for each dependency.

Going off-topic slightly but when you do run into issues and if they
aren't already listed as compatibility issues then please bring them up.
So far then I think the vast majority of issues that we have heard about
relate to the changes in JEP 220 (tools.jar going away etc), JEP 223 and
the new version-string scheme, the class file version bump in jdk-9+119,
and then all the issues that we have listed in JEP 261. Any help getting
bugs submitted to projects would be appreciated too.

The issue that `public` no longer implies accessible is listed in JEP
261 (first item in the Risks and Assumption) but to be honest, has
barely come up to date. That probably isn't too surprising as it's still
early days for modules and many projects aren't trying out JDK 9 yet.
Anyone trying out modules where a "module unaware" framework gets a
reference to a public type in a non-exported package might run into it
of course but I'm not aware of any reports yet.

-Alan.

[David M. Lloyd]

Post by Alan Bateman

Post by Sanne Grinovero
As a maintainer and contributor to several popular Java open source
libraries my experience is that in practice very few existing
libraries will "just work" in Java 9 out of the box: people will have
to update their code. This is based on my experience, as some of these
projects have dozens of dependencies and it's taking a long time to
identify each problematic point, discuss patches, getting other
communities to release timely, an often there is need for "recursive
releases" and various iterations for each problem as they get
identified, for each dependency.

Going off-topic slightly but when you do run into issues and if they
aren't already listed as compatibility issues then please bring them up.
So far then I think the vast majority of issues that we have heard about
relate to the changes in JEP 220 (tools.jar going away etc), JEP 223 and
the new version-string scheme, the class file version bump in jdk-9+119,
and then all the issues that we have listed in JEP 261. Any help getting
bugs submitted to projects would be appreciated too.
The issue that `public` no longer implies accessible is listed in JEP
261 (first item in the Risks and Assumption) but to be honest, has
barely come up to date. That probably isn't too surprising as it's still
early days for modules and many projects aren't trying out JDK 9 yet.
Anyone trying out modules where a "module unaware" framework gets a
reference to a public type in a non-exported package might run into it
of course but I'm not aware of any reports yet.

Isn't that what this entire thread is about? And also, what the whole
#ReflectiveAccessToNonExportedTypes issue is about? If not, consider
this the official report that dozens if not hundreds of such frameworks
are broken under Java 9. I have been bringing it up for many months,
and this is a result of testing, not of guesswork.

[Jochen Theodorou]

[...]

Post by David M. Lloyd

Post by Alan Bateman
The issue that `public` no longer implies accessible is listed in JEP
261 (first item in the Risks and Assumption) but to be honest, has
barely come up to date. That probably isn't too surprising as it's still
early days for modules and many projects aren't trying out JDK 9 yet.
Anyone trying out modules where a "module unaware" framework gets a
reference to a public type in a non-exported package might run into it
of course but I'm not aware of any reports yet.

Isn't that what this entire thread is about? And also, what the whole
#ReflectiveAccessToNonExportedTypes issue is about? If not, consider
this the official report that dozens if not hundreds of such frameworks
are broken under Java 9. I have been bringing it up for many months,
and this is a result of testing, not of guesswork.

There is for example setAccessible throwing
java.lang.reflect.InaccessibleObjectException. What happens in Groovy
then? We go the hierarchy up till we find a class for which we can do
that, or stop at Object. That of course is also because of referencing a
public type in a non-exported package. See this list 10 months ago.

And even though we tried to start testing JDK9 early, we still do not
have a Groovy module today. And the lack of information about how to
handle the dynamic aspects of modules does not make this more easy. One
time it is about "we have to still write this down", other times it is
about the discussion just dying down. Every time there might have been
good reasons for this, but the effect is clear on me: I am missing
information. And the javadoc plus the "The State of the Module System"
is not clear enough for me. And my summary is: without blaming anyone
really, it is still just frustrating.

bye Jochen

[Alan Bateman]

Post by David M. Lloyd
Isn't that what this entire thread is about? And also, what the whole
#ReflectiveAccessToNonExportedTypes issue is about?

I think that's a good question, esp as some frameworks allow for
annotations or configuration on non-public types or members. The
`exports dynamic` proposal on the #ReflectiveAccessToNonExportedTypes
thread exports the package at runtime and so allows the slimy
setAccessible(true) to break in. In the very long term then
setAccessible needs to go away of course but I do think non-public types
in non-exported packages is part of the discussion too.

Post by David M. Lloyd
If not, consider this the official report that dozens if not hundreds
of such frameworks are broken under Java 9. I have been bringing it
up for many months, and this is a result of testing, not of guesswork.

It would be better to bring along some specific examples so that they
can be studied. I would expect these frameworks, and consumers of, to
work in JDK 9 as they did with JDK 8 assuming they aren't dependent on
JDK-internal APIs (or other compatibility issues that we've listed in
the JEPs). It's when consumers are migrated to explicit modules that it
gets tricky as the existing frameworks don't yet know about the updated
accessibility landscape yet.

-Alan

[Andrew Dinn]

Post by Alan Bateman

Post by David M. Lloyd
Isn't that what this entire thread is about? And also, what the whole
#ReflectiveAccessToNonExportedTypes issue is about?

I think that's a good question, esp as some frameworks allow for
annotations or configuration on non-public types or members. The
`exports dynamic` proposal on the #ReflectiveAccessToNonExportedTypes
thread exports the package at runtime and so allows the slimy
setAccessible(true) to break in. In the very long term then
setAccessible needs to go away of course but I do think non-public types
in non-exported packages is part of the discussion too.

Alan, you previously described setAccessible as a 'sledge hammer that
breaks down the door' and now resort to calling it 'slimy'. Clearly, you
don't think much of this API. However, I'll put aside the
appropriateness of such descriptions in order to challenge the assertion
that it 'needs to go away'.

Does it? Why and on whose say so? This is a pretty significant change to
the nature of the runtime that you are talking about. It removes a
capability that is critical to certain important types of program
operation and which has been used by many tools and libraries in
previous JDK releases. If removing this API is an agreed long term goal
of Jigsaw then where is the discussion to justify that goal? Have you
established that Java users actually want a JDK which does not provide
this means of privileged access? I don't see any such agreement in the
EG archive, for example.

My agent has very good reason to want to do precisely what you wish to
eject from the runtime. I have many users who rely on this ability to be
able to test, monitor and debug their applications. Without this API a
great deal of testing becomes very difficult or even impossible,
Likewise, without it the job of many of Red Hat's support staff and
consultants will be severely compromised.

If this aspect of how Java currently works is to be removed then I
believe it needs to be done so on the basis of a publicly established
consensus, preferably under the aegis of the JSR EG. It certainly does
not seem right to me that such a goal should be adopted by an
implementation team without such consultation.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

[dalibor topic]

Post by Andrew Dinn

Post by Alan Bateman

Post by David M. Lloyd
Isn't that what this entire thread is about? And also, what the whole
#ReflectiveAccessToNonExportedTypes issue is about?

I think that's a good question, esp as some frameworks allow for
annotations or configuration on non-public types or members. The
`exports dynamic` proposal on the #ReflectiveAccessToNonExportedTypes
thread exports the package at runtime and so allows the slimy
setAccessible(true) to break in. In the very long term then
setAccessible needs to go away of course but I do think non-public types
in non-exported packages is part of the discussion too.

Alan, you previously described setAccessible as a 'sledge hammer that
breaks down the door' and now resort to calling it 'slimy'. Clearly, you
don't think much of this API. However, I'll put aside the
appropriateness of such descriptions in order to challenge the assertion
that it 'needs to go away'.

I believe that this was discussed before at
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2015-September/000122.html
.

cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Andrew Dinn]

Post by dalibor topic
I believe that this was discussed before at
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2015-September/000122.html

Thank you for the link, Dalibor. It does indeed appear that the
possibility of removal of this API has indeed been mentioned in a
discussion on the EG list. However, I'm not convinced that constitutes
sufficient discussion to merit such a change.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

[dalibor topic]

Post by Andrew Dinn

Post by dalibor topic
I believe that this was discussed before at
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2015-September/000122.html

Thank you for the link, Dalibor. It does indeed appear that the
possibility of removal of this API has indeed been mentioned in a
discussion on the EG list. However, I'm not convinced that constitutes
sufficient discussion to merit such a change.

Given that the JSR (or its implementation in this Project) does not
propose making such a change, establishing sufficient discussion to
merit such a change would be a task for someone at some other point in
time.

So, to avoid going down a highly speculative alley, I'd suggest tabling
the arguments for or against such a change until it is actually being
proposed by someone for discussion, rather than just being speculated
about for a very long term future.

I have to run to catch my whale bus now.[0]

cheers,
dalibor topic

[0]
http://publicdomainreview.org/collections/france-in-the-year-2000-1899-1910/
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Mario Torre]

Post by dalibor topic
I have to run to catch my whale bus now.[0]

It's a set of incremental updates that made France being slightly
different than this vision, it's a good thing to discuss some of those
changes that will motivate wether or not whales will be able to
instrument JVM byte code or not 100 years from now.

Cheers,
Mario
--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

[Alan Bateman]

Post by Andrew Dinn
If this aspect of how Java currently works is to be removed then I
believe it needs to be done so on the basis of a publicly established
consensus, preferably under the aegis of the JSR EG. It certainly does
not seem right to me that such a goal should be adopted by an
implementation team without such consultation.

It goes much wider than Jigsaw: such a basic language change needs all
stakeholders to be consulted. Most of them surely won't be reading
the Jigsaw list, yet will still be affected.

This project (and JSR) is not proposing to remove setAccessible as that
would break many things. The comment that Andrew Dinn picked up started
with "In the very long term ..." and is a throw away comment on where
the platform needs to go long term. In general then we need to find
better solutions for things that setAccessible is used for today. That
could take years and many major releases. It's a bit like the Unsafe
issue in that regard.

Confusion aside, we have taken a first baby step towards degrading
setAccessible so that it can't be used to break into non-exported
packages. There was discussion about this on the EG mailing list last
year, much discussion on it here too.

Agents (Andrew Dinn's main interest I think) have the power to change
bytecode and extend modules at runtime to break encapsulation. I don't
think there is anything to be overly concerned here.

-Alan

[Andrew Dinn]

Post by Alan Bateman
This project (and JSR) is not proposing to remove setAccessible as that
would break many things. The comment that Andrew Dinn picked up started
with "In the very long term ..." and is a throw away comment on where
the platform needs to go long term. In general then we need to find
better solutions for things that setAccessible is used for today. That
could take years and many major releases. It's a bit like the Unsafe
issue in that regard.

I am glad to hear that

"In the very long term then setAccessible needs to go away of course"

is an opinion not a decision.

Post by Alan Bateman
Confusion aside, we have taken a first baby step towards degrading
setAccessible so that it can't be used to break into non-exported
packages. There was discussion about this on the EG mailing list last
year, much discussion on it here too.

There has indeed also been discussion recently about retaining the
ability to break into non-exported packages. I think it needs to continue.

Post by Alan Bateman
Agents (Andrew Dinn's main interest I think) have the power to change
bytecode and extend modules at runtime to break encapsulation. I don't
think there is anything to be overly concerned here.

I am doubtful that my interests are the only ones at stake here nor
indeed that the problem relates solely to agents. I suspect solving the
related problems faced by middleware will also have some dependence on
the ability to use setEnabled. I hope the current discussion will
clarify whether the need for this API extends to those cases. However,
let us just consider agents for the present.

Changing bytecode is a powerful capability but it is limited by the
constraints imposed by the verifier. In particular, that means a
dynamically loaded agent is only capable of providing access to
immediate private members from bytecode directly injected into the
owning class.

That provides highly limited access. For example, accessing through an
object network with disparate link types is not possible in this way
(i.e. accessing this.bar.baz where baz and bar are both private fields
of different types to this). Of course, reflection combined with
setEnabled does indeed permit such accesses.

Redefining modules via the Instrumentation class looks like it may well
provide a means to access module non-public data/behaviours in a
restricted fashion from an agent (I am close to having a working version
of Byteman using this new capability). However, it only does so thanks
to the continued presence of the setEnabled API. So, if this API is to
be removed -- whether in the sort or long term -- then this will be
problematic for my agent and, no doubt, for others. I believe that at
the very least requires wider consultation.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

[Alan Bateman]

OK. But "in the very long term" such a basic language change needs
all stakeholders to be consulted.

I agree (although it's not really a language change in that it's API way
to suppress access checks specified by the language).

Yes, indeed, and that is potentially a significant problem. My
comment stands: there is a serious possibility that his will make it
impossible to use (non-exported) Jigsaw modules for some kinds of
programming. This is exactly the kind of decision that needs all
stakeholders to be consulted.
It ought to be possible to have some kind of conditional export which
only allows such access by (e.g.) suitably privileged frameworks or
tools. But I have no desire to get involved in such design issues: I
am only going to say that this is an issue which requires wider
consultation.

This is #ReflectiveAccessToNonExportedTypes on the JSR 376 issues list.
The problem is reasonably well understood and there are several
proposals and approaches being discussed and considered.

-Alan

[Alan Bateman]

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me."

It's related. If a package is exported to you then you can break into
its non-public types and members with setAccessible. So I think it will
fall out of the wash once there is agreement on
#ReflectiveAccessToNonExportedTypes.

-Alan

[Andrew Haley]

Post by Alan Bateman

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me."

It's related. If a package is exported to you then you can break into
its non-public types and members with setAccessible. So I think it will
fall out of the wash once there is agreement on
#ReflectiveAccessToNonExportedTypes.

At Red Hat we have many Java programmers. We also have many customers
who are Java programmers. I am trying to persuade people to try out
JDK 9 in order to give us the feedback we need to ratify the JDK 9
specification. But you know where this is going: the changes to
Jigsaw which we need are TBD, but with an assurance of "I think we'll
be able to work something out." This is a tough sell for me.

I am also rather concerned that too much is going into Jigsaw. Along
with much-needed modularization of the JDK itself we're getting access
controls such as the one we're discussing. I understand the argument
for restricting setAccessible() in a module context (and the computer
scientist in me says "Yay!") but it's not a necessary part of
modularization and it should be discussed on its own merits. Everyone
is going to be affected by Jigsaw, but you can't expect everyone to
read the minutiae of the discussion.

Andrew.

[m***@oracle.com]

Post by Andrew Haley
At Red Hat we have many Java programmers. We also have many customers
who are Java programmers. I am trying to persuade people to try out
JDK 9 in order to give us the feedback we need to ratify the JDK 9
specification. But you know where this is going: the changes to
Jigsaw which we need are TBD, but with an assurance of "I think we'll
be able to work something out." This is a tough sell for me.

Is this any different than the usual chicken-and-egg problem of
live software development? Much of Jigsaw is still in flux as we
work through issues in both the specification and the implementation.
If you have specific suggestions about how we could make it easier to
persuade people to try JDK 9 then I'd like to hear them.

Also, nobody is asking you, or anyone else, to take anything on faith.
We are -- and long have been -- asking for feedback, both practical
feedback based on running existing code against EA builds and design
feedback from developers of all skill levels.

Post by Andrew Haley
I am also rather concerned that too much is going into Jigsaw. Along
with much-needed modularization of the JDK itself we're getting access
controls such as the one we're discussing. I understand the argument
for restricting setAccessible() in a module context (and the computer
scientist in me says "Yay!") but it's not a necessary part of
modularization and it should be discussed on its own merits. Everyone
is going to be affected by Jigsaw, but you can't expect everyone to
read the minutiae of the discussion.

No, of course not, and that's exactly why we've been writing and talking
publicly, for years now, about this work.

- Mark

[Andrew Haley]

Post by Andrew Haley
At Red Hat we have many Java programmers. We also have many
customers who are Java programmers. I am trying to persuade people
to try out JDK 9 in order to give us the feedback we need to ratify
the JDK 9 specification. But you know where this is going: the
changes to Jigsaw which we need are TBD, but with an assurance of
"I think we'll be able to work something out." This is a tough
sell for me.

Is this any different than the usual chicken-and-egg problem of live
software development? Much of Jigsaw is still in flux as we work
through issues in both the specification and the implementation. If
you have specific suggestions about how we could make it easier to
persuade people to try JDK 9 then I'd like to hear them.

I take your point. Many suggestions are likely to be of the form "fix
this issue on the known issues list."

Also, nobody is asking you, or anyone else, to take anything on
faith. We are -- and long have been -- asking for feedback, both
practical feedback based on running existing code against EA builds
and design feedback from developers of all skill levels.

Sure. I suppose the only way to make this work is to break things,
wait for feedback about what people need, and fix them. The problem
that we have is that we are fairly late in the JDK 9 release cycle,
and still not sure if the specification of Jigsaw will work for us.
Or, for that matter, many large-scale Java programs.

Post by Andrew Haley
I am also rather concerned that too much is going into Jigsaw. Along
with much-needed modularization of the JDK itself we're getting access
controls such as the one we're discussing. I understand the argument
for restricting setAccessible() in a module context (and the computer
scientist in me says "Yay!") but it's not a necessary part of
modularization and it should be discussed on its own merits. Everyone
is going to be affected by Jigsaw, but you can't expect everyone to
read the minutiae of the discussion.

No, of course not, and that's exactly why we've been writing and
talking publicly, for years now, about this work.

Indeed, but the kinds of changes that we're talking about --
particularly this one -- are not the things which people notice until
they bang their heads on them. With JDK 9 it's quite likely that
people won't be deploying their own modules for some time, and
initially will want to run their JDK 8 programs with as little change
as possible. It's quite likely that they wouldn't even know that the
setAccessible issue even existed until much later in the cycle,
*after* the specification was frozen. And at that point they would
just decide to export everything to everyone because it would be the
easy way to get their job done.

Of course, some developers will do that anyway, but we don't want them
to have to.

Andrew.

[Jochen Theodorou]

Post by m***@oracle.com

Post by Andrew Haley
At Red Hat we have many Java programmers. We also have many customers
who are Java programmers. I am trying to persuade people to try out
JDK 9 in order to give us the feedback we need to ratify the JDK 9
specification. But you know where this is going: the changes to
Jigsaw which we need are TBD, but with an assurance of "I think we'll
be able to work something out." This is a tough sell for me.

Is this any different than the usual chicken-and-egg problem of
live software development? Much of Jigsaw is still in flux as we
work through issues in both the specification and the implementation.
If you have specific suggestions about how we could make it easier to
persuade people to try JDK 9 then I'd like to hear them.
Also, nobody is asking you, or anyone else, to take anything on faith.
We are -- and long have been -- asking for feedback, both practical
feedback based on running existing code against EA builds and design
feedback from developers of all skill levels.

What I would wish for at this moment is a document explaining the
runtime aspects of the module system, including limitations reflection
and normal method calls, as well as things like layers / runtime
generated modules - as well as the methods to be used to create and
change these. I don´t anything JLS ready here and just the behaviour of
the one of the latest releases is also good. I cannot give feedback on a
design I do not know. Because maybe some frameworks are content with the
compile time side of this, I am not.

bye Jochen

[Alan Bateman]

Post by Jochen Theodorou
What I would wish for at this moment is a document explaining the
runtime aspects of the module system, including limitations reflection
and normal method calls, as well as things like layers / runtime
generated modules - as well as the methods to be used to create and
change these. I don´t anything JLS ready here and just the behaviour
of the one of the latest releases is also good. I cannot give feedback
on a design I do not know. Because maybe some frameworks are content
with the compile time side of this, I am not.

Jochen - have you worked through the documents, slides and recordings
that are linked from the main Project Jigsaw page [1]?

I know in other threads we we alluded to slides on how
Nashorn/Javascript has been updated to work with modules (and spin
modules at runtime). I realize Groovy is different but I think we should
do this as it may be useful to other language implementers too.

-Alan

[1] http://openjdk.java.net/projects/jigsaw/

[Jochen Theodorou]

Post by Alan Bateman

Post by Jochen Theodorou
What I would wish for at this moment is a document explaining the
runtime aspects of the module system, including limitations reflection
and normal method calls, as well as things like layers / runtime
generated modules - as well as the methods to be used to create and
change these. I don´t anything JLS ready here and just the behaviour
of the one of the latest releases is also good. I cannot give feedback
on a design I do not know. Because maybe some frameworks are content
with the compile time side of this, I am not.

Jochen - have you worked through the documents, slides and recordings
that are linked from the main Project Jigsaw page [1]?
I know in other threads we we alluded to slides on how
Nashorn/Javascript has been updated to work with modules (and spin
modules at runtime). I realize Groovy is different but I think we should
do this as it may be useful to other language implementers too.
[1] http://openjdk.java.net/projects/jigsaw/

Well, I was hoping for a document, videos do not work well for me and
slides are better, but often useless without the actual talk... depends
of course

The problem is that this is a lot of material that barely scratches the
points I am wondering about.

Example http://openjdk.java.net/projects/jigsaw/talks/

Javaone 2015
Prepare for JDK 9 -> for me useless
Introduction to Modular Development -> for runtime aspects useless
Advanced Modular Development -> mentions addReads on slide 36 and is
done with the runtime aspect with that basically.
Project Jigsaw: Under the Hood -> is more interesting from slide 34 on,
but does not answer my questions enough.
Project Jigsaw Hack Session -> the whole thing as video

Devoxx BE 2015 is only video

I will give you an analysis of my situation so far:

assuming the whole Groovy runtime is a module, and assuming I have other
modules as well, and then I want to use a Groovy script at runtime, to
access those modules. And let us assume those modules do not know of
Groovy. Let us further assume the Groovy runtime will be in a loader
that delegates to a loader knowing the classes of those modules, or is
in the same loader.

So first problem is ... can I use the unnamed module for the scripts?
They read every (named) module, thus they can access the types.
Next are 4 types of method invocations to consider for the case of an
invocation from script in the unnamed module to named module:
reflective, indy, direct, generated bytecode for callsites (I name this
one csgen in short). The method itself is looked up by reflection

* for indy it is only a question of the lookup object, which will be
provided by the script and can do the call
* direct calls will work
* reflection would be done from the groovy runtime, thus the runtime
need to have a read on the named module. That´s the first time I will
need Java9 special code in this scenario
* csgen is more unclear. Let us assume it is in the unnamed package as
well, in a loader that delegates to the script loader. Basically it
becomes a direct call and works.

Next step is for my Groovy runtime having to call a method from the
script. Since everything is exported there is no problem on this side.
* indy will work after adding the read
* reflection - same
* direct call - same
* cs gen... the call to the csgen method will be done by an
interface/class of the Groovy runtime, and since the csgen class is in
the unnamed package, it has no problem calling the script methods.

Let us look at the same scenario with a named module for scripts. I will
use that as a step in between to think about actual compile time modules
written in Groovy. So for simplicity I assume I can create such a module
at runtime and it will export everything. How to do that? I have not the
slightest idea. Anyway, that means now I have calls from named script
module to named module:

* indy will work after the script module added a read to the target
module. This will require in the script, with a direct call to addReads,
because of caller sensitivity. Which means instead of just having JDK9
specific code in my runtime, I will now need JDK9 specific code to my
bytecode as well. And this needs to happen before any normal method
invocation can take place, thus static initializer, first thing... which
means completely new code for the compiler, that will work only on JDK9.
This means the compiler will then have to at least know if he compiles
for JDK9 or not...
* direct call - same
* reflection - the call is effectively done from the Groovy runtime, so
the Groovy runtime will have to add the read.
* csgen in the unnamed module means.. since the call is gated by an
interface/class from the runtime I assume I do not need a read edge from
the script module to the unnamed module csgen is in. The csgen class
then will be able to read the exported class of the target module, thus
has no problem.

Calls from the groovy runtime to the script are not different to before.

Next step is then precompiled Groovy code in a named module. The
difference to before is that we now have to look at calls from hidden
API to hidden API of the same module and we have compile time defined
requires and exports.

Since the Groovy runtime needs to be able to call some methods by
reflection all packages, including the hidden API, will have to be
exported to the groovy runtime and of course there will be a require for
the Groovy runtime.

Let us briefly look at a call from the script module to an exported
class in another named module again... the script module will now
require the other named module, which means we have a reads here already.
* direct call works
* reflection - runtime has to add read
* indy works
* csgen in the unnamed module... the call is again done using
interfaces/classes from the runtime, meaning it should work.

Then let us look at a call from hidden/public API to hidden API of the
same module of our precompiled script.
* direct call works
* indy works
* reflection works, since everything is exported to the groovy runtime..
* csgen in the unnamed module... while the call to csgen will work
(gated by classes/interfaces of the runtime), the call from csgen to
hidden API will fail. Adding a read won´t help, end of line. A named
module would have the same problem. Solution unclear.

As for calls from the groovy runtime to the hidden API of the script
module... Some csgen problem, everything else is fine.

Finally calls from the script module to the hidden API of another
module... I can make this sometimes work with indy and with reflection.
csgen will have the known problem, direct calls will not work. Of course
there is now the question of if I need such a call... assume you did
write a program like this:

Module A, export exported
package exported
class MyExportedList extends ArrayList {
def sum(init){
def ret = init
for (it in this) ret +=it
return ret
}
}

Module B, hidden API, require A
class HiddenX {
int value
def plus(HiddenX other) { return value+other.value }
}

def list = [new HiddenX(value:1), new HiddenX(value:2), new
HiddenX(value:3)] as MyExportedList
assert list.sum(new HiddenX(value:0)) == 6

This will require the sum method in Module A to call plus on HiddenX for
the "+=". There is no interface or class for this, just the public
method. Since the class is in the hidden API of B, A will not be able to
access it.
* A direct method call is out of question for this of course, so no
static case at all.
* reflection through the groovy runtime
* indy... before I would have used the lookup object of the caller to
realize the call. Since A cannot read HiddenX, this won´t work anymore.
I would have to resort to a private "everything is allowed" constructor
in Lookup. Basically a hack, of which I won´t know if it will work in
the next JDK version. Maybe I could "steal" a lookup object from HiddenX
to avoid this, but it is very allowed to write HiddenX in another
language like Java, and then I will not be able to produce a helper
method for this. Meaning I will have to stay with the hackish
maintenance nightmare of using a private constructor.
* csgen... the sad story continues.

And btw, how am I supposed to decide if this call is allowed or not?

To sum things up:
* module runtime creation and handling beyond adding reads is unclear
* setAccessible failing on public classes from hidden APIs is
inconvenient, but at least we can find code for this, which is
compatible for multiple JDK versions
* csgen will probably have to fall back to reflection for most cases,
which means it will be a lot slower. A bigger change would be to give it
an indy backend. We would probably still suffer a performance penalty,
since we then have an additional layer with wrapping in between, that
cannot be removed by the JIT, as long as the JIT is unable to look
beyond the indy callsite itself.
* the indy part will have to use "forbidden" API now.
* runtime scripts will go into the unnamed module
* hidden APIs and non-public modifiers will be ignored by Groovy for as
long as the "forbidden" API works, which keeps current semantics
* most of the groovy runtime makes still heavy use of reflection and
will probably have to be reworked to use indy everywhere, if on JDK9
(probably JDK8+).

And I am not happy with this outcome so far.

bye Jochen

[Remi Forax]

The problem of addRead being CallerSensitive can be solve by adding a way to add a read edge from a Lookup object ?

Rémi

----- Mail original -----

Envoyé: Vendredi 15 Juillet 2016 15:21:01
Objet: Re: Should setAccessible be part of Java or not? (was Re: It's not too late for access control)

Post by Alan Bateman

Post by Jochen Theodorou
What I would wish for at this moment is a document explaining the
runtime aspects of the module system, including limitations reflection
and normal method calls, as well as things like layers / runtime
generated modules - as well as the methods to be used to create and
change these. I don´t anything JLS ready here and just the behaviour
of the one of the latest releases is also good. I cannot give feedback
on a design I do not know. Because maybe some frameworks are content
with the compile time side of this, I am not.

Jochen - have you worked through the documents, slides and recordings
that are linked from the main Project Jigsaw page [1]?
I know in other threads we we alluded to slides on how
Nashorn/Javascript has been updated to work with modules (and spin
modules at runtime). I realize Groovy is different but I think we should
do this as it may be useful to other language implementers too.
[1] http://openjdk.java.net/projects/jigsaw/

Well, I was hoping for a document, videos do not work well for me and
slides are better, but often useless without the actual talk... depends
of course
The problem is that this is a lot of material that barely scratches the
points I am wondering about.
Example http://openjdk.java.net/projects/jigsaw/talks/
Javaone 2015
Prepare for JDK 9 -> for me useless
Introduction to Modular Development -> for runtime aspects useless
Advanced Modular Development -> mentions addReads on slide 36 and is
done with the runtime aspect with that basically.
Project Jigsaw: Under the Hood -> is more interesting from slide 34 on,
but does not answer my questions enough.
Project Jigsaw Hack Session -> the whole thing as video
Devoxx BE 2015 is only video
assuming the whole Groovy runtime is a module, and assuming I have other
modules as well, and then I want to use a Groovy script at runtime, to
access those modules. And let us assume those modules do not know of
Groovy. Let us further assume the Groovy runtime will be in a loader
that delegates to a loader knowing the classes of those modules, or is
in the same loader.
So first problem is ... can I use the unnamed module for the scripts?
They read every (named) module, thus they can access the types.
Next are 4 types of method invocations to consider for the case of an
reflective, indy, direct, generated bytecode for callsites (I name this
one csgen in short). The method itself is looked up by reflection
* for indy it is only a question of the lookup object, which will be
provided by the script and can do the call
* direct calls will work
* reflection would be done from the groovy runtime, thus the runtime
need to have a read on the named module. That´s the first time I will
need Java9 special code in this scenario
* csgen is more unclear. Let us assume it is in the unnamed package as
well, in a loader that delegates to the script loader. Basically it
becomes a direct call and works.
Next step is for my Groovy runtime having to call a method from the
script. Since everything is exported there is no problem on this side.
* indy will work after adding the read
* reflection - same
* direct call - same
* cs gen... the call to the csgen method will be done by an
interface/class of the Groovy runtime, and since the csgen class is in
the unnamed package, it has no problem calling the script methods.
Let us look at the same scenario with a named module for scripts. I will
use that as a step in between to think about actual compile time modules
written in Groovy. So for simplicity I assume I can create such a module
at runtime and it will export everything. How to do that? I have not the
slightest idea. Anyway, that means now I have calls from named script
* indy will work after the script module added a read to the target
module. This will require in the script, with a direct call to addReads,
because of caller sensitivity. Which means instead of just having JDK9
specific code in my runtime, I will now need JDK9 specific code to my
bytecode as well. And this needs to happen before any normal method
invocation can take place, thus static initializer, first thing... which
means completely new code for the compiler, that will work only on JDK9.
This means the compiler will then have to at least know if he compiles
for JDK9 or not...
* direct call - same
* reflection - the call is effectively done from the Groovy runtime, so
the Groovy runtime will have to add the read.
* csgen in the unnamed module means.. since the call is gated by an
interface/class from the runtime I assume I do not need a read edge from
the script module to the unnamed module csgen is in. The csgen class
then will be able to read the exported class of the target module, thus
has no problem.
Calls from the groovy runtime to the script are not different to before.
Next step is then precompiled Groovy code in a named module. The
difference to before is that we now have to look at calls from hidden
API to hidden API of the same module and we have compile time defined
requires and exports.
Since the Groovy runtime needs to be able to call some methods by
reflection all packages, including the hidden API, will have to be
exported to the groovy runtime and of course there will be a require for
the Groovy runtime.
Let us briefly look at a call from the script module to an exported
class in another named module again... the script module will now
require the other named module, which means we have a reads here already.
* direct call works
* reflection - runtime has to add read
* indy works
* csgen in the unnamed module... the call is again done using
interfaces/classes from the runtime, meaning it should work.
Then let us look at a call from hidden/public API to hidden API of the
same module of our precompiled script.
* direct call works
* indy works
* reflection works, since everything is exported to the groovy runtime..
* csgen in the unnamed module... while the call to csgen will work
(gated by classes/interfaces of the runtime), the call from csgen to
hidden API will fail. Adding a read won´t help, end of line. A named
module would have the same problem. Solution unclear.
As for calls from the groovy runtime to the hidden API of the script
module... Some csgen problem, everything else is fine.
Finally calls from the script module to the hidden API of another
module... I can make this sometimes work with indy and with reflection.
csgen will have the known problem, direct calls will not work. Of course
there is now the question of if I need such a call... assume you did
Module A, export exported
package exported
class MyExportedList extends ArrayList {
def sum(init){
def ret = init
for (it in this) ret +=it
return ret
}
}
Module B, hidden API, require A
class HiddenX {
int value
def plus(HiddenX other) { return value+other.value }
}
def list = [new HiddenX(value:1), new HiddenX(value:2), new
HiddenX(value:3)] as MyExportedList
assert list.sum(new HiddenX(value:0)) == 6
This will require the sum method in Module A to call plus on HiddenX for
the "+=". There is no interface or class for this, just the public
method. Since the class is in the hidden API of B, A will not be able to
access it.
* A direct method call is out of question for this of course, so no
static case at all.
* reflection through the groovy runtime
* indy... before I would have used the lookup object of the caller to
realize the call. Since A cannot read HiddenX, this won´t work anymore.
I would have to resort to a private "everything is allowed" constructor
in Lookup. Basically a hack, of which I won´t know if it will work in
the next JDK version. Maybe I could "steal" a lookup object from HiddenX
to avoid this, but it is very allowed to write HiddenX in another
language like Java, and then I will not be able to produce a helper
method for this. Meaning I will have to stay with the hackish
maintenance nightmare of using a private constructor.
* csgen... the sad story continues.
And btw, how am I supposed to decide if this call is allowed or not?
* module runtime creation and handling beyond adding reads is unclear
* setAccessible failing on public classes from hidden APIs is
inconvenient, but at least we can find code for this, which is
compatible for multiple JDK versions
* csgen will probably have to fall back to reflection for most cases,
which means it will be a lot slower. A bigger change would be to give it
an indy backend. We would probably still suffer a performance penalty,
since we then have an additional layer with wrapping in between, that
cannot be removed by the JIT, as long as the JIT is unable to look
beyond the indy callsite itself.
* the indy part will have to use "forbidden" API now.
* runtime scripts will go into the unnamed module
* hidden APIs and non-public modifiers will be ignored by Groovy for as
long as the "forbidden" API works, which keeps current semantics
* most of the groovy runtime makes still heavy use of reflection and
will probably have to be reworked to use indy everywhere, if on JDK9
(probably JDK8+).
And I am not happy with this outcome so far.
bye Jochen

[Alex Buckley]

Post by Jochen Theodorou
assuming the whole Groovy runtime is a module, and assuming I have other
modules as well, and then I want to use a Groovy script at runtime, to
access those modules. And let us assume those modules do not know of
Groovy. Let us further assume the Groovy runtime will be in a loader
that delegates to a loader knowing the classes of those modules, or is
in the same loader.
So first problem is ... can I use the unnamed module for the scripts?
They read every (named) module, thus they can access the types.
Next are 4 types of method invocations to consider for the case of an
reflective, indy, direct, generated bytecode for callsites (I name this
one csgen in short). The method itself is looked up by reflection
* for indy it is only a question of the lookup object, which will be
provided by the script and can do the call
* direct calls will work
* reflection would be done from the groovy runtime, thus the runtime
need to have a read on the named module. That´s the first time I will
need Java9 special code in this scenario
* csgen is more unclear. Let us assume it is in the unnamed package as
well, in a loader that delegates to the script loader. Basically it
becomes a direct call and works.
Next step is for my Groovy runtime having to call a method from the
script. Since everything is exported there is no problem on this side.
* indy will work after adding the read
* reflection - same
* direct call - same
* cs gen... the call to the csgen method will be done by an
interface/class of the Groovy runtime, and since the csgen class is in
the unnamed package, it has no problem calling the script methods.

With one exception, everything above is correct. (Exception: reflection
from the Groovy module to a named module, performed on behalf of the
script, does NOT need Java 9 code, because reflection gets readability
for free. Also, you mean that csgen code is in the unnamed MODULE; its
package is not so important.)

Key point: Just because a framework is delivered as a module, does not
mean the framework has to change its class loading behavior or require
that its plugins/scripts/introspectees be delivered as modules. Groovy
can continue to load scripts (or classes derived therefrom) into
user-defined loaders. The scripts can continue to access each other
(assuming suitable loader delegation) and Java platform APIs and Groovy
runtime APIs (assuming the Groovy module exports some packages).

Post by Jochen Theodorou
Let us look at the same scenario with a named module for scripts. I will
use that as a step in between to think about actual compile time modules
written in Groovy. So for simplicity I assume I can create such a module
at runtime and it will export everything. How to do that? I have not the
slightest idea. Anyway, that means now I have calls from named script
* indy will work after the script module added a read to the target
module. This will require in the script, with a direct call to addReads,
because of caller sensitivity. Which means instead of just having JDK9
specific code in my runtime, I will now need JDK9 specific code to my
bytecode as well. And this needs to happen before any normal method
invocation can take place, thus static initializer, first thing... which
means completely new code for the compiler, that will work only on JDK9.
This means the compiler will then have to at least know if he compiles
for JDK9 or not...
* direct call - same
* reflection - the call is effectively done from the Groovy runtime, so
the Groovy runtime will have to add the read.
* csgen in the unnamed module means.. since the call is gated by an
interface/class from the runtime I assume I do not need a read edge from
the script module to the unnamed module csgen is in. The csgen class
then will be able to read the exported class of the target module, thus
has no problem.
Calls from the groovy runtime to the script are not different to before.

For creating named modules at run time, see the javadoc for
java.lang.reflect.Layer and all the linked javadoc down to
java.lang.module.ModuleDescriptor.Builder.

You're right about indy calls and direct calls following readability, so
you can either have the script's module require another named module in
the Configuration that you pass to the Layer, or you can have the
script's own code add readability as you describe.

(For reflection, note again that the Groovy runtime does NOT have to add
a read from its module to another named module.)

Now for a pivot. All the issues you raise in the rest of this email,
about how named modules containing bytecode-derived-from-Groovy-source
interact with named modules of the Groovy runtime or otherwise, are
familiar. Why? Because Nashorn faced exactly the same issues. There will
be a presentation at the JVM Language Summit (jvmlangsummit.com) on how
Nashorn uses modules to encapsulate its generated bytecode. The video
will be online at YouTube the next day. It'll make discussing the issues
below much easier.

Alex

Post by Jochen Theodorou
Next step is then precompiled Groovy code in a named module. The
difference to before is that we now have to look at calls from hidden
API to hidden API of the same module and we have compile time defined
requires and exports.
Since the Groovy runtime needs to be able to call some methods by
reflection all packages, including the hidden API, will have to be
exported to the groovy runtime and of course there will be a require for
the Groovy runtime.
Let us briefly look at a call from the script module to an exported
class in another named module again... the script module will now
require the other named module, which means we have a reads here already.
* direct call works
* reflection - runtime has to add read
* indy works
* csgen in the unnamed module... the call is again done using
interfaces/classes from the runtime, meaning it should work.
Then let us look at a call from hidden/public API to hidden API of the
same module of our precompiled script.
* direct call works
* indy works
* reflection works, since everything is exported to the groovy runtime..
* csgen in the unnamed module... while the call to csgen will work
(gated by classes/interfaces of the runtime), the call from csgen to
hidden API will fail. Adding a read won´t help, end of line. A named
module would have the same problem. Solution unclear.
As for calls from the groovy runtime to the hidden API of the script
module... Some csgen problem, everything else is fine.
Finally calls from the script module to the hidden API of another
module... I can make this sometimes work with indy and with reflection.
csgen will have the known problem, direct calls will not work. Of course
there is now the question of if I need such a call... assume you did
Module A, export exported
package exported
class MyExportedList extends ArrayList {
def sum(init){
def ret = init
for (it in this) ret +=it
return ret
}
}
Module B, hidden API, require A
class HiddenX {
int value
def plus(HiddenX other) { return value+other.value }
}
def list = [new HiddenX(value:1), new HiddenX(value:2), new
HiddenX(value:3)] as MyExportedList
assert list.sum(new HiddenX(value:0)) == 6
This will require the sum method in Module A to call plus on HiddenX for
the "+=". There is no interface or class for this, just the public
method. Since the class is in the hidden API of B, A will not be able to
access it.
* A direct method call is out of question for this of course, so no
static case at all.
* reflection through the groovy runtime
* indy... before I would have used the lookup object of the caller to
realize the call. Since A cannot read HiddenX, this won´t work anymore.
I would have to resort to a private "everything is allowed" constructor
in Lookup. Basically a hack, of which I won´t know if it will work in
the next JDK version. Maybe I could "steal" a lookup object from HiddenX
to avoid this, but it is very allowed to write HiddenX in another
language like Java, and then I will not be able to produce a helper
method for this. Meaning I will have to stay with the hackish
maintenance nightmare of using a private constructor.
* csgen... the sad story continues.
And btw, how am I supposed to decide if this call is allowed or not?
* module runtime creation and handling beyond adding reads is unclear
* setAccessible failing on public classes from hidden APIs is
inconvenient, but at least we can find code for this, which is
compatible for multiple JDK versions
* csgen will probably have to fall back to reflection for most cases,
which means it will be a lot slower. A bigger change would be to give it
an indy backend. We would probably still suffer a performance penalty,
since we then have an additional layer with wrapping in between, that
cannot be removed by the JIT, as long as the JIT is unable to look
beyond the indy callsite itself.
* the indy part will have to use "forbidden" API now.
* runtime scripts will go into the unnamed module
* hidden APIs and non-public modifiers will be ignored by Groovy for as
long as the "forbidden" API works, which keeps current semantics
* most of the groovy runtime makes still heavy use of reflection and
will probably have to be reworked to use indy everywhere, if on JDK9
(probably JDK8+).
And I am not happy with this outcome so far.
bye Jochen

[Jochen Theodorou]

Post by Alex Buckley

Post by Jochen Theodorou
assuming the whole Groovy runtime is a module, and assuming I have other
modules as well, and then I want to use a Groovy script at runtime, to
access those modules. And let us assume those modules do not know of
Groovy. Let us further assume the Groovy runtime will be in a loader
that delegates to a loader knowing the classes of those modules, or is
in the same loader.
So first problem is ... can I use the unnamed module for the scripts?
They read every (named) module, thus they can access the types.
Next are 4 types of method invocations to consider for the case of an
reflective, indy, direct, generated bytecode for callsites (I name this
one csgen in short). The method itself is looked up by reflection
* for indy it is only a question of the lookup object, which will be
provided by the script and can do the call
* direct calls will work
* reflection would be done from the groovy runtime, thus the runtime
need to have a read on the named module. That´s the first time I will
need Java9 special code in this scenario
* csgen is more unclear. Let us assume it is in the unnamed package as
well, in a loader that delegates to the script loader. Basically it
becomes a direct call and works.
Next step is for my Groovy runtime having to call a method from the
script. Since everything is exported there is no problem on this side.
* indy will work after adding the read
* reflection - same
* direct call - same
* cs gen... the call to the csgen method will be done by an
interface/class of the Groovy runtime, and since the csgen class is in
the unnamed package, it has no problem calling the script methods.

With one exception, everything above is correct. (Exception: reflection
from the Groovy module to a named module, performed on behalf of the
script, does NOT need Java 9 code, because reflection gets readability
for free. Also, you mean that csgen code is in the unnamed MODULE; its
package is not so important.)

yes, my mistake, looks like I did not see it even the second or third
time I did read through my post. Anyway, thanks for adding the part
about reflection being able to read everything. I did forget that. This
simplifies matters for the reflection part.

Post by Alex Buckley
Key point: Just because a framework is delivered as a module, does not
mean the framework has to change its class loading behavior or require
that its plugins/scripts/introspectees be delivered as modules. Groovy
can continue to load scripts (or classes derived therefrom) into
user-defined loaders. The scripts can continue to access each other
(assuming suitable loader delegation) and Java platform APIs and Groovy
runtime APIs (assuming the Groovy module exports some packages).

Well, that is exactly what I am wondering about. Should we even go the
module route or not? I did for example not attempt to make the runtime
scripts modules, as I did not found need for actually doing that so far.
And those runtime scripts are what you are talking about mostly. But
people are supposed to be able to use precompiled Groovy as much as Java
as possible. Not being able to make a module for Groovy code would be
bad then. And if you have Java code, that depends on a library written
in Groovy, do you really want to be blocked from making this a module,
just because of that library being in Groovy? That will be bad for
Groovy and the library.

Groovy is not like many JVM languages, in which you basically switch
over into a new universe and have to do conversions and special things
to go back to Java or use something of that language in Java. With the
effect of staying in one universe as much as possible. Instead we target
full integration. It is very common for us to switch between Groovy and
Java all the time in the same program. That´s why a library written in
Groovy is supposed to be usable from Java without quirks... and that
includes the module system.

[...]

Post by Alex Buckley
For creating named modules at run time, see the javadoc for
java.lang.reflect.Layer and all the linked javadoc down to
java.lang.module.ModuleDescriptor.Builder.

What if I have to make anamed module at runtime to realize a call that
involves sibling layers? I do not see an answer to that in the javadoc.
Not that I know if I would need that... but that is also information I
do not have in the javadoc

Post by Alex Buckley
You're right about indy calls and direct calls following readability, so
you can either have the script's module require another named module in
the Configuration that you pass to the Layer, or you can have the
script's own code add readability as you describe.

so far the named module version looks much more complicated...

Post by Alex Buckley
Now for a pivot. All the issues you raise in the rest of this email,
about how named modules containing bytecode-derived-from-Groovy-source
interact with named modules of the Groovy runtime or otherwise, are
familiar. Why? Because Nashorn faced exactly the same issues. There will
be a presentation at the JVM Language Summit (jvmlangsummit.com) on how
Nashorn uses modules to encapsulate its generated bytecode. The video
will be online at YouTube the next day. It'll make discussing the issues
below much easier.

ah well, since I waited already for several months, I guess I can wait
another 2 or 3 weeks.

bye Jochen

[David M. Lloyd]

Post by Alan Bateman

OK. But "in the very long term" such a basic language change needs
all stakeholders to be consulted.

I agree (although it's not really a language change in that it's API way
to suppress access checks specified by the language).

Yes, indeed, and that is potentially a significant problem. My
comment stands: there is a serious possibility that his will make it
impossible to use (non-exported) Jigsaw modules for some kinds of
programming. This is exactly the kind of decision that needs all
stakeholders to be consulted.
It ought to be possible to have some kind of conditional export which
only allows such access by (e.g.) suitably privileged frameworks or
tools. But I have no desire to get involved in such design issues: I
am only going to say that this is an issue which requires wider
consultation.

This is #ReflectiveAccessToNonExportedTypes on the JSR 376 issues list.
The problem is reasonably well understood and there are several
proposals and approaches being discussed and considered.

An important aspect of my proposal is that, by separating exports from
accessibility, any framework which operates on public classes will
continue to work without any modifications to exports. Dropping
'setAccessible' becomes more of a likelihood when frameworks have *some*
option (like the ability to reflect any public class) available to them;
even if it means, long-term, that these specifications and frameworks
slowly have to make their way towards operating only on public classes.

In fact I would say that it makes a hypothetical migration path
available for this purpose:

Time Action
---- ------
0 Jigsaw introduces friend packages
1 Frameworks slowly move to hide non-public public things
n Java makes setAccessible() disablable via some --safe-mode switch
Users begin to move to public types for their frameworked classes
m Java makes --safe-mode the default, adds --unsafe-mode for legacy
things
More users move to public types
z Java drops support for --unsafe-mode, setAccessible throws UOE
forever after

In this example, n could even equal 0 - that is, we could start this
right now.

[John Rose]

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me."

Saying setAccessible is like "borrowing" (without owner permission) a
key to one locked door, if a non-public method is like a locked door.

Today's MethodHandles.Lookup object gives another way to open such
doors. But you have to obtain the lookup object from a party that already
has access rights. It is like the owner of a building (a class) giving a key
which opens all the doors in the building, or all the doors not marked "Private".

With both setA Methods and Lookups, once you have the key in hand,
you have to lock it up to prevent bad guys from stealing it from you.
And if you loan it out, you have to loan it to trustworthy parties.

Somewhere in between the two (unrestricted "borrowing" vs. direct delegation
of original access rights) must be some better conventions for reflecting into frameworks.

— John

[Jason Greene]

Post by John Rose

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me."

Saying setAccessible is like "borrowing" (without owner permission) a
key to one locked door, if a non-public method is like a locked door.

Not to sound like a broken record, but not all systems want the module to control its own security. They want an intermediary.

And that's a perfectly fine and reasonable security model.

Post by John Rose
Today's MethodHandles.Lookup object gives another way to open such
doors. But you have to obtain the lookup object from a party that already
has access rights. It is like the owner of a building (a class) giving a key
which opens all the doors in the building, or all the doors not marked "Private".
With both setA Methods and Lookups, once you have the key in hand,
you have to lock it up to prevent bad guys from stealing it from you.
And if you loan it out, you have to loan it to trustworthy parties.
Somewhere in between the two (unrestricted "borrowing" vs. direct delegation
of original access rights) must be some better conventions for reflecting into frameworks.
— John

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

[Paul Benedict]

Agreed with Jason. It's okay to say thank you, but no thank you. A third
party library maintainer, no matter how well-intentioned, has absolutely no
say over the way I design, assemble, and run my operations. Reflection is
risky, yes, but it's my risk to take. If I bust down the wrong wall and do
something "wrong", that's my responsibility... but it's still my decision
to make as a consumer.

Cheers,
Paul

Post by Jason Greene

Post by John Rose

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me."

Saying setAccessible is like "borrowing" (without owner permission) a
key to one locked door, if a non-public method is like a locked door.

Not to sound like a broken record, but not all systems want the module to
control its own security. They want an intermediary.
And that's a perfectly fine and reasonable security model.

Post by John Rose
Today's MethodHandles.Lookup object gives another way to open such
doors. But you have to obtain the lookup object from a party that

already

Post by John Rose
has access rights. It is like the owner of a building (a class) giving

a key

Post by John Rose
which opens all the doors in the building, or all the doors not marked

"Private".

Post by John Rose
With both setA Methods and Lookups, once you have the key in hand,
you have to lock it up to prevent bad guys from stealing it from you.
And if you loan it out, you have to loan it to trustworthy parties.
Somewhere in between the two (unrestricted "borrowing" vs. direct

delegation

Post by John Rose
of original access rights) must be some better conventions for

reflecting into frameworks.

Post by John Rose
— John

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

[John Rose]

Post by Jason Greene
Not to sound like a broken record, but not all systems want the module to control its own security. They want an intermediary.
And that's a perfectly fine and reasonable security model.

I agree, as long as the intermediary is highly predictable and accountable
in its violations of standard access rules.

And that's in the middle ground I'm talking about, where a module (or class)
is deemed (without direct delegation on its part) to extend trusted access to some
intermediary, which is generally trusted to do things on behalf of a broad range of
such modules (or classes). It's middle ground (not the extreme of setA) because
(a) the intermediary would be more clearly marked as trusted (not just contain
a setA call somewhere in its guts), and (b) the task entrusted to the intermediary
would not require trusted access to all classes, but some well-defined "slice"
across a well-defined set of classes.

To reason about setA you have to predict the actions of a Turing machine,
as it assembles Class and String constants to pick out any API point whatever,
and then make a skeleton key for it.

A good framework makes those actions predictable, based on rules which
allow predictions based on the form of a program rather than the steps it
takes during execution.

Personally, I prefer the MethodHandles.Lookup API to setA (when I have
a choice between the two) because Lookup respects the existing access
control boundaries, while allowing delegation. I think the data flow of
capabilities is easier to reason about than the data flow of random String
and Class values, followed by an unpredictable setA call. But a Lookup
(today) must be created by the principal that owns the access rights,
which is what I think Jason is objecting to.

If we were to make some sort of middle ground between setA and Lookup,
it would look like a kind of Lookup that was obtained with upgraded access
(relative to publicLookup), but without direct creation by the target class.

In exchange for that power, there would be rules for limiting and configuring
the effect of such a Lookup. For example, it might work only for API points
carrying a particular annotation.

You can build this sort of thing on top of setA of course, since setA is
basically root privileges. My point is not that access rules should be
inviolate, but that exceptions to the rules should be as predictable and
even configurable as possible.

A final point: Predictability and configurability leads to alternative
implementations, such as AOP-like injection of code at AOT time.
That's one of the promises of jlink.

HTH
— John

[Gregg Wonderly]

Post by Alan Bateman

Yes, indeed, and that is potentially a significant problem. My
comment stands: there is a serious possibility that his will make it
impossible to use (non-exported) Jigsaw modules for some kinds of
programming. This is exactly the kind of decision that needs all
stakeholders to be consulted.
It ought to be possible to have some kind of conditional export which
only allows such access by (e.g.) suitably privileged frameworks or
tools. But I have no desire to get involved in such design issues: I
am only going to say that this is an issue which requires wider
consultation.

This is #ReflectiveAccessToNonExportedTypes on the JSR 376 issues list.
The problem is reasonably well understood and there are several
proposals and approaches being discussed and considered.

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me.”

That is the question though. Why does it seem like a good idea to limit accessibility when there are so many other ways that the software can be exploited without this single limit being able to control all the others? Do you think that people would not decide to decompile your module definition and change all the details for visibility so that they can then do with it exactly as they need? In this day and age, and especially in the public software realm, control of anything does not exist in practicality. No matter what you believe might limit any aspect of your software, there is no way to categorically enforce that.

Only on closed systems with all kinds of access control limits could you start to have some ability to limit what happens with/for the software use.

Gregg

[Andrew Haley]

Post by Gregg Wonderly

Post by Alan Bateman
This is #ReflectiveAccessToNonExportedTypes on the JSR 376 issues list.
The problem is reasonably well understood and there are several
proposals and approaches being discussed and considered.

Forgive me if I've missed something, but
#ReflectiveAccessToNonExportedTypes does not deal with the need to
make fields or methods accessible to the framework. That's what
setAccessible is used for. It would certainly be nice for a
framework to be able to say "make it accessible, but only to me.”

That is the question though. Why does it seem like a good idea to
limit accessibility when there are so many other ways that the
software can be exploited without this single limit being able to
control all the others?

It's no more than a practical way of reducing software complexity. (I
don't intend to explain why complexity is the enemy of reliability.)

System complexity is related to the number of connections between
components in a system. By opening up access with setAccessible() you
add an interface to a module. If that interface is only accessible to
one client, it's a 1:1 connection. Making it globally accessible is
(potentially) a 1:N connection. The wider the access, the more scope
for complex interfaces.

Also, making a method or field globally accessible may mean that the
author of the code can no longer guarantee its correctness.

This is commonplace computer science: it should be familiar to all of
us.

Post by Gregg Wonderly
Do you think that people would not decide to decompile your module
definition and change all the details for visibility so that they
can then do with it exactly as they need? In this day and age, and
especially in the public software realm, control of anything does
not exist in practicality. No matter what you believe might limit
any aspect of your software, there is no way to categorically
enforce that.

No, but we can reduce risk and the size of attack surfaces. And, all
other things being equal, we should.

But we must do so in full awareness of the techniques that real-world
Java software uses. The plasticity of Java has, to a very large
extent, led to its pre-eminence. We must not lose that. There are
bad uses of setAccessible() and good ones.

Andrew.

[John Rose]

Thanks, Andrew, for a good summary of the practicalities behind
access overrides.

The wider the access, the more scope for complex interfaces.
Also, making a method or field globally accessible may mean that the
author of the code can no longer guarantee its correctness.
… we can reduce risk and the size of attack surfaces

I especially agree with these parts, from bitter experience. When you
widen an interface, you add attack surface. If you widen an interface
with clear documentation and buy-in from the author of the interface,
and can add both negative and positive tests for the new API surface,
then you can sleep at night. Meanwhile, the best attack surfaces are
implicitly defined (e.g., privates which aren't really private after all)
and are activated by remote control (e.g., setA), without the buy-in
of the library author.

I'm not saying this to condemn frameworks which perform polymorphic,
pattern-driven access overrides, but to explain their special risks. And
I think in the future we will have a much more explicit and robust set
of rules for opt-in and opt-out into such frameworks, including optioning
which is delegated to third parties, yet is still limited to a small subset
of what is available to root privileges (aka. setA).

Finally, as JIT guy, I'd like to note that one way to crisply evaluate
the "weight" of an API surface given away by some "fancy feature"
(ubiquitous override-ables, dynamic loading, monitor-per-object, AOP,
access overrides, …) is to see how many optimizations it makes difficult
or impossible. Yes, Java is wonderfully plastic, and yes, we need to
supply hardening transformations so the plastic can be made durable.

— John

[Robert Muir]

Going off-topic slightly but when you do run into issues and if they aren't
already listed as compatibility issues then please bring them up. So far
then I think the vast majority of issues that we have heard about relate to
the changes in JEP 220 (tools.jar going away etc), JEP 223 and the new
version-string scheme, the class file version bump in jdk-9+119, and then
all the issues that we have listed in JEP 261. Any help getting bugs
submitted to projects would be appreciated too.

Java 9 breaks log4j 1.2.x out of box. Think about how widely used that
one is! And its no longer maintained!

Sure, its only if you use MDC, and its not even Jigsaw that does it
(relates to version changes). See
https://github.com/elastic/elasticsearch/blob/f01f15d3b8592db4210f725f0c37baff4a554a35/core/src/main/java/org/apache/log4j/Java9Hack.java
if you want a hack.

Its just an example of how much software won't work with java 9 out of
box. Still looking forward to it!

[Alan Bateman]

Post by Robert Muir
Java 9 breaks log4j 1.2.x out of box. Think about how widely used that
one is! And its no longer maintained!
Sure, its only if you use MDC, and its not even Jigsaw that does it
(relates to version changes). See
https://github.com/elastic/elasticsearch/blob/f01f15d3b8592db4210f725f0c37baff4a554a35/core/src/main/java/org/apache/log4j/Java9Hack.java
if you want a hack.
Its just an example of how much software won't work with java 9 out of
box. Still looking forward to it!

Yeah, changing the version-string scheme, particularly dropping the
leading `1.` has impacted a lot of libraries. Every effort was made to
communicate that change and has all the others with significant
compatibility impact. I would hope that most libraries that need update
will be updated by the time that JDK 9 ships. Good point on unmaintained
libraries.

-Alan

[dalibor topic]

Post by Robert Muir
Java 9 breaks log4j 1.2.x out of box. Think about how widely used that
one is! And its no longer maintained!

I'd suggest moving on [1] to a maintained version of that dependency,
such as 2.6.x currently seems to be.

While that doesn't necessarily mean it won't break in the same or a
different fashion in the future, at least you would have a chance to fix
such issues or to have someone else fix them for you. ;)

cheers,
dalibor topic

[1] https://logging.apache.org/log4j/2.x/manual/migration.html
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Robert Muir]

I'd suggest moving on [1] to a maintained version of that dependency, such
as 2.6.x currently seems to be.

I'm not complaining about the issue: I'm simply trying to put things
in perspective, communicate a bit of a reality check as to what is
going to happen. *tons* of libraries depend on log4j 1.x, it may even
be more widely used than guava.

We try to do the right thing and fix the issues we encounter, when
testing java 9, and send patches where they should go. But it is hard
when the software is unmaintained, or stubborn (e.g.
https://github.com/aws/aws-sdk-java/pull/718)

[dalibor topic]

Post by Robert Muir

I'd suggest moving on [1] to a maintained version of that dependency, such
as 2.6.x currently seems to be.

I'm not complaining about the issue: I'm simply trying to put things
in perspective, communicate a bit of a reality check as to what is
going to happen. *tons* of libraries depend on log4j 1.x, it may even
be more widely used than guava.

It's pretty close, yeah:

https://mvnrepository.com/artifact/com.google.guava/guava/usages: 7490
https://mvnrepository.com/artifact/log4j/log4j/usages : 7439

The migration to the next version seems to have
https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core/usages
: 859

so it looks like it's getting adopted a little bit faster than log4j 1.x
originally was.

Post by Robert Muir
We try to do the right thing and fix the issues we encounter, when
testing java 9, and send patches where they should go.

Thank you for doing that hard work!

Post by Robert Muir
But it is hard
when the software is unmaintained, or stubborn (e.g.
https://github.com/aws/aws-sdk-java/pull/718)

In general, awareness of the importance of planning to adopt future
revisions of one's dependencies seems to be very low across the open
source development spectrum, with very few exceptions (Linux distro
rebuilds with latest GCC versions, etc.), often leading to ad-hoc
decision making in many projects about such upgrades, which is fueled in
turn by the lack of reliable release or support roadmaps from their open
source dependencies.

With respect to the JDK and the Java community specifically, I think
things have got a bit better then they were a few years ago, thanks to
the work Rory and his collaborators like yourself are doing on raising
awareness of upcoming JDK changes through the Quality Outreach efforts,
but as you say, there is still a long way to go, and we could always use
more collaborators. ;)

cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Robert Muir]

Post by dalibor topic
so it looks like it's getting adopted a little bit faster than log4j 1.x
originally was.

Yes my concern there esp is around the way in which it breaks: subtle
change in behavior, not some obvious exception message (such as when
someone abuses an internal api). My coworker and I spent half a day
tracking down what was happening, and we at least have some knowledge
about what to look for.

So I just sense some trouble around this stuff for java 9: and some
users won't care if its jigsaw or a versioning change or whatever, it
just won't work for them.

Post by dalibor topic
With respect to the JDK and the Java community specifically, I think things
have got a bit better then they were a few years ago, thanks to the work
Rory and his collaborators like yourself are doing on raising awareness of
upcoming JDK changes through the Quality Outreach efforts, but as you say,
there is still a long way to go, and we could always use more collaborators.
;)

I know its super-offtopic but this has been a great change.

[dalibor topic]

Post by Robert Muir
So I just sense some trouble around this stuff for java 9: and some
users won't care if its jigsaw or a versioning change or whatever, it
just won't work for them.

Yeah, there is a lot more than just the changes within this Project that
can affect code migrating to JDK 9.

I maintain a list of things to consider at
https://wiki.openjdk.java.net/display/Adoption/JDK+9+Outreach and would
welcome suggestions for improving and expanding that list over on the
adoption-discuss mailing list.

cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Paul Benedict]

I'll raise this subject over in the Log4J community where I am also active.
Just note 1.x is EOL and is no longer maintained. Moving to 2.x is a whole
redesign and the two are not compatible (neither binary nor configuration
wise). 2.x is way better but it's not drop-and-replace. If you have a
dependency on Log4J 1.x and cannot move forward for whatever reason, I
think the only relief you will have is to patch yourself, a patch from your
EE server provider, or ping the Log4J community to do an OOB patch.

Cheers,
Paul

Post by dalibor topic

Post by Robert Muir

I'd suggest moving on [1] to a maintained version of that dependency, such
as 2.6.x currently seems to be.

I'm not complaining about the issue: I'm simply trying to put things
in perspective, communicate a bit of a reality check as to what is
going to happen. *tons* of libraries depend on log4j 1.x, it may even
be more widely used than guava.

https://mvnrepository.com/artifact/com.google.guava/guava/usages: 7490
https://mvnrepository.com/artifact/log4j/log4j/usages : 7439
The migration to the next version seems to have
https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core/usages
: 859
so it looks like it's getting adopted a little bit faster than log4j 1.x
originally was.
We try to do the right thing and fix the issues we encounter, when

Post by Robert Muir
testing java 9, and send patches where they should go.

Thank you for doing that hard work!
But it is hard

Post by Robert Muir
when the software is unmaintained, or stubborn (e.g.
https://github.com/aws/aws-sdk-java/pull/718)

In general, awareness of the importance of planning to adopt future
revisions of one's dependencies seems to be very low across the open source
development spectrum, with very few exceptions (Linux distro rebuilds with
latest GCC versions, etc.), often leading to ad-hoc decision making in many
projects about such upgrades, which is fueled in turn by the lack of
reliable release or support roadmaps from their open source dependencies.
With respect to the JDK and the Java community specifically, I think
things have got a bit better then they were a few years ago, thanks to the
work Rory and his collaborators like yourself are doing on raising
awareness of upcoming JDK changes through the Quality Outreach efforts, but
as you say, there is still a long way to go, and we could always use more
collaborators. ;)
cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>
ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg
ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Sanne Grinovero]

Post by Sanne Grinovero
As a maintainer and contributor to several popular Java open source
libraries my experience is that in practice very few existing
libraries will "just work" in Java 9 out of the box: people will have
to update their code. This is based on my experience, as some of these
projects have dozens of dependencies and it's taking a long time to
identify each problematic point, discuss patches, getting other
communities to release timely, an often there is need for "recursive
releases" and various iterations for each problem as they get
identified, for each dependency.

Going off-topic slightly but when you do run into issues and if they aren't
already listed as compatibility issues then please bring them up. So far
then I think the vast majority of issues that we have heard about relate to
the changes in JEP 220 (tools.jar going away etc), JEP 223 and the new
version-string scheme, the class file version bump in jdk-9+119, and then
all the issues that we have listed in JEP 261. Any help getting bugs
submitted to projects would be appreciated too.
The issue that `public` no longer implies accessible is listed in JEP 261
(first item in the Risks and Assumption) but to be honest, has barely come
up to date. That probably isn't too surprising as it's still early days for
modules and many projects aren't trying out JDK 9 yet. Anyone trying out
modules where a "module unaware" framework gets a reference to a public type
in a non-exported package might run into it of course but I'm not aware of
any reports yet.

Hi Alan, you're spot-on about JEP-220: myself and my colleagues have
sent patches to numerous projects to which I normally don't contribute
to, just to fix our dependencies, such as Maven.

But please be aware of how many of the OSS projects in the Java
community are organised: some will have integration tests running in
modular environments, like WildFly which is based on David Lloyd's
JBoss Modules, or for OSGi tests it seems popular to use Apache Karaf.
Many others don't have any "modular" or "container" integration tests
and are not aware of what's coming.. since in my team we package some
of these for usage in modular environments, I'm painfully aware of
problems coming but it's no easy task to convince them for the need to
test now.

When I get in touch to try convey a sense of urgent need to test
Jigsaw, and sometimes successfully, then people realise that their
whole toolchain, not least the testing environment, blows up with
errors which are often too complex or simply too unrelated to the
project itself.

So what happens is that such projects go to communities like WildFly
and OSGi for advice, where the typical answer is among the lines "I
know, we're discussing such things on the jigsaw-dev list, watch
thread XY, or watch #MyIssueCategory on Mark's blogs".

In short, please be aware that David Lloyd isn't making such
suggestions just for WildFly itself, but is pre-processing and
summarising feedback from hundreds of projects there. The ones who
care are lurking on this list hoping to see answers for issues which
have been raised already, but not many feel competent enough to join
the conversation directly (and this might be a good thing!).

Some other hundred of projects are relying on Andrew Dinn's project
Byteman for testing, they have also been concerned but didn't back
Dinn's proposals explicitly on this list as we feel he represents the
interests of all such users. Not sure if you want us to organise a
storm of "+1" emails as I'm confident that reason will win over
spamming strategies and populism (or should I say "modern democracy").

I can't say what other OSGi frameworks are doing, but for example
testing Hibernate in OSGi requires Apache Karaf, which is stuck on
issues like:
- https://issues.apache.org/jira/browse/KARAF-3518

(I've talked again with some friends in the Karaf team last week and
they will try again now, but you get the point that people are waiting
rather than reporting here)

Clearly some projects are waiting for alternatives and answers to be
provided before proposing their own solutions. As an example,
personally I'm on the Hibernate team and we're very interested to see
how reflective access is going to work, but since Jason Greene
summarised our concerns very well already on a different thread, I'll
just let you know that we stand behind this and feel no need to add
further details.. I hope it's clear though that such things need to be
answered before we can decide how to test things further, and I
wouldn't expect large numbers of complaints from all other projects:
as these platforms are waiting for answers to very concrete issues,
there's a bottleneck and you probably won't see feedback on the
smaller issues either, as there's no much progress in testing until
Jigsaw brings some clarity and concrete solutions materialise.

I hope this helps understanding why such issues have "barely come up
to date". That doesn't make them less of a blocker issue though.

Thanks,
Sanne

-Alan.

[dalibor topic]

Post by Sanne Grinovero
So what happens is that such projects go to communities like WildFly
and OSGi for advice, where the typical answer is among the lines "I
know, we're discussing such things on the jigsaw-dev list, watch
thread XY, or watch #MyIssueCategory on Mark's blogs".

Please also ask them to the Quality Outreach effort, as well and send
them my or Rory's way.

Post by Sanne Grinovero
summarising feedback from hundreds of projects there. The ones who
care are lurking on this list hoping to see answers for issues which
have been raised already, but not many feel competent enough to join
the conversation directly (and this might be a good thing!).

(Breaking the fourth wall)

If you're lurking on this list, are a lead developer on a popular open
source project, and aren't listed on
https://wiki.openjdk.java.net/display/quality/Quality+Outreach yet,
please get in touch with me to get listed there and start testing your
project with JDK 9 Early Access builds, letting us know about things
that don't work as they should.

Post by Sanne Grinovero
interests of all such users. Not sure if you want us to organise a
storm of "+1" emails as I'm confident that reason will win over
spamming strategies and populism (or should I say "modern democracy").

I don't think that would be a productive use of anyone's resources.

Post by Sanne Grinovero
I can't say what other OSGi frameworks are doing, but for example
testing Hibernate in OSGi requires Apache Karaf, which is stuck on
- https://issues.apache.org/jira/browse/KARAF-3518

Judging by http://markmail.org/message/6v3kiqabebwtuwpv they use the
endorsed standards mechanism to override non-endorsed classes. That's
not how the Java Endorsed Standards Override Mechanism
works - please see
http://docs.oracle.com/javase/8/docs/technotes/guides/standards/ for
details.

A large part of what we do initially within the Quality Outreach effort
is to help open source projects understand problems with their own code
and practices that can adversely affect their ability to migrate to new
JDK (update) releases as they start testing against JDK 9 and discover
one or more issues with their code.

cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Jochen Theodorou]

On 15.07.2016 14:55, dalibor topic wrote:
[...]

Post by dalibor topic
A large part of what we do initially within the Quality Outreach effort
is to help open source projects understand problems with their own code
and practices that can adversely affect their ability to migrate to new
JDK (update) releases as they start testing against JDK 9 and discover
one or more issues with their code.

And if adopting the more correct practices requires them to make a huge
breaking change? Maybe even getting rid of base principles?

bye Jochen

[dalibor topic]

Post by Jochen Theodorou
[...]
And if adopting the more correct practices requires them to make a huge
breaking change? Maybe even getting rid of base principles?

Commitment bias to accumulated technical debt is completely
understandable. Unfortunately, it's also a bad guiding principle, as the
nice picture of "shoveling forward" in
http://ferd.ca/on-technical-debt-shoveling-forward.html illustrates.

cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Jochen Theodorou]

Post by dalibor topic

Post by Jochen Theodorou
[...]
And if adopting the more correct practices requires them to make a huge
breaking change? Maybe even getting rid of base principles?

Commitment bias to accumulated technical debt is completely
understandable. Unfortunately, it's also a bad guiding principle, as the
nice picture of "shoveling forward" in
http://ferd.ca/on-technical-debt-shoveling-forward.html illustrates.

You seem to be talking about achieving a certain goal and doing that
using the wrong mechanisms... What about features that exist only
because you have been able to trick the API or the JVM?

Just as an example... because the JVM used to have a URLClassloader when
starting, it was possible to add classes there at runtime and @Grab was
born to make use of that. Now this is no longer possible, @Grab will no
longer work on JDK9 in many many cases. Why is that technical debt?
There is simply no way to make this work like before anymore.

Or another example... back in early Java8, when the Verifier was changed
to be more strict and broke about every single Groovy program out there,
that did have overloaded constructors in the super class... if that
change had not been undone, this would have been the end of Groovy,
since there is no alternative way. And the verification was perfectly
valid for Java. So is that technical debt?

bye Jochen

[Alan Bateman]

Post by Jochen Theodorou
Just as an example... because the JVM used to have a URLClassloader
when starting, it was possible to add classes there at runtime and
@Grab was born to make use of that. Now this is no longer possible,
@Grab will no longer work on JDK9 in many many cases. Why is that
technical debt? There is simply no way to make this work like before
anymore.

I realize this is off-topic for the thread but just to say that the JDK
has never had (aside from the APIs in JVM TI and java.lang.instrument) a
supported/documented way to do this. You are right that some tools and
libraries have restored to a bit of hackery, specifically casting the
system class loader to a URLClassLoader and hacking into its protected
addURL method but that is making an implementation assumption. We know
at least some has been to work around command line length limits -
Gradle comes to mind but then have switched to using @argfile. The more
general "I want to extend the class path at runtime" is probably
something that should be discussed (on another thread).

Separately, is @Grab partly to workaround issues with @CS methods? The
issue with JDBC using @CS methods is one that you brought up on another
thread and there is work in progress to fix that long standing issue.

-Alan

[1] https://issues.gradle.org/browse/GRADLE-3287

[Jochen Theodorou]

On 16.07.2016 10:02, Alan Bateman wrote:
[...]

Post by Alan Bateman
thread and there is work in progress to fix that long standing issue.

The JDBC issue is one problem. But the usage is more general: to have a
script with only Groovy as dependency and loading other libraries at
runtime, without using module path or classpath.

With a script in source there should be no problem. But if the script is
precompiled, then we may have no classloader we can use to add the paths
to. So instead of just being able to distribute a small jar with just
your application classes in it and having a dependency on the minimal
groovy runtime, you are now forced to choose between

(a) requiring a groovy installation, unpacking the jar, having the entry
point in source and the user using the groovy command to start the
application.
(b) have the entry script load another entry script, loaded in a new
URLClassloader loader and hope the setup will be done right

both ways are inconvenient. (a) for the user of the application, (b) for
the writer of the application... plus of course the risk of a wrong
class loader setup. The solution will probably be a helper method to
load the second class, which does the class loader setup. This is still
no good solution, since it still has more error sources than it was the
case before, but I have no idea so far for a better one

The point is, that the mentioned failure will continue being a failure
and old code depending on this "hack" will no longer work. And we cannot
solve it by just changing things in our code, we will have to depend on
the users of Groovy to fix this. Which for us means a breaking change.

bye Jochen

[dalibor topic]

Post by Jochen Theodorou
You seem to be talking about achieving a certain goal and doing that
using the wrong mechanisms... What about features that exist only
because you have been able to trick the API or the JVM?

In that case, I'd suggest first working on bringing the necessary
features into the missing platform layers, and once they are available,
migrating to them.

Post by Jochen Theodorou
Just as an example... because the JVM used to have a URLClassloader when
longer work on JDK9 in many many cases. Why is that technical debt?

When someone deliberately breaks out of the boundary of what's supported
functionality in a platform, they consciously trade off the benefits of
that action against the risk of their code falling apart further down
the road.

In other words, the benefit needs to be sufficiently advantageous to
allow them to invest a part of its gains in eliminating the problem
before the risk eventually materializes, for example by working on
introducing the feature in a supported way at the appropriate layer of
the platform, and then migrating to it.

If the benefit isn't sufficiently advantageous for that, or if they fail
to plan adequately to eliminate the risk, then they might have a problem
in some version of the future. But the core problem is not that a risk
they consciously undertook eventually materializes, it's with risk
assessment and decision making - and it's an entirely human thing to not
get quite right.

A nice, long read on that from a security perspective is here:
https://www.schneier.com/essays/archives/2008/01/the_psychology_of_se.html

In open source ecosystems, there is often an additional human factor -
people responsible for making the trade offs may be long gone from a
project (as well as the associated benefits), by the time the risk they
consciously introduced into a piece of software materializes.

So tools for conscious risk (re-)assessment are important, such as
jdeps, or -XX:+CheckEndorsedAndExtDirs, ... as is empathy.

cheers,
dalibor topic
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Alan Bateman]

Post by dalibor topic

Post by Sanne Grinovero
I can't say what other OSGi frameworks are doing, but for example
testing Hibernate in OSGi requires Apache Karaf, which is stuck on
- https://issues.apache.org/jira/browse/KARAF-3518

Judging by http://markmail.org/message/6v3kiqabebwtuwpv they use the
endorsed standards mechanism to override non-endorsed classes.

If I read this correctly then they are attempting to override
java.lang.Exception for some reason. That "worked" because the legacy
endorsed standard override mechanism was essentially -Xbootclasspath/p.

For debugging purposes then way to override a class in a module is
-Xpatch (or --patch-module with the new option). In this case, it would
be `--patch-module java.base=myexception.jar` although I don't expect it
will work (at either compile-time or run-time) because code in java.base
can't link to code in java.xml.bind (JAXB). They will need
`--add-modules java.xml.bind --add-reads java.base=java.xml.bind` and
additionally -Xmodule:java.base at compile-time.

As regards the endorsed standard override mechanism then it has been
removed in JDK 9. It was deprecated in a MR of JSR 337. Also
-XX:+CheckEndorsedAndExtDirs is documented in the release notes for JDK
8 updates as a quick way to have the VM fail at startup if you are
(unknowing) using this or the legacy extension mechanism.

-Alan

[Alan Bateman]

Post by Sanne Grinovero
Hi Alan, you're spot-on about JEP-220: myself and my colleagues have
sent patches to numerous projects to which I normally don't contribute
to, just to fix our dependencies, such as Maven.

Good to hear this.

Post by Sanne Grinovero
But please be aware of how many of the OSS projects in the Java
community are organised: some will have integration tests running in
modular environments, like WildFly which is based on David Lloyd's
JBoss Modules, or for OSGi tests it seems popular to use Apache Karaf.
Many others don't have any "modular" or "container" integration tests
and are not aware of what's coming.. since in my team we package some
of these for usage in modular environments, I'm painfully aware of
problems coming but it's no easy task to convince them for the need to
test now.

I understand, we've found it hard in the past to get projects to test
too. However, for JDK 9 then I see a lot more projects engaged and
actively testing. As we've been saying at conferences for several years
now, the changes in JDK 9 have a huge impact on the eco system, this
update is not a JDK-only update.

Post by Sanne Grinovero
When I get in touch to try convey a sense of urgent need to test
Jigsaw, and sometimes successfully, then people realise that their
whole toolchain, not least the testing environment, blows up with
errors which are often too complex or simply too unrelated to the
project itself.

There are folks working on several Oracle's products trying out JDK 9
builds too. A lot of issues that I hear about tend to be somewhat
mundane but they are some complicated issues too - particularly around
deploying upgraded versions of the components that overlap with what is
the JDK (the EE modules and CORBA mostly).

Post by Sanne Grinovero
So what happens is that such projects go to communities like WildFly
and OSGi for advice, where the typical answer is among the lines "I
know, we're discussing such things on the jigsaw-dev list, watch
thread XY, or watch #MyIssueCategory on Mark's blogs".
In short, please be aware that David Lloyd isn't making such
suggestions just for WildFly itself, but is pre-processing and
summarising feedback from hundreds of projects there. The ones who
care are lurking on this list hoping to see answers for issues which
have been raised already, but not many feel competent enough to join
the conversation directly (and this might be a good thing!).

Do you know if these hundreds of projects are really trying out modules?

-Alan

[Sanne Grinovero]

Post by Alan Bateman

Post by Sanne Grinovero
Hi Alan, you're spot-on about JEP-220: myself and my colleagues have
sent patches to numerous projects to which I normally don't contribute
to, just to fix our dependencies, such as Maven.

Good to hear this.

Post by Sanne Grinovero
But please be aware of how many of the OSS projects in the Java
community are organised: some will have integration tests running in
modular environments, like WildFly which is based on David Lloyd's
JBoss Modules, or for OSGi tests it seems popular to use Apache Karaf.
Many others don't have any "modular" or "container" integration tests
and are not aware of what's coming.. since in my team we package some
of these for usage in modular environments, I'm painfully aware of
problems coming but it's no easy task to convince them for the need to
test now.

I understand, we've found it hard in the past to get projects to test too.
However, for JDK 9 then I see a lot more projects engaged and actively
testing. As we've been saying at conferences for several years now, the
changes in JDK 9 have a huge impact on the eco system, this update is not a
JDK-only update.

Post by Sanne Grinovero
When I get in touch to try convey a sense of urgent need to test
Jigsaw, and sometimes successfully, then people realise that their
whole toolchain, not least the testing environment, blows up with
errors which are often too complex or simply too unrelated to the
project itself.

There are folks working on several Oracle's products trying out JDK 9 builds
too. A lot of issues that I hear about tend to be somewhat mundane but they
are some complicated issues too - particularly around deploying upgraded
versions of the components that overlap with what is the JDK (the EE modules
and CORBA mostly).

Post by Sanne Grinovero
So what happens is that such projects go to communities like WildFly
and OSGi for advice, where the typical answer is among the lines "I
know, we're discussing such things on the jigsaw-dev list, watch
thread XY, or watch #MyIssueCategory on Mark's blogs".
In short, please be aware that David Lloyd isn't making such
suggestions just for WildFly itself, but is pre-processing and
summarising feedback from hundreds of projects there. The ones who
care are lurking on this list hoping to see answers for issues which
have been raised already, but not many feel competent enough to join
the conversation directly (and this might be a good thing!).

Do you know if these hundreds of projects are really trying out modules?

The only one I know of is Hibernate Validator, Gunnar Morling has been
trying it:
- http://in.relation.to/2016/06/17/bean-validation-and-the-jigsaw-liaison/

I'm not sure about all those other projects though, I personally deal
with only a fraction of them: we interact across projects on a need-be
case. (While Gunnar is on my same team).

My point was just that many of those developers - including often
myself - would reach out to David first rather than to this list, when
needing advice about modularity and classloading issues.

Thanks,
Sanne

Post by Alan Bateman
-Alan

[John Rose]

Post by Remi Forax
First, there is already in the Java ecosystem a notion of non-exported package, packages startings with com.sun or packages containing internal, it was just a convention and not something enforced by the VM. What the JPMS spec does is just to normalize how to declare an exported package and mandate that the VM check this new rule.
Sure it means that public classes are not accessible/visible by everybody anymore, but a class like sun.misc.Unsafe was never really accessible by everybody despite being declared public.
Furthermore, declaring if something is exported or not at class level instead of at package level seems wrong to me, usually, several classes works together for a purpose and you want these classes to be exported or not, so it's not something that should be decided at class level.
So i see the JPMS spec conept of non-exported package as a standardization of an existing practice not something new that people will have trouble to understand and reason about.

My $0.02:

As an author of Unsafe I agree enthusiastically with this viewpoint. (Well stated, Remi.) System assemblies (not individual classes or packages) need to isolate portion A of the global package namespace from portion B.

For me, declaring such assembly instructions inside class definitions would be a scale error. The access control in question is not a relation between individual classes; it's a relation between larger entities: packages.

It's a relief to see the old ad hoc package-scale hiding mechanisms (restricted packages, shared secrets, providers, class loaders, etc., etc.) getting supplanted by a first-class formalism. It's also scary to see the amount of work involved, but it's worth it.

— John

[David M. Lloyd]

Post by John Rose

Post by Remi Forax
First, there is already in the Java ecosystem a notion of non-exported
package, packages startings with com.sun or packages containing
internal, it was just a convention and not something enforced by the
VM. What the JPMS spec does is just to normalize how to declare an
exported package and mandate that the VM check this new rule.
Sure it means that public classes are not accessible/visible by
everybody anymore, but a class like sun.misc.Unsafe was never really
accessible by everybody despite being declared public.
Furthermore, declaring if something is exported or not at class level
instead of at package level seems wrong to me, usually, several
classes works together for a purpose and you want these classes to be
exported or not, so it's not something that should be decided at class
level.
So i see the JPMS spec conept of non-exported package as a
standardization of an existing practice not something new that people
will have trouble to understand and reason about.

As an author of Unsafe I agree enthusiastically with this viewpoint.
(Well stated, Remi.) System assemblies (not individual classes or
packages) need to isolate portion A of the global package namespace from
portion B.

I agree, though I disagree with the concept of a "global" package
namespace, but that's a different discussion.

Post by John Rose
For me, declaring such assembly instructions inside class definitions
would be a scale error. The access control in question is not a
relation between individual classes; it's a relation between larger
entities: packages.

Absolutely, and I want to re-emphasize that I do not advocate for
changing anything inside class definitions in any way compared to what
is in Java 8. That is in fact the whole point of my proposal: do not
change anything over Java 8; just add a new capability which can be used
to improve security, and which would then subsequently be used within
the JDK to close the various problems which exist due to internal
classes being public.

Post by John Rose
It's a relief to see the old ad hoc package-scale hiding mechanisms
(restricted packages, shared secrets, providers, class loaders, etc.,
etc.) getting supplanted by a first-class formalism. It's also scary to
see the amount of work involved, but it's worth it.

Sure, and I don't believe my proposal deviates all that far from the
existing work, but it does solve the reflective access problems that
everyone keeps hitting, which arise from the redefinition of public to
relate to exports.

[Sanne Grinovero]

Post by Remi Forax
----- Mail original -----

Envoyé: Lundi 11 Juillet 2016 16:21:46
Objet: It's not too late for access control
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear. The
word "public" literally means "accessible to all" after all; that's why
the term was selected in the first place, and it is nearly certain that
this was the intent of the thing up to this date.
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list and
in this email, but that even now, it's not too late to change the approach.
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. Users would make public any type or member which is
*meant* to be public, i.e. accessible by all. Rather than (at best)
changing their expectations as to the behavior of "public" only to
immediately betray that expectation by forcing them through a backdoor
in order to meet practical needs, we ensure that their expectations
remain: public members are public, and things that are secret are not
public.
Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?
--
- DML

Hi David,
package private means package private :)

I'm sorry but I don't think this is valid, as there is no such explicit keyword.

The source code that people will be reading doesn't explicitly state
"package private" at all, there just is a notion of default
visibility, and those rules are always a bit confusing to newcomers of
the language. Please don't look at this issue just with the eyes of an
expert: I assume we're all familiar with such peculiarities here, but
that doesn't make it any less weird to newcomers.

Jigsaw is already changing the rules about visibility, so having to
change textbooks and update people on what "default visibility" means
is a minor concern.

On the other hand, having an explicit "public" keyword which doesn't
mean quite what it should, is counter intuitive and doesn't help with
the long term consistency of the language.

I think David's proposal is very sensible, and indeed I also believe
that while we need modularity, this doesn't have to be at the cost of
screwing up the notion of "public".

"public means public" ;)

Regards,
Sanne

Post by Remi Forax
One early design idea of jigsaw was to introduce a new modifier "module" with a visibility in between public and package private.
It's a bad idea !
First, there is already in the Java ecosystem a notion of non-exported package, packages startings with com.sun or packages containing internal, it was just a convention and not something enforced by the VM. What the JPMS spec does is just to normalize how to declare an exported package and mandate that the VM check this new rule.
Sure it means that public classes are not accessible/visible by everybody anymore, but a class like sun.misc.Unsafe was never really accessible by everybody despite being declared public.
Furthermore, declaring if something is exported or not at class level instead of at package level seems wrong to me, usually, several classes works together for a purpose and you want these classes to be exported or not, so it's not something that should be decided at class level.
So i see the JPMS spec conept of non-exported package as a standardization of an existing practice not something new that people will have trouble to understand and reason about.
regards,
Rémi

[Alan Bateman]

Post by David M. Lloyd
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear.
The word "public" literally means "accessible to all" after all;
that's why the term was selected in the first place, and it is nearly
certain that this was the intent of the thing up to this date.
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list
and in this email, but that even now, it's not too late to change the
approach.
I propose, once again, that rather than changing the meaning of
"public" to something unintuitive (and indeed counter to the
definition of the actual word), we instead allow the selective
extension of package-private. Users would make public any type or
member which is *meant* to be public, i.e. accessible by all. Rather
than (at best) changing their expectations as to the behavior of
"public" only to immediately betray that expectation by forcing them
through a backdoor in order to meet practical needs, we ensure that
their expectations remain: public members are public, and things that
are secret are not public.
Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?

It's hard to know where to start on this as what you proposing would
have massive implications. I think what you are proposing is to drop the
notion of exporting packages. You didn't mention the dependency module
or the concept of readability - are you suggesting to drop that too? I
can't tell if you've grokked that this is the basis for reliable
configuration. You haven't said how I could release a library that works
on the class path in some environments and as a module in other
environments. I don't see how you proposing to do the equivalent of
qualified exports. I could go on, smarter much people than me could go
further, but I'm not sure that it's worth spending time on now.

At a high level then I think it's important to understand that a lot of
time has been put into the design. The proposal in the JPMS has to take
account that the core platform is very mature. This means compatibility,
migration and co-existence with the existing world significantly
influences the design. I have no doubt that if the modules were in the
Java Language from the start or adding in the early days then the design
would be different. As regards changing `public` so that it no longer
implies "accessible to all" then it is unfortunate but it's a change
that you should be able to get used too quickly.

-Alan

[David M. Lloyd]

Post by Alan Bateman

Post by David M. Lloyd
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear.
The word "public" literally means "accessible to all" after all;
that's why the term was selected in the first place, and it is nearly
certain that this was the intent of the thing up to this date.
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list
and in this email, but that even now, it's not too late to change the
approach.
I propose, once again, that rather than changing the meaning of
"public" to something unintuitive (and indeed counter to the
definition of the actual word), we instead allow the selective
extension of package-private. Users would make public any type or
member which is *meant* to be public, i.e. accessible by all. Rather
than (at best) changing their expectations as to the behavior of
"public" only to immediately betray that expectation by forcing them
through a backdoor in order to meet practical needs, we ensure that
their expectations remain: public members are public, and things that
are secret are not public.
Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control. Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?

It's hard to know where to start on this as what you proposing would
have massive implications. I think what you are proposing is to drop the
notion of exporting packages. You didn't mention the dependency module
or the concept of readability - are you suggesting to drop that too?

No, exporting and dependencies would remain. The concept of readability
would go, I think, as it wouldn't add anything (and I'm really not sure
it adds anything today, as the "direction" of the readability grant
seems backwards to me in terms of security).

Post by Alan Bateman
I can't tell if you've grokked that this is the basis for reliable
configuration.

Module linkage can still be reliable using only an exports graph, and
keeping public as public. We've been doing it for years.

Post by Alan Bateman
You haven't said how I could release a library that works
on the class path in some environments and as a module in other
environments.

You would not need to make any changes to most libraries. If you wanted
to export package A to package B in a non-module environment, and if we
wanted to support that, I don't see it being any more difficult than,
say, supporting *both* module dependencies *and* Class-Path.

Post by Alan Bateman
I don't see how you proposing to do the equivalent of
qualified exports.

It wouldn't have to be any different; instead of re-enabling public
access to a dependency, you'd have the option to enable private access,
thus ensuring that security expectations are consistent: accessibility
is never weaker than a module author expects.

Post by Alan Bateman
I could go on, smarter much people than me could go
further, but I'm not sure that it's worth spending time on now.
At a high level then I think it's important to understand that a lot of
time has been put into the design.

I have sympathy for this, and I also understand (as well as anyone,
trust me) that it is possible to spend a lot of time thinking about a
design only to find a problem at the end. The amount of time spent on a
design is no proof against changing it.

Post by Alan Bateman
The proposal in the JPMS has to take
account that the core platform is very mature. This means compatibility,
migration and co-existence with the existing world significantly
influences the design.

Of course! That's the main point of this: the export access controls
are so completely incompatible that just about every reflective
framework - and there are many, many of them (a couple searches on
GitHub yield hundreds of thousands to millions of Java code results) -
will break. Every single one of these bits of code are going to assume
that a public member in a public type can be publicly accessed.
Compatibility means public stays public. This is the "real" world.

Post by Alan Bateman
I have no doubt that if the modules were in the
Java Language from the start or adding in the early days then the design
would be different. As regards changing `public` so that it no longer
implies "accessible to all" then it is unfortunate but it's a change
that you should be able to get used too quickly.

How can one massive compatibility change be "unfortunate" while another
is otherwise somehow invalid?

[Eric Johnson]

Post by David M. Lloyd
The crux of this access control discussion is that, up until JDK 9,
"public" meant "public". End of story. If you did not want something
to be visible, you made it not public. Very simple and very clear.
The word "public" literally means "accessible to all" after all;
that's why the term was selected in the first place, and it is nearly
certain that this was the intent of the thing up to this date.

To be fair, OSGi established a precedent - a "bundle" can make a
distinction between that which is public and that which is "exported".
(I think of this as the distinction between public vs. published,
because I like the alliteration).

Mostly, I agree with your point, though! OSGi makes this distinction,
because it is worried about having multiple versions of the same package
available at run-time, and even that bundle A can have a complete copy
of JAR B - from a completely different project - and not want to take
responsibility for the "public" version of the API of JAR B. So OSGi
explicitly needs to be able to hide ostensibly public APIs, because
author A cannot possibly correctly guess the intent of author B in all
cases. However, Jigsaw is explicitly not concerned about this problem,
so the extra distinction seems spurious to me.

Post by David M. Lloyd
What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.
The existing Java language accessibility model is (of course)
well-understood by experts. For new developers, the concepts takes a
bit of time to explain but can generally be grasped. In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.
What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking. Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed. I argue that this
approach is not optimal, for at least reasons discussed on this list
and in this email, but that even now, it's not too late to change the
approach.

What infuriates me is that in all this discussion, I don't see anyone
talking about a threat analysis. What are we trying to protect, from
whom, and why? I see comments about how implementation details of the
JRE (such as "com.sun" packages) must be hidden, but without reference
to the threats that cause a problem.

I've seen enough security issues about Java to believe that there /are/
threats that Oracle should be concerned about. And I'm happy to have
Oracle hide ostensibly public classes in the JRE at runtime, because I
want the JRE to have a smaller attack surface.

At compile time, project Jigsaw makes total sense to me. A modularized
Java makes sense, both because it highlights an excessive dependency
tree, and because it can help reduce the footprint of the code. In other
words, it isn't a security measure, it is a design tool. At runtime,
though, does the JVM benefit from something more complex than a simply
layering model? Layer 0 - the boot system, Layer 1 - the JRE, Layer 2-N
- defined by the developer. I'm guessing in the server model, Layer 2 is
the "container" (Servlet, Spring, etc.), and Layer 3 is the "application".

However, I'm unable to assess whether my proposed JVM model meets the
security requirements. Why? Because I've never seen the threat analysis
spelled out or referred to on this list. Maybe such a document exists?
With the simple layered model, almost all applications (except those
that load JRE internal details either via reflection or direct linking)
would continue to work as is. With the Jigsaw model, I feel like I have
to assume everything will break, unless otherwise fixed.

Hoping something about this will change.

Post by David M. Lloyd
I propose, once again, that rather than changing the meaning of
"public" to something unintuitive (and indeed counter to the
definition of the actual word), we instead allow the selective
extension of package-private. Users would make public any type or
member which is *meant* to be public, i.e. accessible by all. Rather
than (at best) changing their expectations as to the behavior of
"public" only to immediately betray that expectation by forcing them
through a backdoor in order to meet practical needs, we ensure that
their expectations remain: public members are public, and things that
are secret are not public.

I think this is an interesting idea to consider. Except again, I'm
unclear on what we're trying to protect from whom, and why? So I don't
know if this solves the problem.

Eric

[Russell Gold]

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

[Eric Johnson]

At least someone replied to my question.

Post by Russell Gold

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

If the entire aim is just a maintenance issue, then I assert that
compiler changes should be sufficient. At that point, anyone who fires
up runtime reflection to work around the compiler gets the benefits and
the costs that they deserve. Benefits include the ability to leverage
code in the guts of any library or the JRE that the authors decided they
didn't want to publish. And they embrace the likelihood that it will
break in the future.

I don't feel a huge need to argue about language design. Other people
can battle over the contentious issues of what is appropriate for
language constructs. That could be module boundaries with metadata, it
could be new keywords like "friend", or even package level declarations
(more like OSGi). I suspect we'll all adopt to the "best" solution
determined by those who want to improve the language.

To make this more concrete, Scala could introduce some kind of scoping
mechanism completely unfamiliar to existing Java developers. Few people,
if any, would be upset that it was merely a compile-time enforcement,
and the runtime didn't enforce the new capability. Same should be true
of Java itself. For one possible framing, perhaps I ought to be able to
compile a module-based application and yet run it on Java 8 without
modification?

So yes, modules for the compiler, if that's what Oracle thinks is best
for the language, I can live with it. Need not affect the runtime, though!

However, like I said in my original email, I believe genuine security
issues can be found in this quicksand. The JRE has valid reasons to
prevent access to implementation details. That reduces the possible
attack footprint, and also strengthens the walls of the "sandbox" that
Java puts up (more code to tunnel through before getting to the
vulnerable chunk). And it seems pretty clear, from the CVEs found
<https://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-19117/Oracle-JRE.html>
in the JRE, that these changes should happen regardless of whether or
not the runtime has a security manager installed, because it is common
enough to run without one. To pick an example from another language, you
can't run Ruby without having "taint" support on.

In other words, the Jigsaw runtime change to enforce module boundaries
amounts to /always/ running with some sort of security manager. If you
frame the implementation that way, though, then it is clear that
alternate implementations might be quite valuable - especially in a
transition phase from prior versions of Java.

While this discussion is ostensibly about reflection and accessibility,
I'm viewing it as really about what kind of security manager is the
minimum allowed at runtime.

I applaud Oracle's effort to tackle the unpleasant task of imposing (on
us) a mandatory minimum security monitor on the JRE classes. However, to
date, Jigsaw is imposing one specific model of mandatory minimum
security on all of us, without entertaining alternate solutions. That's
what I find enormously frustrating. It is pretty clear to me from the
discussions on this mailing list that the one they've enforced so far is
the wrong one, at least if Java 9 is going to be adopted widely.

[Gregg Wonderly]

Post by Eric Johnson
At least someone replied to my question.

Post by Russell Gold

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

If the entire aim is just a maintenance issue, then I assert that compiler changes should be sufficient. At that point, anyone who fires up runtime reflection to work around the compiler gets the benefits and the costs that they deserve. Benefits include the ability to leverage code in the guts of any library or the JRE that the authors decided they didn't want to publish. And they embrace the likelihood that it will break in the future.

More rigorous versioning could help mediate this so that when my introspecting code which works on JDK7 is deployed on JDK8, it would not run if I declared a version dependency on JDK7. We currently have versioning that works with newer JDKs by default, without the ability to declare that only one JDK works.

Believe me, I understand how ugly many people in the JDK development team feel it is for all this reflection to be done, and how responsible they seem to feel, given the direction JigSaw is going, to try and keep people from “breaking” their own software. But practically, they are taking that risk. Leave them in the dust by ignoring their dependencies. But, don’t shut the door on long term solutions evolving from that experimentation/introspection which allows the platform to become better because they help you understand that an API change is actually in order because there is a use case that was missed in the original JDK design.

What will happen if JigSaw is deployed as it is today, is that people will elect to use something else. They literally will run as fast as they can to a “place” where participation is in their control, not some outside party.

Think of the Boston Tea Party. From Hamilton lyrics, “Why should some tiny island on the other side of the world set the price of tea?” Making the JDK seem like a King of overlord with all kinds of limits applied will make people really mad. Also from Hamilton, “I’ll kill your family to remind you of my love” is sort of how people look at this kind of “control” and “overreach” of responsibility in software. It’s like JDK-9 shuts down all kinds of responsibilities and totally disrupts interactions that used to feel friendly and inviting and instead makes them feel like a teach standing over you with a ruler, smacking your hand every time you try and touch something they don’t want you to.

Gregg

[Claes Redestad]

Post by Gregg Wonderly

Post by Eric Johnson
At least someone replied to my question.

Post by Russell Gold

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

If the entire aim is just a maintenance issue, then I assert that compiler changes should be sufficient. At that point, anyone who fires up runtime reflection to work around the compiler gets the benefits and the costs that they deserve. Benefits include the ability to leverage code in the guts of any library or the JRE that the authors decided they didn't want to publish. And they embrace the likelihood that it will break in the future.

More rigorous versioning could help mediate this so that when my introspecting code which works on JDK7 is deployed on JDK8, it would not run if I declared a version dependency on JDK7. We currently have versioning that works with newer JDKs by default, without the ability to declare that only one JDK works.

http://openjdk.java.net/jeps/238 allows graceful transition from
introspecting code and other hacks to supported APIs in 9, 10 etc.

Post by Gregg Wonderly
Believe me, I understand how ugly many people in the JDK development team feel it is for all this reflection to be done, and how responsible they seem to feel, given the direction JigSaw is going, to try and keep people from “breaking” their own software. But practically, they are taking that risk. Leave them in the dust by ignoring their dependencies. But, don’t shut the door on long term solutions evolving from that experimentation/introspection which allows the platform to become better because they help you understand that an API change is actually in order because there is a use case that was missed in the original JDK design.
What will happen if JigSaw is deployed as it is today, is that people will elect to use something else. They literally will run as fast as they can to a “place” where participation is in their control, not some outside party.
Think of the Boston Tea Party. From Hamilton lyrics, “Why should some tiny island on the other side of the world set the price of tea?” Making the JDK seem like a King of overlord with all kinds of limits applied will make people really mad. Also from Hamilton, “I’ll kill your family to remind you of my love” is sort of how people look at this kind of “control” and “overreach” of responsibility in software. It’s like JDK-9 shuts down all kinds of responsibilities and totally disrupts interactions that used to feel friendly and inviting and instead makes them feel like a teach standing over you with a ruler, smacking your hand every time you try and touch something they don’t want you to.

Sorry if this sounds rude, but it seems like you're all arguing some
straw man idea of jigsaw rather than the actual implementation,
in addition to going slightly off-topic.

It's still perfectly possible to break through to non-exported
packages: it will simply require use of either new flags
(addExports/addReads, upgraded modules etc) or use of the newly added
extensions to the reflection API. And as has been stated again and again
elsewhere: setAccessible and friends are not going anywhere soon.

Rather than an unfriendly or even despotic attempt to wrest control
from free developer souls, I find it quite agreeable to give
module/library/JDK authors a stronger way of saying "this is
internal, please keep out". Unless there is a security manager
prohibiting you, but hey, no change there.

What will break is the ability for libraries to go ahead and do this
without explicit approval from the "deployer", application owner, user.

Currently plenty of libraries take these freedoms on the behalf of that
guy or gal, who might not even be aware of the library authors decision
to cut corners. Some of these people might not think it's worth the
cost or risk of breaking future compatibility for some convenience, a
few points on a benchmark or whatever other reason.

As we're being emotional: I feel good about the way jigsaw brings
control back to the application owner to do more informed decisions and
to library authors to be more confident about evolving their code, and
hope that someday you will feel that there's actually a big deal of
common sense invested into jigsaw.

Now, can we go back to arguing about public vs package private?

/Claes

[Jason T. Greene]

Claes,

I ask that you take a look at the extensive arguments I have presented on the topic of reflection. The assumption you seem to make is that the use case of reflective access to internal packages is wrong, poor programming practice, or an error.

That couldn't be further from the truth. The use cases are widely adopted by the industry, and they enable rich and powerful programming models that developers depend on.

I was personally hoping to be able to promote the jigsaw model to developers, but the technology as it is currently defined is insufficient to meet the demands of today, let alone tomorrow. I am still holding out hope this will be addressed,l.

Post by Claes Redestad

Post by Gregg Wonderly

Post by Eric Johnson
At least someone replied to my question.

Post by Russell Gold

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

If the entire aim is just a maintenance issue, then I assert that compiler changes should be sufficient. At that point, anyone who fires up runtime reflection to work around the compiler gets the benefits and the costs that they deserve. Benefits include the ability to leverage code in the guts of any library or the JRE that the authors decided they didn't want to publish. And they embrace the likelihood that it will break in the future.

More rigorous versioning could help mediate this so that when my introspecting code which works on JDK7 is deployed on JDK8, it would not run if I declared a version dependency on JDK7. We currently have versioning that works with newer JDKs by default, without the ability to declare that only one JDK works.

http://openjdk.java.net/jeps/238 allows graceful transition from
introspecting code and other hacks to supported APIs in 9, 10 etc.

Post by Gregg Wonderly
Believe me, I understand how ugly many people in the JDK development team feel it is for all this reflection to be done, and how responsible they seem to feel, given the direction JigSaw is going, to try and keep people from “breaking” their own software. But practically, they are taking that risk. Leave them in the dust by ignoring their dependencies. But, don’t shut the door on long term solutions evolving from that experimentation/introspection which allows the platform to become better because they help you understand that an API change is actually in order because there is a use case that was missed in the original JDK design.
What will happen if JigSaw is deployed as it is today, is that people will elect to use something else. They literally will run as fast as they can to a “place” where participation is in their control, not some outside party.
Think of the Boston Tea Party. From Hamilton lyrics, “Why should some tiny island on the other side of the world set the price of tea?” Making the JDK seem like a King of overlord with all kinds of limits applied will make people really mad. Also from Hamilton, “I’ll kill your family to remind you of my love” is sort of how people look at this kind of “control” and “overreach” of responsibility in software. It’s like JDK-9 shuts down all kinds of responsibilities and totally disrupts interactions that used to feel friendly and inviting and instead makes them feel like a teach standing over you with a ruler, smacking your hand every time you try and touch something they don’t want you to.

Sorry if this sounds rude, but it seems like you're all arguing some straw man idea of jigsaw rather than the actual implementation,
in addition to going slightly off-topic.
It's still perfectly possible to break through to non-exported
packages: it will simply require use of either new flags
(addExports/addReads, upgraded modules etc) or use of the newly added
extensions to the reflection API. And as has been stated again and again elsewhere: setAccessible and friends are not going anywhere soon.
Rather than an unfriendly or even despotic attempt to wrest control
from free developer souls, I find it quite agreeable to give
module/library/JDK authors a stronger way of saying "this is
internal, please keep out". Unless there is a security manager
prohibiting you, but hey, no change there.
What will break is the ability for libraries to go ahead and do this
without explicit approval from the "deployer", application owner, user.
Currently plenty of libraries take these freedoms on the behalf of that
guy or gal, who might not even be aware of the library authors decision
to cut corners. Some of these people might not think it's worth the
cost or risk of breaking future compatibility for some convenience, a
few points on a benchmark or whatever other reason.
As we're being emotional: I feel good about the way jigsaw brings
control back to the application owner to do more informed decisions and
to library authors to be more confident about evolving their code, and
hope that someday you will feel that there's actually a big deal of
common sense invested into jigsaw.
Now, can we go back to arguing about public vs package private?

[Claes Redestad]

Jason,

Post by Jason T. Greene
Claes,
I ask that you take a look at the extensive arguments I have presented on the topic of reflection. The assumption you seem to make is that the use case of reflective access to internal packages is wrong, poor programming practice, or an error.
That couldn't be further from the truth.

I'm refraining from such generalization - although I do take the view
that most reflective *and* static uses of JDK internals comes at a cost
and that replacing such uses with public - or at least officially
supported - APIs should be the preferred long term solution, since
it'll give the internals more leeway to change and improve.

I hope that you can somewhat agree with that, and that a platform which
is able to evolve - be it by adding, changing or even removing code -
will live a healthier life in the long term.

Post by Jason T. Greene
The use cases are widely adopted by the industry, and they enable rich and powerful programming models that developers depend on.

I assume you're referring to some specific concern here, which I hope
can be discussed more in-depth either in a new thread or where it's
already been brought up.

Post by Jason T. Greene
I was personally hoping to be able to promote the jigsaw model to developers, but the technology as it is currently defined is insufficient to meet the demands of today, let alone tomorrow. I am still holding out hope this will be addressed,l.

There are surely open questions, and I too hope they can be addressed
in a manner satisfactory to most, if not all.

/Claes

Post by Jason T. Greene

Post by Claes Redestad

Post by Gregg Wonderly

Post by Eric Johnson
At least someone replied to my question.

Post by Russell Gold

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

If the entire aim is just a maintenance issue, then I assert that compiler changes should be sufficient. At that point, anyone who fires up runtime reflection to work around the compiler gets the benefits and the costs that they deserve. Benefits include the ability to leverage code in the guts of any library or the JRE that the authors decided they didn't want to publish. And they embrace the likelihood that it will break in the future.

More rigorous versioning could help mediate this so that when my introspecting code which works on JDK7 is deployed on JDK8, it would not run if I declared a version dependency on JDK7. We currently have versioning that works with newer JDKs by default, without the ability to declare that only one JDK works.

http://openjdk.java.net/jeps/238 allows graceful transition from
introspecting code and other hacks to supported APIs in 9, 10 etc.

Post by Gregg Wonderly
Believe me, I understand how ugly many people in the JDK development team feel it is for all this reflection to be done, and how responsible they seem to feel, given the direction JigSaw is going, to try and keep people from “breaking” their own software. But practically, they are taking that risk. Leave them in the dust by ignoring their dependencies. But, don’t shut the door on long term solutions evolving from that experimentation/introspection which allows the platform to become better because they help you understand that an API change is actually in order because there is a use case that was missed in the original JDK design.
What will happen if JigSaw is deployed as it is today, is that people will elect to use something else. They literally will run as fast as they can to a “place” where participation is in their control, not some outside party.
Think of the Boston Tea Party. From Hamilton lyrics, “Why should some tiny island on the other side of the world set the price of tea?” Making the JDK seem like a King of overlord with all kinds of limits applied will make people really mad. Also from Hamilton, “I’ll kill your family to remind you of my love” is sort of how people look at this kind of “control” and “overreach” of responsibility in software. It’s like JDK-9 shuts down all kinds of responsibilities and totally disrupts interactions that used to feel friendly and inviting and instead makes them feel like a teach standing over you with a ruler, smacking your hand every time you try and touch something they don’t want you to.

Sorry if this sounds rude, but it seems like you're all arguing some straw man idea of jigsaw rather than the actual implementation,
in addition to going slightly off-topic.
It's still perfectly possible to break through to non-exported
packages: it will simply require use of either new flags
(addExports/addReads, upgraded modules etc) or use of the newly added
extensions to the reflection API. And as has been stated again and again elsewhere: setAccessible and friends are not going anywhere soon.
Rather than an unfriendly or even despotic attempt to wrest control
from free developer souls, I find it quite agreeable to give
module/library/JDK authors a stronger way of saying "this is
internal, please keep out". Unless there is a security manager
prohibiting you, but hey, no change there.
What will break is the ability for libraries to go ahead and do this
without explicit approval from the "deployer", application owner, user.
Currently plenty of libraries take these freedoms on the behalf of that
guy or gal, who might not even be aware of the library authors decision
to cut corners. Some of these people might not think it's worth the
cost or risk of breaking future compatibility for some convenience, a
few points on a benchmark or whatever other reason.
As we're being emotional: I feel good about the way jigsaw brings
control back to the application owner to do more informed decisions and
to library authors to be more confident about evolving their code, and
hope that someday you will feel that there's actually a big deal of
common sense invested into jigsaw.
Now, can we go back to arguing about public vs package private?
/Claes

[Alan Bateman]

Post by Claes Redestad

Post by Jason T. Greene
The use cases are widely adopted by the industry, and they enable
rich and powerful programming models that developers depend on.

I assume you're referring to some specific concern here, which I hope
can be discussed more in-depth either in a new thread or where it's
already been brought up.

It's the issue named #ReflectiveAccessToNonExportedTypes on the JSR
issues list. There is parallel thread discussing it, along with the
merits and issues with the `exports dynamic` proposal that Mark brought
to the EG list recently.

-Alan.

[dalibor topic]

Post by Jason T. Greene
The assumption you seem to make is that the use case of reflective access to internal packages is wrong, poor programming practice, or an error.
That couldn't be further from the truth.

As with many things, it kind of depends on who you ask:
https://www.securecoding.cert.org/confluence/display/java/SEC05-J.+Do+not+use+reflection+to+increase+accessibility+of+classes,+methods,+or+fields


There is a case to be made for experts consciously making expertly
assessments of risk and benefits of using sharp edged tools, and it
seems to have been made a few times in different forms on this thread.
There is also a case to be made that some people may unfortunately have
not made expertly assessments about using sharp edged tools in the past,
and therefore could see risks materialize which they didn't anticipate
adequately. I think that argument was made a few times in different
forms on this thread.

All of those are fine arguments, and so is the need to consciously keep
moving the set of current development practices along with evolving the
state of the art, to, in the long term, avoid a situation in which, for
example, some C/C++ users & developers find themselves in today, where
some development practices have not evolved along with the platform,
creating a growing tension [0] between users depending on large bodies
of presumably partly incorrect & unsafe [1] code, and the thrust of
development of the programming language and associated development tools
[2].

In short, let's not argue about absolute statements one way or the other
if we can avoid it.

cheers,
dalibor topic

[0]
https://groups.google.com/forum/m/#!msg/boring-crypto/48qa1kWignU/o8GGp2K1DAAJ
[1] https://twitter.com/robilad/status/754084363017465857
[2]
http://stackoverflow.com/questions/36893251/why-does-the-enhanced-gcc-6-optimizer-break-practical-c-code
--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: [phone] <tel:[phone]> | Mobile: [phone]
<tel:[phone]>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

[Andrew Dinn]

Post by dalibor topic

Post by Jason T. Greene
The assumption you seem to make is that the use case of reflective
access to internal packages is wrong, poor programming practice, or
an error.
That couldn't be further from the truth.

https://www.securecoding.cert.org/confluence/display/java/SEC05-J.+Do+not+use+reflection+to+increase+accessibility+of+classes,+methods,+or+fields
...
In short, let's not argue about absolute statements one way or the other
if we can avoid it.

If you reread Jason's statement above I think you will notice that this
is the point of his statement, to reject one such extreme. He did not
thereby recommend careless use of a potentially insecure capability.
Indeed he has taken great care to emphasise that what he wants (and what
I want) is a module system which provides a safe, controlled way of
opening up access to non-public members, retaining the opportunity to
implement the rich software tools that we currently have.

So, in sum, straw man, Dalibor.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

[Claes Redestad]

Hmmm....

Post by Claes Redestad
It's still perfectly possible to break through to non-exported
packages: it will simply require use of either new flags
(addExports/addReads, upgraded modules etc) or use of the newly added
extensions to the reflection API. And as has been stated again and
again elsewhere: setAccessible and friends are not going anywhere soon.

Let me play out an analogy: City upgrades the building codes. You as a
* house undergoes a significant, externally visible change
* buy/sell transaction
(In other words, when the owner clearly demonstrates they likely have
the financial resources to conform.)
With JDK 9, however, we all know that JDK 8 will fall out of support.
Which means that changes in JDK 9 /must/ be accommodated by any project
/ product that wishes to maintain viability past the support window for
JDK 8. Except, given the maturity of the Java ecosystem, in most cases
most projects don't want to change, or fundamentally don't need to change.
So, based on what you're saying about the act of "simply require" a
small amount of new configuration, or a few code tweaks - JDK 9 is sort
of like having the building codes change, and being given a very short
window to conform - or be forced to move out.
I think that's where the "tea party" analogy that Gregg came up with
comes out. It feels like government usurpation. Maybe it is done with
the best of intentions - just like an upgrade to building codes might be
- but /forcing/ everyone to change is counterproductive.

Running with that analogy: most applications can opt out of "going
modular" and stick with putting everything on the classpath. Many
applications we run internally run just fine on JDK 9 without any
modifications, and can likely continue to do so through many major
releases to come. Sooner or later the opportunity cost of not upgrading
to a modular approach might grow too large, but I think most will
experience a more gentler pressure to upgrade than you're suggesting.

The obvious exception is of course those applications which ignored all
the warnings and use non-public internals - I guess the home owner
analogy equivalent would be someone hooking up to the power grid in
some unordained or illegal way: Should the government be blocked from
doing much-needed upgrades to the power grid for everyone since some
use unlicenced equipment? Would it be OK if they did so with some due
warning (major release)? Or should they be free to do whatever,
whenever (minor, patch or security release)?

/Claes

[Gregg Wonderly]

Post by Gregg Wonderly

Post by Eric Johnson
At least someone replied to my question.

Post by Russell Gold

What infuriates me is that in all this discussion, I don't see anyone talking about a threat analysis. What are we trying to protect, from whom, and why? I see comments about how implementation details of the JRE (such as "com.sun" packages) must be hidden, but without reference to the threats that cause a problem.

It’s primarily a maintenance issue, IMO. It is common that we provide classes and methods that are intended to be used from elsewhere inside a product, but which we do not want users to see. That is, it is much the same as the reason you use “private” for class internals - if everything is publicly accessible, people use it, and you cannot refactor your code without breaking theirs.

If the entire aim is just a maintenance issue, then I assert that compiler changes should be sufficient. At that point, anyone who fires up runtime reflection to work around the compiler gets the benefits and the costs that they deserve. Benefits include the ability to leverage code in the guts of any library or the JRE that the authors decided they didn't want to publish. And they embrace the likelihood that it will break in the future.

More rigorous versioning could help mediate this so that when my introspecting code which works on JDK7 is deployed on JDK8, it would not run if I declared a version dependency on JDK7. We currently have versioning that works with newer JDKs by default, without the ability to declare that only one JDK works.

http://openjdk.java.net/jeps/238 <http://openjdk.java.net/jeps/238> allows graceful transition from
introspecting code and other hacks to supported APIs in 9, 10 etc.

Post by Gregg Wonderly
Believe me, I understand how ugly many people in the JDK development team feel it is for all this reflection to be done, and how responsible they seem to feel, given the direction JigSaw is going, to try and keep people from “breaking” their own software. But practically, they are taking that risk. Leave them in the dust by ignoring their dependencies. But, don’t shut the door on long term solutions evolving from that experimentation/introspection which allows the platform to become better because they help you understand that an API change is actually in order because there is a use case that was missed in the original JDK design.
What will happen if JigSaw is deployed as it is today, is that people will elect to use something else. They literally will run as fast as they can to a “place” where participation is in their control, not some outside party.
Think of the Boston Tea Party. From Hamilton lyrics, “Why should some tiny island on the other side of the world set the price of tea?” Making the JDK seem like a King of overlord with all kinds of limits applied will make people really mad. Also from Hamilton, “I’ll kill your family to remind you of my love” is sort of how people look at this kind of “control” and “overreach” of responsibility in software. It’s like JDK-9 shuts down all kinds of responsibilities and totally disrupts interactions that used to feel friendly and inviting and instead makes them feel like a teach standing over you with a ruler, smacking your hand every time you try and touch something they don’t want you to.

Sorry if this sounds rude, but it seems like you're all arguing some straw man idea of jigsaw rather than the actual implementation,
in addition to going slightly off-topic.
It's still perfectly possible to break through to non-exported
packages: it will simply require use of either new flags
(addExports/addReads, upgraded modules etc) or use of the newly added
extensions to the reflection API. And as has been stated again and again elsewhere: setAccessible and friends are not going anywhere soon.
Rather than an unfriendly or even despotic attempt to wrest control
from free developer souls, I find it quite agreeable to give
module/library/JDK authors a stronger way of saying "this is
internal, please keep out". Unless there is a security manager
prohibiting you, but hey, no change there.
What will break is the ability for libraries to go ahead and do this
without explicit approval from the "deployer", application owner, user.

But, it’s impossible for this ability to be removed. People will decompile modules, alter their declarations and make things accessible if they need access. There is no hope that JigSaw mechanisms can keep people out of your code. Once you release it, every bit of it is accessible, regardless of what JigSaw attempts to prohibit.

As I said before, massive encryption and an hardened deployment environment, inaccessible to the developer will be the only way. Since the developer has to compile against something, this would require an interface only module deployment for compilation and a secondary hard deployment environment for the developer to deploy into. Common java deployments have none of these capabilities.

The attempts at security and isolation through declarative mechanisms is just not changing anything about what already is happening/possible in the language with keywords. And reflection removes those limits to allow systems to utilize the software APIs in ways that were unknown at the time of packaging. If modules are built with limits on deployment and controls asserted to attempt to keep parts of the “system” inaccessible, why are we releasing software for others to use?

Currently plenty of libraries take these freedoms on the behalf of that
guy or gal, who might not even be aware of the library authors decision
to cut corners. Some of these people might not think it's worth the
cost or risk of breaking future compatibility for some convenience, a
few points on a benchmark or whatever other reason.

It’s not about benchmarks. It’s about getting access to internal data structures to assert proper locking to use Collections inside your code with the needed locking in new ways that you didn’t imaging. People are decompiling Jars, inspecting the functionality, learning, and then adapting things or fixing things that they need to work or be different.

As we're being emotional: I feel good about the way jigsaw brings
control back to the application owner to do more informed decisions and
to library authors to be more confident about evolving their code, and
hope that someday you will feel that there's actually a big deal of
common sense invested into jigsaw.

How is this any different from just changing versions? If you want your software systems to evolve with the users, are you sure you want the burden of changing module declarations and redeploying/releasing new versions continuously as you find out that your users are needing? How will that enable you, or your users to be more productive? They will be waiting for you to change things, and you will be hoping for them to give you that feedback. How often do you want to hear about “packaging needs”?

Again, many will decompile your module, change the declarations to do what they need, and await “code changes” instead of just packaging changes I would guess.

Gregg

Now, can we go back to arguing about public vs package private?
/Claes

[m***@oracle.com]

Post by David M. Lloyd
...
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. ...

FYI, to jigsaw-dev readers: This approach was discussed on the JPMS EG
list late last year. Here are links to the relevant messages:

http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-November/000194.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000215.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000219.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000222.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000223.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000227.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000228.html

- Mark

[David M. Lloyd]

Post by m***@oracle.com

Post by David M. Lloyd
...
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. ...

FYI, to jigsaw-dev readers: This approach was discussed on the JPMS EG
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-November/000194.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000215.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000219.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000222.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000223.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000227.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000228.html

Also note that the discussion tapered off inconclusively before really
discussing the possibility of selectively opening the package-private
level to friends. The above links are mostly about the idea of changing
package-private to mean module-private, which was dismissed as problematic.

Using the selective extension of package-private does not suffer from
the fatal security problems caused by simple recompilation from -target
8 to -target 9. The status quo is maintained in this case; users would
have to opt in to extending access, just as Jigsaw requires users to opt
in to make public classes available right now.

[Paul Benedict]

If I may opine on this matter -- and do so respectfully toward all parties
mentioned -- aside from Tim Ellison responding first, every other message
is between David and Mark. The discussion thread is a really good read and
a strong point/counterpoint match. However, there are 9 people on the
Expert Group [1]. What do the other 6 experts think? Being an observer, I
can see nothing but public discussion, and so all appearances on this list
tell me the item was left unresolved. I have no idea where the EG actually
stands as a whole on David's suggestion. I remember reading these exchanges
live, and curiously wondering why there are no additional agreements or
disagreements? My best theory then and now is this: Thanksgiving and
Christmas happened. It appears the holidays interrupted. Just my 2 cents.

[1] http://openjdk.java.net/projects/jigsaw/spec/

Cheers,
Paul

Post by David M. Lloyd

Post by m***@oracle.com

Post by David M. Lloyd
...
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. ...

FYI, to jigsaw-dev readers: This approach was discussed on the JPMS EG
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-November/000194.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000215.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000219.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000222.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000223.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000227.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000228.html

Also note that the discussion tapered off inconclusively before really
discussing the possibility of selectively opening the package-private level
to friends. The above links are mostly about the idea of changing
package-private to mean module-private, which was dismissed as problematic.
Using the selective extension of package-private does not suffer from the
fatal security problems caused by simple recompilation from -target 8 to
-target 9. The status quo is maintained in this case; users would have to
opt in to extending access, just as Jigsaw requires users to opt in to make
public classes available right now.
--
- DML

[Neil Bartlett]

Post by Paul Benedict
If I may opine on this matter -- and do so respectfully toward all parties
mentioned -- aside from Tim Ellison responding first, every other message
is between David and Mark. The discussion thread is a really good read and
a strong point/counterpoint match. However, there are 9 people on the
Expert Group [1]. What do the other 6 experts think? Being an observer, I
can see nothing but public discussion, and so all appearances on this list
tell me the item was left unresolved. I have no idea where the EG actually
stands as a whole on David's suggestion.

The EG is not a democracy – they are there to advise and challenge the Spec Lead. So you can interpret the direction of the JSR based on whether EG members (mostly David) have had any success in persuading Mark of their arguments.

I do share your concern about the lack of engagement by the other EG members. Most egregiously: Jason van Zyl, Wayne Beaton and Hans Dockter have posted no messages after their initial hellos, and Bob Lee did not even do that basic courtesy and has posted exactly zero messages to date.

Apparently most of the experts have no opinion on the biggest ever change in the Java platform — unless they all agree with Mark so completely that they feel no need to comment?

Post by Paul Benedict
I remember reading these exchanges
live, and curiously wondering why there are no additional agreements or
disagreements? My best theory then and now is this: Thanksgiving and
Christmas happened. It appears the holidays interrupted. Just my 2 cents.

Thanksgiving is only an excuse for the Americans on the list ;-)

Neil

Post by Paul Benedict
[1] http://openjdk.java.net/projects/jigsaw/spec/
Cheers,
Paul

Post by David M. Lloyd

Post by m***@oracle.com

Post by David M. Lloyd
...
I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private. ...

FYI, to jigsaw-dev readers: This approach was discussed on the JPMS EG
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-November/000194.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000215.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000219.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000222.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000223.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000227.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000228.html

Also note that the discussion tapered off inconclusively before really
discussing the possibility of selectively opening the package-private level
to friends. The above links are mostly about the idea of changing
package-private to mean module-private, which was dismissed as problematic.
Using the selective extension of package-private does not suffer from the
fatal security problems caused by simple recompilation from -target 8 to
-target 9. The status quo is maintained in this case; users would have to
opt in to extending access, just as Jigsaw requires users to opt in to make
public classes available right now.
--
- DML