Discussion:
New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
m***@oracle.com
2016-10-27 16:07:09 UTC
Permalink
Thanks to everyone who commented on the weak-modules proposal. I've
just posted a third proposal for #ReflectiveAccessToNonExportedTypes,
along with an analysis of the proposals presented thus far:

http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html

Further comments most welcome, as usual!

- Mark
Alan Snyder
2016-10-27 18:10:52 UTC
Permalink
To ensure the integrity of the platform we expect that all the modules of
the JDK itself will be normal modules and that very few, if any, packages
will be opened.
I’m wondering if it would be possible to open the package(s) used by Swing Look and Feels, to better support custom look and feels.

The most obvious candidates are the packages containing the Portable Look and Feel. The PLAF design is somewhat schizophrenic. It is clearly designed for reuse, yet it has encapsulations that sometimes get in the way of reuse. (You can see examples in the Aqua LAF, originally coded by Apple, where there are workarounds with comments like “bad Sun”.)

Other candidates are AWT support packages, which I have found useful in implementing a custom platform-specific look and feel.

The basic issue for my look and feel is whether I will need to include instructions for developers on special command line arguments to allow the code to run in JDK 9. Obviously, I would rather not have to do this.

What is not clear to me is whether the goal of protecting the integrity of the platform is layered in some way, with some parts of the platform more critical than others.

Alan
Thanks to everyone who commented on the weak-modules proposal. I've
just posted a third proposal for #ReflectiveAccessToNonExportedTypes,
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html
Further comments most welcome, as usual!
- Mark
Alan Bateman
2016-10-28 09:48:41 UTC
Permalink
Post by Alan Snyder
I’m wondering if it would be possible to open the package(s) used by Swing Look and Feels, to better support custom look and feels.
The most obvious candidates are the packages containing the Portable Look and Feel. The PLAF design is somewhat schizophrenic. It is clearly designed for reuse, yet it has encapsulations that sometimes get in the way of reuse. (You can see examples in the Aqua LAF, originally coded by Apple, where there are workarounds with comments like “bad Sun”.)
Other candidates are AWT support packages, which I have found useful in implementing a custom platform-specific look and feel.
The basic issue for my look and feel is whether I will need to include instructions for developers on special command line arguments to allow the code to run in JDK 9. Obviously, I would rather not have to do this.
These are really topics for swing-dev and awt-dev. All the supported
APIs in the java.desktop module are already exported unconditionally.
The packages for those APIs are not proposed to be open. AFAIK, the name
of the LAFs are a supported interface (as the Swing API requires the
class name) but directly using or extending is not. JEP 272 [1] defines
several new APIs for desktop applications that would have traditionally
needed to use unsupported APIs. In addition, Phil Race has proposed a
jdk.desktop module [2] to provide a solution for yet more cases where
applications have been hacking into JDK internals.

-Alan

[1] http://openjdk.java.net/jeps/272
[2] http://mail.openjdk.java.net/pipermail/awt-dev/2016-October/012089.html
Gregg Wonderly
2016-10-27 18:37:29 UTC
Permalink
I find this text much more demonstrative of the needs of the many. Thanks for continuing to work through things with the community and understand needs that we all have for a good path forward which allows everyone to use JDK9 and for the future to hold a much better view of encapsulation without the pain of needing to unscrew the whole framework to get to details that are vital today.

Gregg
Post by m***@oracle.com
Thanks to everyone who commented on the weak-modules proposal. I've
just posted a third proposal for #ReflectiveAccessToNonExportedTypes,
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html
Further comments most welcome, as usual!
- Mark
Stephen Colebourne
2016-10-28 02:20:52 UTC
Permalink
Thanks for this new proposal, I am (unsurprisingly) happy with it. The use
of 'open' and 'opens' seems to work well too.
Stephen
Post by m***@oracle.com
Thanks to everyone who commented on the weak-modules proposal. I've
just posted a third proposal for #ReflectiveAccessToNonExportedTypes,
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/
2016-October/000430.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/
2016-October/000431.html
Further comments most welcome, as usual!
- Mark
Sander Mak
2016-10-28 11:55:22 UTC
Permalink
No specific comments (yet), but wanted to express a sincere thanks for taking into account the community feedback around this issue. Things are looking good.


Sander
Post by m***@oracle.com
Thanks to everyone who commented on the weak-modules proposal. I've
just posted a third proposal for #ReflectiveAccessToNonExportedTypes,
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html
Further comments most welcome, as usual!
- Mark
John Rose
2016-10-28 22:37:45 UTC
Permalink
Post by m***@oracle.com
Further comments most welcome, as usual!
+100 for *qualified* opens; this puts an important limit on deep reflection.

My main concern with reflection is to avoid "falling off the encapsulation cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected. Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.

So qualified opens is a partial solution. And, I think it is exactly right for
today, because it can be extended tomorrow. I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".

A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection. Call such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set. In this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.

For future JDK releases we can consider layering of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open. This meta-framework can
then perform deep reflection on behalf of self-describing "moderately reflecting"
client frameworks. User modules would not directly open themselves to any
module other than MR.

The advantage would come when the client frameworks explicitly declare their
self-restraint ("moderation") in reflection. For example, the "Foo" framework
might limit itself to names annotated with @FooFrameworkHook.

An AOT compiler (or other jlink-time reorganization tool) would (1) note that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework. At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo. MR would then
gain the requested access and delegate it back to Foo. Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.

(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo. Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)

Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on reflection.
Thus, MR-specific metadata might prevent MR from granting bar
access to normally-trusted frameworks that bar doesn't trust, if
bar can blacklist them for itself in a place that MR checks.

And so on. None of this needs to be built into the module system.
To gain the benefit of it, basic parametric policies need to be implemented,
adopted by client frameworks, and used by tools to perform optimizations.

A typical optimization is "tree shaking", where a class or method that is
never used is dropped from the application. Or, if it is used in a way that
can be inlined locally, the metadata can still be dropped. In either case,
making code go dead is an important ingredient to many optimizations.

— John
Paul Bakker
2016-10-31 17:06:27 UTC
Permalink
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?

Thanks,

Paul
Post by John Rose
Post by m***@oracle.com
Further comments most welcome, as usual!
+100 for *qualified* opens; this puts an important limit on deep reflection.
My main concern with reflection is to avoid "falling off the encapsulation cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected. Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.
So qualified opens is a partial solution. And, I think it is exactly right for
today, because it can be extended tomorrow. I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".
A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection. Call such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set. In this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.
For future JDK releases we can consider layering of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open. This meta-framework can
then perform deep reflection on behalf of self-describing "moderately reflecting"
client frameworks. User modules would not directly open themselves to any
module other than MR.
The advantage would come when the client frameworks explicitly declare their
self-restraint ("moderation") in reflection. For example, the "Foo" framework
An AOT compiler (or other jlink-time reorganization tool) would (1) note that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework. At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo. MR would then
gain the requested access and delegate it back to Foo. Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.
(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo. Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)
Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on reflection.
Thus, MR-specific metadata might prevent MR from granting bar
access to normally-trusted frameworks that bar doesn't trust, if
bar can blacklist them for itself in a place that MR checks.
And so on. None of this needs to be built into the module system.
To gain the benefit of it, basic parametric policies need to be implemented,
adopted by client frameworks, and used by tools to perform optimizations.
A typical optimization is "tree shaking", where a class or method that is
never used is dropped from the application. Or, if it is used in a way that
can be inlined locally, the metadata can still be dropped. In either case,
making code go dead is an important ingredient to many optimizations.
— John
Alan Bateman
2016-10-31 17:20:21 UTC
Permalink
Post by Paul Bakker
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?
We have an initial implementation in the jake forest and expect to have
new builds + javadoc on the java.net download site tomorrow. The new
build will be based on jdk-9+142, assuming we can get the merge/sync-up
done today.

-Alan
Stephen Felts
2016-10-31 17:44:33 UTC
Permalink
That means in the next build as of tomorrow the class format is changed and classes need to be re-generated?

-----Original Message-----
From: Alan Bateman
Sent: Monday, October 31, 2016 1:20 PM
To: Paul Bakker; jigsaw-***@openjdk.java.net
Subject: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Post by Paul Bakker
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?
We have an initial implementation in the jake forest and expect to have new builds + javadoc on the java.net download site tomorrow. The new build will be based on jdk-9+142, assuming we can get the merge/sync-up done today.

-Alan
Alan Bateman
2016-10-31 18:33:39 UTC
Permalink
Post by Stephen Felts
That means in the next build as of tomorrow the class format is changed and classes need to be re-generated?
Yes, any module-info.java from previous builds will need to be
re-compiled. There is also an update to the Module attribute to rev the
format of the provides_table. If that gets in today (Jon is working on
it) then it means it will be in the same build so only one transition.

-Alan
Andrew Dinn
2016-10-31 17:19:24 UTC
Permalink
Post by Paul Bakker
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?
Alan just pushed an 'initial implementation of open modules' into the
jake forest.

http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-October/009860.html

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Remi Forax
2016-10-31 23:57:47 UTC
Permalink
Hi Paul,
I think you can already sketch an implementation of the proposal now by using either an annotation processor + a jlink plugin, or by a jlink plugin + a bytecode processor tool.
I believe that you can push the information needed by the meta-framework to fulfill requests by adding some method calls inside the static initializer of the classes that are annotated.
The runtime part can be something along that line

https://gist.github.com/forax/1511fae2273f04273ff9463c6fbbdfbc

Instead of providing method handles as John suggests, i provide the lookup object which is less secure.
And the Class object in the map are stored as strong ref instead of being an ephemeron (the lambda used as value in the map has also a strong reference on the class).

Rémi

----- Mail original -----
Envoyé: Lundi 31 Octobre 2016 18:06:27
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?
Thanks,
Paul
Post by John Rose
Post by m***@oracle.com
Further comments most welcome, as usual!
+100 for *qualified* opens; this puts an important limit on deep reflection.
My main concern with reflection is to avoid "falling off the encapsulation cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected. Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.
So qualified opens is a partial solution. And, I think it is exactly right for
today, because it can be extended tomorrow. I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".
A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection. Call such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set. In this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.
For future JDK releases we can consider layering of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open. This meta-framework can
then perform deep reflection on behalf of self-describing "moderately reflecting"
client frameworks. User modules would not directly open themselves to any
module other than MR.
The advantage would come when the client frameworks explicitly declare their
self-restraint ("moderation") in reflection. For example, the "Foo" framework
An AOT compiler (or other jlink-time reorganization tool) would (1) note that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework. At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo. MR would then
gain the requested access and delegate it back to Foo. Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.
(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo. Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)
Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on reflection.
Thus, MR-specific metadata might prevent MR from granting bar
access to normally-trusted frameworks that bar doesn't trust, if
bar can blacklist them for itself in a place that MR checks.
And so on. None of this needs to be built into the module system.
To gain the benefit of it, basic parametric policies need to be implemented,
adopted by client frameworks, and used by tools to perform optimizations.
A typical optimization is "tree shaking", where a class or method that is
never used is dropped from the application. Or, if it is used in a way that
can be inlined locally, the metadata can still be dropped. In either case,
making code go dead is an important ingredient to many optimizations.
— John
Remi Forax
2016-11-01 12:12:10 UTC
Permalink
----- Mail original -----
Envoyé: Mardi 1 Novembre 2016 00:57:47
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Hi Paul,
I think you can already sketch an implementation of the proposal now by using
either an annotation processor + a jlink plugin, or by a jlink plugin + a
bytecode processor tool.
I believe that you can push the information needed by the meta-framework to
fulfill requests by adding some method calls inside the static initializer of
the classes that are annotated.
The runtime part can be something along that line
https://gist.github.com/forax/1511fae2273f04273ff9463c6fbbdfbc
Instead of providing method handles as John suggests, i provide the lookup
object which is less secure.
And the Class object in the map are stored as strong ref instead of being an
ephemeron (the lambda used as value in the map has also a strong reference on
the class).
After shower edit,
i've updated the gist to use a ClassValue + a CAS so there is no strong ref to the classes now.
Rémi
Rémi
----- Mail original -----
Envoyé: Lundi 31 Octobre 2016 18:06:27
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules &
open packages
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?
Thanks,
Paul
Post by John Rose
Post by m***@oracle.com
Further comments most welcome, as usual!
+100 for *qualified* opens; this puts an important limit on deep reflection.
My main concern with reflection is to avoid "falling off the encapsulation cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected. Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.
So qualified opens is a partial solution. And, I think it is exactly right for
today, because it can be extended tomorrow. I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".
A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection. Call such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set. In this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.
For future JDK releases we can consider layering of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open. This meta-framework can
then perform deep reflection on behalf of self-describing "moderately reflecting"
client frameworks. User modules would not directly open themselves to any
module other than MR.
The advantage would come when the client frameworks explicitly declare their
self-restraint ("moderation") in reflection. For example, the "Foo" framework
An AOT compiler (or other jlink-time reorganization tool) would (1) note that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework. At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo. MR would then
gain the requested access and delegate it back to Foo. Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.
(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo. Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)
Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on reflection.
Thus, MR-specific metadata might prevent MR from granting bar
access to normally-trusted frameworks that bar doesn't trust, if
bar can blacklist them for itself in a place that MR checks.
And so on. None of this needs to be built into the module system.
To gain the benefit of it, basic parametric policies need to be implemented,
adopted by client frameworks, and used by tools to perform optimizations.
A typical optimization is "tree shaking", where a class or method that is
never used is dropped from the application. Or, if it is used in a way that
can be inlined locally, the metadata can still be dropped. In either case,
making code go dead is an important ingredient to many optimizations.
— John
David M. Lloyd
2016-11-01 13:53:19 UTC
Permalink
I just want to point out the two obvious (closely related) major
problems with this approach which prevent it from being a practical
replacement for setAcceptable, lest they are forgotten/ignored in the
excitement around the new ideas:

1. It requires the target class to be initialized
2. It requires the target class to proactively donate MethodHandles or a
Lookup to the lookup class

Both of these are effectively pretty substantial usability regressions
compared to the status quo. These problems are why I'd rather see some
kind of static grant mechanism, and either a special static API to get
an extra-privileged Lookup or the appropriate magic to allow a class's
"normal" Lookup object to "see" everything that was granted to it; but
really any idea which would solve both of these problems in any way
should be put forward for consideration IMO.
Post by Remi Forax
Hi Paul,
I think you can already sketch an implementation of the proposal now by using either an annotation processor + a jlink plugin, or by a jlink plugin + a bytecode processor tool.
I believe that you can push the information needed by the meta-framework to fulfill requests by adding some method calls inside the static initializer of the classes that are annotated.
The runtime part can be something along that line
https://gist.github.com/forax/1511fae2273f04273ff9463c6fbbdfbc
Instead of providing method handles as John suggests, i provide the lookup object which is less secure.
And the Class object in the map are stored as strong ref instead of being an ephemeron (the lambda used as value in the map has also a strong reference on the class).
Rémi
----- Mail original -----
Envoyé: Lundi 31 Octobre 2016 18:06:27
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?
Thanks,
Paul
Post by John Rose
Post by m***@oracle.com
Further comments most welcome, as usual!
+100 for *qualified* opens; this puts an important limit on deep reflection.
My main concern with reflection is to avoid "falling off the encapsulation cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected. Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.
So qualified opens is a partial solution. And, I think it is exactly right for
today, because it can be extended tomorrow. I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".
A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection. Call such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set. In this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.
For future JDK releases we can consider layering of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open. This meta-framework can
then perform deep reflection on behalf of self-describing "moderately reflecting"
client frameworks. User modules would not directly open themselves to any
module other than MR.
The advantage would come when the client frameworks explicitly declare their
self-restraint ("moderation") in reflection. For example, the "Foo" framework
An AOT compiler (or other jlink-time reorganization tool) would (1) note that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework. At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo. MR would then
gain the requested access and delegate it back to Foo. Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.
(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo. Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)
Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on reflection.
Thus, MR-specific metadata might prevent MR from granting bar
access to normally-trusted frameworks that bar doesn't trust, if
bar can blacklist them for itself in a place that MR checks.
And so on. None of this needs to be built into the module system.
To gain the benefit of it, basic parametric policies need to be implemented,
adopted by client frameworks, and used by tools to perform optimizations.
A typical optimization is "tree shaking", where a class or method that is
never used is dropped from the application. Or, if it is used in a way that
can be inlined locally, the metadata can still be dropped. In either case,
making code go dead is an important ingredient to many optimizations.
— John
--
- DML
John Rose
2016-11-01 14:04:33 UTC
Permalink
Post by David M. Lloyd
1. It requires the target class to be initialized
2. It requires the target class to proactively donate MethodHandles or a Lookup to the lookup class
Both of these can be overcome, though only by privileged code.
The privileged code would forge (uh, "mint") a legitimate Lookup to the not-yet-initialized class.
A "Vault" meta-framework doesn't need to inject a Lookup donation statement into anybody's <clinit>.
Instead, it needs to do the super-user operation of making a trusted lookup.
It must also fulfill the super-user *responsibility* of not leaking such lookups, just using them in a predictable, rule-based manner.
— John
Jochen Theodorou
2016-11-01 14:22:07 UTC
Permalink
Post by John Rose
Post by David M. Lloyd
1. It requires the target class to be initialized
2. It requires the target class to proactively donate MethodHandles or a Lookup to the lookup class
Both of these can be overcome, though only by privileged code.
The privileged code would forge (uh, "mint") a legitimate Lookup to the not-yet-initialized class.
A "Vault" meta-framework doesn't need to inject a Lookup donation statement into anybody's <clinit>.
Instead, it needs to do the super-user operation of making a trusted lookup.
It must also fulfill the super-user *responsibility* of not leaking such lookups, just using them in a predictable, rule-based manner.
Can we clarify "privileged code"? Privileged like in a SecurityManager
in a PrivilegedAction for example, for privileged like only jdk internal
code? Just to see it black on white ;)

bye Jochen
John Rose
2016-11-01 14:23:18 UTC
Permalink
Can we clarify "privileged code"? Privileged like in a SecurityManager in a PrivilegedAction for example, for privileged like only jdk internal code? Just to see it black on white ;)
Good question: I mean the basic JDK platform implementation. Something deep in java.base. Like Unsafe.
David M. Lloyd
2016-11-01 14:39:01 UTC
Permalink
Post by John Rose
Can we clarify "privileged code"? Privileged like in a SecurityManager in a PrivilegedAction for example, for privileged like only jdk internal code? Just to see it black on white ;)
Good question: I mean the basic JDK platform implementation. Something deep in java.base. Like Unsafe.
I don't see why this can't be a "regular" API though, rather than a
super-user sledgehammer every single time. If user code can be
statically granted access, and that user code can deliberately acquire a
narrowly-scoped object which can access those Lookups/*Handles, then
isn't that better than using Unsafe, which not only represents
unrestricted system-wide access, but can undermine even the JVM's
integrity if leaked?
--
- DML
Andrew Dinn
2016-11-01 15:09:59 UTC
Permalink
Post by David M. Lloyd
Post by John Rose
Post by Jochen Theodorou
Can we clarify "privileged code"? Privileged like in a
SecurityManager in a PrivilegedAction for example, for privileged
like only jdk internal code? Just to see it black on white ;)
Good question: I mean the basic JDK platform implementation.
Something deep in java.base. Like Unsafe.
I don't see why this can't be a "regular" API though, rather than a
super-user sledgehammer every single time. If user code can be
statically granted access, and that user code can deliberately acquire a
narrowly-scoped object which can access those Lookups/*Handles, then
isn't that better than using Unsafe, which not only represents
unrestricted system-wide access, but can undermine even the JVM's
integrity if leaked?
If making this a 'regular' API means allowing any code to use it then
that clearly undermines /all/ the access restrictions provided by
modules. The point is not to unlock all doors in the castle but to open
doors for trusted code when needed (and only then).

There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.reflect.MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel

regards,


Andrew Dinn
-----------
Andrew Dinn
2016-11-01 15:14:17 UTC
Permalink
Oops, wrong package! -- corrected inline
Post by Andrew Dinn
Post by David M. Lloyd
Post by John Rose
Post by Jochen Theodorou
Can we clarify "privileged code"? Privileged like in a
SecurityManager in a PrivilegedAction for example, for privileged
like only jdk internal code? Just to see it black on white ;)
Good question: I mean the basic JDK platform implementation.
Something deep in java.base. Like Unsafe.
I don't see why this can't be a "regular" API though, rather than a
super-user sledgehammer every single time. If user code can be
statically granted access, and that user code can deliberately acquire a
narrowly-scoped object which can access those Lookups/*Handles, then
isn't that better than using Unsafe, which not only represents
unrestricted system-wide access, but can undermine even the JVM's
integrity if leaked?
If making this a 'regular' API means allowing any code to use it then
that clearly undermines /all/ the access restrictions provided by
modules. The point is not to unlock all doors in the castle but to open
doors for trusted code when needed (and only then).
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.reflect.MethodHandles to a nominated module provided by your
^
+--java.lang.invoke.MethodHandles
Post by Andrew Dinn
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
regards,
Andrew Dinn
-----------
--
regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
David M. Lloyd
2016-11-01 15:35:42 UTC
Permalink
Post by Andrew Dinn
Post by David M. Lloyd
Post by John Rose
Post by Jochen Theodorou
Can we clarify "privileged code"? Privileged like in a
SecurityManager in a PrivilegedAction for example, for privileged
like only jdk internal code? Just to see it black on white ;)
Good question: I mean the basic JDK platform implementation.
Something deep in java.base. Like Unsafe.
I don't see why this can't be a "regular" API though, rather than a
super-user sledgehammer every single time. If user code can be
statically granted access, and that user code can deliberately acquire a
narrowly-scoped object which can access those Lookups/*Handles, then
isn't that better than using Unsafe, which not only represents
unrestricted system-wide access, but can undermine even the JVM's
integrity if leaked?
If making this a 'regular' API means allowing any code to use it then
that clearly undermines /all/ the access restrictions provided by
modules. The point is not to unlock all doors in the castle but to open
doors for trusted code when needed (and only then).
Maybe I wasn't clear; I was suggesting that any code be able to use the
API, but only to access things which were explicitly granted. I'm
suggesting *less* access, not *more*.

Saying "suitably privileged" and "Unsafe" seems sufficient until you
realize that many many frameworks will need this API, which means they
will need Unsafe, which means they will *get* Unsafe (one way or
another), which is certainly a far more dangerous thing to have floating
around than (say) a Lookup that has access to a few specific private
members.
Post by Andrew Dinn
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
I don't see how this is safer though. You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.

At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly). The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.

Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO. An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.
--
- DML
Andrew Dinn
2016-11-01 16:02:37 UTC
Permalink
Post by David M. Lloyd
Post by Andrew Dinn
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
I don't see how this is safer though. You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.
No, I'm suggesting handing the keys to a single module provided by your
framework -- which module will then use a key to open a room on demand
for your code and your code only. That means the privilege can be
granted using one command line switch with use of that privilege
controlled by a runtime mechanism of your choosing.
Post by David M. Lloyd
At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly). The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.
I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.
Post by David M. Lloyd
Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO. An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.
If you implement a module with the ability to hand out Lookup instances
you can use whatever control model you like. For example, you might park
an instance which hands out Lookups in a location only available to your
framework or ensure that any caller asking for a Lookup belongs in a
package implemented by your framework or whatever you want. the point is
that once you have granted a class in our code the ability to create
Lookups you can implement whatever runtime access control you want.
Clearly,as John said, you need to do that responsibly.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
David M. Lloyd
2016-11-01 20:00:35 UTC
Permalink
Post by Andrew Dinn
Post by David M. Lloyd
Post by Andrew Dinn
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
I don't see how this is safer though. You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.
No, I'm suggesting handing the keys to a single module provided by your
framework -- which module will then use a key to open a room on demand
for your code and your code only. That means the privilege can be
granted using one command line switch with use of that privilege
controlled by a runtime mechanism of your choosing.
Yes, however that means that I'm on the hook to make sure that the keys
aren't mislaid. And given the fact of open modules - which already
semantically grants me privileges - requiring a special command line
switch (which is itself undesirable) and custom code to enable those
privileges in the *preferred* manner (i.e. MethodHandles) seems
backwards when open modules would already grant me the ability to
utilize a much finer instrument albeit with the legacy approach (i.e.
reflection). It seems quite logical to extend this mechanism to the
preferred MethodHandle approach instead, and forget about Unsafe and
anything like it, which I think is in everyone's best interests in the
long term.
Post by Andrew Dinn
Post by David M. Lloyd
At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly). The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.
I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.
That is marginally better, of course, but I think the rest of my
objections still stand against this argument.
Post by Andrew Dinn
Post by David M. Lloyd
Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO. An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.
If you implement a module with the ability to hand out Lookup instances
you can use whatever control model you like. For example, you might park
an instance which hands out Lookups in a location only available to your
framework or ensure that any caller asking for a Lookup belongs in a
package implemented by your framework or whatever you want. the point is
that once you have granted a class in our code the ability to create
Lookups you can implement whatever runtime access control you want.
Clearly,as John said, you need to do that responsibly.
regards,
Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
--
- DML
Remi Forax
2016-11-01 22:59:01 UTC
Permalink
Iteration 3,
https://gist.github.com/forax/c08ce0a8dc88705bce97b17b63e7e2d5

GrantAccess can now be put on a class, on a package or on a module, and give access to several frameworks by their names. Getting access to the lookup of a class is now a 2 steps process, first claim a framework name to get an AccessFacade and then through the AccessFacade, a framework have access to all classes annotated (directly or indirectly) by GrantAccess and allowing the framework. The facade corresponding to a framework name can only be requested once so how to store the FacadeAccess is let to the framework implementation and if a code try to sneak in and use a framework name it will be detected.

Rémi

----- Mail original -----
Envoyé: Mardi 1 Novembre 2016 21:00:35
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Post by Andrew Dinn
Post by David M. Lloyd
Post by Andrew Dinn
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
I don't see how this is safer though. You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.
No, I'm suggesting handing the keys to a single module provided by your
framework -- which module will then use a key to open a room on demand
for your code and your code only. That means the privilege can be
granted using one command line switch with use of that privilege
controlled by a runtime mechanism of your choosing.
Yes, however that means that I'm on the hook to make sure that the keys
aren't mislaid. And given the fact of open modules - which already
semantically grants me privileges - requiring a special command line
switch (which is itself undesirable) and custom code to enable those
privileges in the *preferred* manner (i.e. MethodHandles) seems
backwards when open modules would already grant me the ability to
utilize a much finer instrument albeit with the legacy approach (i.e.
reflection). It seems quite logical to extend this mechanism to the
preferred MethodHandle approach instead, and forget about Unsafe and
anything like it, which I think is in everyone's best interests in the
long term.
Post by Andrew Dinn
Post by David M. Lloyd
At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly). The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.
I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.
That is marginally better, of course, but I think the rest of my
objections still stand against this argument.
Post by Andrew Dinn
Post by David M. Lloyd
Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO. An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.
If you implement a module with the ability to hand out Lookup instances
you can use whatever control model you like. For example, you might park
an instance which hands out Lookups in a location only available to your
framework or ensure that any caller asking for a Lookup belongs in a
package implemented by your framework or whatever you want. the point is
that once you have granted a class in our code the ability to create
Lookups you can implement whatever runtime access control you want.
Clearly,as John said, you need to do that responsibly.
regards,
Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
--
- DML
Peter Levart
2016-11-02 10:31:04 UTC
Permalink
Hi Remi,
Post by Remi Forax
Iteration 3,
https://gist.github.com/forax/c08ce0a8dc88705bce97b17b63e7e2d5
GrantAccess can now be put on a class, on a package or on a module, and give access to several frameworks by their names. Getting access to the lookup of a class is now a 2 steps process, first claim a framework name to get an AccessFacade and then through the AccessFacade, a framework have access to all classes annotated (directly or indirectly) by GrantAccess and allowing the framework. The facade corresponding to a framework name can only be requested once so how to store the FacadeAccess is let to the framework implementation and if a code try to sneak in and use a framework name it will be detected.
Rémi
Hm, this variant requires the real framework code to race against its
impostor(s) for the access facade granted to a particular name. What if
an impostor gets it first?

In addition, I see a similarity of this approach to simply "opening" the
required packages of the "granting" module to a chosen "granted" module
(i.e. the "hibernate") in the granting module declaration. At least the
module name is harder to "spoof" as it is embedded in the module search
path layout. But I get the point. Instead of "hibernate", the granting
module could declare that it gives access to a "jpa" framework, which is
a more general name that several implementations could claim. But I
think that they should not simply claim the name and get it on a
first-comes basis. What about introducing a general registry of: (module
name -> access name) mappings controlled by privileged code or some
policy? The problem I see is a problem of choosing the names in general
as they get to be hard-coded in the code or declaration of the granting
module. Who chooses that "jpa" is the name of the JPA framework? Module
names are chosen by their authors. There has to be a module 1st and then
one can "require" it. OTOH there has to be a general agreement on the
"access name" among several authors of the modules that claim that name.
And this requires a central name registry like IANA. And that's not
something jigsaw is going to dive into.

I tried to tackle the issue from another perspective. If you ask
yourself what is common to all JPA framework implementations, for
example, then the answer is: they all implement the same API (in the
form of interface(s) or abstract classes). This API will usually be a
module by itself. 1st there has to be an API module with the agreed-upon
API and then the authors can produce implementation modules (hibernate,
eclipse-link, ...). So I thought that the name of the API module could
be used as the "access name" of the grant and every implementation
module that could prove it contains the implementation of this API would
inherit the grant.

Ideally, but not necessarily, services and providers could be leveraged
in the implementation. The mechanism could be "secured" on a
service-by-service basis using SecurityManager permission(s) like it is
done today (the security check is put into the constructor(s) of the
abstract service class) so that impostor(s) are kept back.

Regards, Peter
Post by Remi Forax
----- Mail original -----
Envoyé: Mardi 1 Novembre 2016 21:00:35
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Post by Andrew Dinn
Post by David M. Lloyd
Post by Andrew Dinn
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
I don't see how this is safer though. You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.
No, I'm suggesting handing the keys to a single module provided by your
framework -- which module will then use a key to open a room on demand
for your code and your code only. That means the privilege can be
granted using one command line switch with use of that privilege
controlled by a runtime mechanism of your choosing.
Yes, however that means that I'm on the hook to make sure that the keys
aren't mislaid. And given the fact of open modules - which already
semantically grants me privileges - requiring a special command line
switch (which is itself undesirable) and custom code to enable those
privileges in the *preferred* manner (i.e. MethodHandles) seems
backwards when open modules would already grant me the ability to
utilize a much finer instrument albeit with the legacy approach (i.e.
reflection). It seems quite logical to extend this mechanism to the
preferred MethodHandle approach instead, and forget about Unsafe and
anything like it, which I think is in everyone's best interests in the
long term.
Post by Andrew Dinn
Post by David M. Lloyd
At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly). The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.
I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.
That is marginally better, of course, but I think the rest of my
objections still stand against this argument.
Post by Andrew Dinn
Post by David M. Lloyd
Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO. An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.
If you implement a module with the ability to hand out Lookup instances
you can use whatever control model you like. For example, you might park
an instance which hands out Lookups in a location only available to your
framework or ensure that any caller asking for a Lookup belongs in a
package implemented by your framework or whatever you want. the point is
that once you have granted a class in our code the ability to create
Lookups you can implement whatever runtime access control you want.
Clearly,as John said, you need to do that responsibly.
regards,
Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
--
- DML
f***@univ-mlv.fr
2016-11-02 13:06:35 UTC
Permalink
Envoyé: Mercredi 2 Novembre 2016 11:31:04
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules &
open packages
Hi Remi,
Iteration 3, https://gist.github.com/forax/c08ce0a8dc88705bce97b17b63e7e2d5
GrantAccess can now be put on a class, on a package or on a module, and give
access to several frameworks by their names. Getting access to the lookup of a
class is now a 2 steps process, first claim a framework name to get an
AccessFacade and then through the AccessFacade, a framework have access to all
classes annotated (directly or indirectly) by GrantAccess and allowing the
framework. The facade corresponding to a framework name can only be requested
once so how to store the FacadeAccess is let to the framework implementation
and if a code try to sneak in and use a framework name it will be detected.
Rémi
Hm, this variant requires the real framework code to race against its
impostor(s) for the access facade granted to a particular name. What if an
impostor gets it first?
The idea was if the impostor wins, you will get an exception when the legit framework will try to request the access, and obviously, this is the weak point of this variant.
In addition, I see a similarity of this approach to simply "opening" the
required packages of the "granting" module to a chosen "granted" module (i.e.
the "hibernate") in the granting module declaration. At least the module name
is harder to "spoof" as it is embedded in the module search path layout. But I
get the point. Instead of "hibernate", the granting module could declare that
it gives access to a "jpa" framework, which is a more general name that several
implementations could claim. But I think that they should not simply claim the
name and get it on a first-comes basis. What about introducing a general
registry of: (module name -> access name) mappings controlled by privileged
code or some policy? The problem I see is a problem of choosing the names in
general as they get to be hard-coded in the code or declaration of the granting
module. Who chooses that "jpa" is the name of the JPA framework? Module names
are chosen by their authors. There has to be a module 1st and then one can
"require" it. OTOH there has to be a general agreement on the "access name"
among several authors of the modules that claim that name. And this requires a
central name registry like IANA. And that's not something jigsaw is going to
dive into.
I tried to tackle the issue from another perspective. If you ask yourself what
is common to all JPA framework implementations, for example, then the answer
is: they all implement the same API (in the form of interface(s) or abstract
classes). This API will usually be a module by itself. 1st there has to be an
API module with the agreed-upon API and then the authors can produce
implementation modules (hibernate, eclipse-link, ...). So I thought that the
name of the API module could be used as the "access name" of the grant and
every implementation module that could prove it contains the implementation of
this API would inherit the grant.
A nice idea,
so iteration 4, you have to provide a lookup of a class inside a module that provide an implementation to the interface used when granting the access
https://github.com/forax/access-broker

So a code of application using a framework will use @GrantAccess (i've put the annotation on the module of the app just for fun)
https://github.com/forax/access-broker/blob/master/src/my.jpa.app/my/jpa/app/Bean.java
https://github.com/forax/access-broker/blob/master/src/my.jpa.app/module-info.java
and the implementation of the framework will use the AccessBroken to get access to private members of the application code
https://github.com/forax/access-broker/blob/master/src/org.hibernate.jpa/org/hibernate/jpa/EntityManagerImpl.java
In that case the AccessBroker is the same as the previous iteration but uses Class instead of String to allow a framework and check that the class that requires a privileged access is in a module that implement the interface of the spec
https://github.com/forax/access-broker/blob/master/src/java.base/java/lang/invoke/AccessBroker.java
Ideally, but not necessarily, services and providers could be leveraged in the
implementation. The mechanism could be "secured" on a service-by-service basis
using SecurityManager permission(s) like it is done today (the security check
is put into the constructor(s) of the abstract service class) so that
impostor(s) are kept back.
Regards, Peter
regards,
Rémi
----- Mail original -----
21:00:35
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules &
open packages
Post by Andrew Dinn
Post by David M. Lloyd
Post by Andrew Dinn
There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel
I don't see how this is safer though. You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.
No, I'm suggesting handing the keys to a single module provided by your
framework -- which module will then use a key to open a room on demand
for your code and your code only. That means the privilege can be
granted using one command line switch with use of that privilege
controlled by a runtime mechanism of your choosing.
Yes, however that means that I'm on the hook to make sure that the keys
aren't mislaid. And given the fact of open modules - which already
semantically grants me privileges - requiring a special command line
switch (which is itself undesirable) and custom code to enable those
privileges in the *preferred* manner (i.e. MethodHandles) seems
backwards when open modules would already grant me the ability to
utilize a much finer instrument albeit with the legacy approach (i.e.
reflection). It seems quite logical to extend this mechanism to the
preferred MethodHandle approach instead, and forget about Unsafe and
anything like it, which I think is in everyone's best interests in the
long term.
Post by Andrew Dinn
Post by David M. Lloyd
At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly). The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.
I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.
That is marginally better, of course, but I think the rest of my
objections still stand against this argument.
Post by Andrew Dinn
Post by David M. Lloyd
Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO. An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.
If you implement a module with the ability to hand out Lookup instances
you can use whatever control model you like. For example, you might park
an instance which hands out Lookups in a location only available to your
framework or ensure that any caller asking for a Lookup belongs in a
package implemented by your framework or whatever you want. the point is
that once you have granted a class in our code the ability to create
Lookups you can implement whatever runtime access control you want.
Clearly,as John said, you need to do that responsibly.
regards,
Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
--
- DML
John Rose
2016-11-02 01:58:08 UTC
Permalink
Post by Andrew Dinn
I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.
The weak link in this otherwise robust scheme is the use of a command line option to break into the jli package. Breaking in by the back door is awkward and might not deliver the desired Lookup if jli code changes. And it probably will, over time. Using the break-in is a good Proof of concept but the finished product needs to use a real API provided by the JDK, and using a wormhole Lookup actually provided by a cooperating jli. I look forward to such a thing in a future JDK.

Perhaps the right surface shape for moderate reflection is Maurizios "reflection manifesto" API, which is completely interface driven. This means a meta-CE with full deep access could mock up a "reflection manifesto" implementation which would expose exactly the right moderate surface area, as negotiated with the target module. Just an idea…

– John
Andrew Dinn
2016-11-02 13:25:01 UTC
Permalink
Post by John Rose
Post by Andrew Dinn
I did actually suggest a way of avoiding the use of Unsafe. You
give your nominated module full reflective access to
java.lang.invoke allowing it to create Lookup instances (it can
actually just create a single all privileges lookup and use this to
clone others). You don't need to insert a class into
java.lang.invoke to do this. You simply add one exports directive
on the command line.
The weak link in this otherwise robust scheme is the use of a command
line option to break into the jli package. Breaking in by the back
door is awkward and might not deliver the desired Lookup if jli code
changes. And it probably will, over time. Using the break-in is a
good Proof of concept but the finished product needs to use a real
API provided by the JDK, and using a wormhole Lookup actually
provided by a cooperating jli. I look forward to such a thing in a
future JDK.
Me too! But for now it's burglary or bust :-)
Post by John Rose
Perhaps the right surface shape for moderate reflection is Maurizios
"reflection manifesto" API, which is completely interface driven.
This means a meta-CE with full deep access could mock up a
"reflection manifesto" implementation which would expose exactly the
right moderate surface area, as negotiated with the target module.
Just an idea…
Hmm, I have not seen said manifesto. Is it available somewhere public?

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Remi Forax
2016-11-02 14:20:27 UTC
Permalink
----- Mail original -----
Envoyé: Mercredi 2 Novembre 2016 14:25:01
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Post by John Rose
Post by Andrew Dinn
I did actually suggest a way of avoiding the use of Unsafe. You
give your nominated module full reflective access to
java.lang.invoke allowing it to create Lookup instances (it can
actually just create a single all privileges lookup and use this to
clone others). You don't need to insert a class into
java.lang.invoke to do this. You simply add one exports directive
on the command line.
The weak link in this otherwise robust scheme is the use of a command
line option to break into the jli package. Breaking in by the back
door is awkward and might not deliver the desired Lookup if jli code
changes. And it probably will, over time. Using the break-in is a
good Proof of concept but the finished product needs to use a real
API provided by the JDK, and using a wormhole Lookup actually
provided by a cooperating jli. I look forward to such a thing in a
future JDK.
Me too! But for now it's burglary or bust :-)
Post by John Rose
Perhaps the right surface shape for moderate reflection is Maurizios
"reflection manifesto" API, which is completely interface driven.
This means a meta-CE with full deep access could mock up a
"reflection manifesto" implementation which would expose exactly the
right moderate surface area, as negotiated with the target module.
Just an idea…
Hmm, I have not seen said manifesto. Is it available somewhere public?
Maurizio talk about it at last JVM Language Summit
https://encrypted.google.com/search?hl=en&q=youtube%20Valhalla%20Reflection

The manifesto is here
http://cr.openjdk.java.net/~mcimadamore/reflection-manifesto.html
regards,
Andrew Dinn
Rémi
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Andrew Dinn
2016-11-11 13:43:53 UTC
Permalink
Hi John,

I'm reviving an old (by 2-3 weeks) conversation here because it seems
the right place to ask about how to deal with a disparity I found
between the use of reflection vs method handles. I'm have retained the
cc to jigsaw-dev because i) the conversation started there and ii) it
relates to the question of how well Lookups and MethdoHandles can stand
in for reflection. Perhaps that's too tenuous to merit a CC but I'm
adopting the excuse that those who are only interested in more direct
Jigsaw content can simply ignore the rest of this thread.

The disparity I am seeing looks like it might just be an argument for
Maurizio's new reflection API more than anything else but it may just be
that I am doing something wrong. I'll present the problem in as reduced
form as I can manage to extract from the Byteman implementation.

Byteman includes a test to ensure that the type checker allows duck
typing when a compatible array type is passed as input to a method. So,
we have a test method

public class TestArrayArgTypeCheck extends Test {
. . .
public void testArrayCall(String[] args)
{
log("inside testArrayCall");
}
}

where log is defined by the parent class Test.

I use a Byteman rule as follows

RULE test array arg type check
CLASS TestArrayArgTypeCheck
METHOD testArrayCall
AT ENTRY
BIND test : Test = $this;
IF TRUE
DO test.log("args : " + java.util.Arrays.asList($args) );
ENDRULE

This rule is 'injected' at the start of the code for method
testArrayCall -- it's not actually executed as inline bytecode, rather a
callout to the rule interpreter executes the rule (out of line bytecode
execution is also an option but I don't [yet?] need to add that
complexity to the mix). In case you are unfamiliar with how Byteman
operates I'll summarise its operation.

$this and $args name bindings for values passed into the interpreter,
respectively:
the instance of TestArrayArgTypeCheck fielding the call to the
method (which Byteman knows is of type TestArrayArgTypeCheck)
the argument to testArrayCall (which Byteman knows is of type String[])

the BIND clause binds a /rule-local/ variable test (of type Test) to
the instance of TestArrayArgTypeCheck fielding the call to the method

the DO clause
passes $this to Arrays.asList()
pastes the result into a String
passes the String to Test.log

So, the test ensures that Byteman's type checker accepts that the
String[] argument passed to Arrays.asList(Object[]) is a legitimate
argument without throwing a type exception. It doesn't really matter
what the log output is but the test does check for an expected output
and that is where the disparity arises.

The call to the target method is made by the test code as follows

. . .
String[] ordinals = { "first", "second", "third" };
log("calling testArrayCall");
testArrayCall(ordinals);
log("called testArrayCall");
. . .

When this is run on JDK8[-] I see this output which matches expectation:

<log>
calling testArrayCall
args : [first, second, third]
inside testArrayCall
called testArrayCall
</log>

When I run this on JDK9 using a modified version of Byteman that relies
on MethodHandles I see this output:

<log>
calling testArrayCall
args : [[Ljava.lang.String;@36bc55de]
inside testArrayCall
called testArrayCall
</log>

So, the String[] array appears to have been wrapped in an Object[]
before being passed on to Arrays.asList().


On JDK8[-] I use reflection to execute the method call. So, essentially
the code looks like this

class MethodExpression {
Method method;
Expression recipient;
List<Expression> arguments;
. . .
Object interpret(...)
. . .
Object recipientValue =
(recipient != null
? recipient.interpret(...)
: null);
int argCount = arguments.size();
Object[] argValues = new Object[argCount];
for (int i = 0; i < argCount; i++) {
argValues[i] = arguments.get(i).interpret(...);
}
. . .
return method.invoke(recipientValue, argValues);

The essential differences in the code that gets executed on JDK9
(ignoring that I am inlining code here from different branches) are as
follows:

class MethodExpression {
Method method;
MethodHandle handle = getMethodHandle(method);
List<Expression> arguments;
. . .
Object interpret(...)
. . .
Object recipientValue =
(recipient != null
? recipient.interpret(...)
: null);
int argCount = arguments.size();
Object[] argValues = new Object[argCount];
for (int i = 0; i < argCount; i++) {
argValues[i] = arguments.get(i).interpret(helper);
}
. . .
if (recipient == null) {
handle.invoke(argValues);
} else {
handle.invokeWithArguments(recipientValue, argValues);
}

The key point is how I construct that method handle. In essence it is as
follows

Lookup theLookup = ...;

MethodHandle getMethodHandle(Method method) {
try {
MethodType methodType =
MethodType.methodType(method.getReturnType(),
method.getParameterTypes());
isStatic = Modifier.isStatic(method.getModifiers());
if (isStatic) {
handle = theLookup.findStatic(method.getDeclaringClass(),
method.getName(),
methodType);
} else {
MethodHandle h =
theLookup.findVirtual(method.getDeclaringClass(),
method.getName(),
methodType);
handle = h.asSpreader(1,
Object[].class,
method.getParameterCount());
}
return handle;
} catch (...) {
throw new RuntimeException(...)
}
}

[n.b. theLookup is a suitable MethodHandles.Lookup that I finagle out of
class Lookup -- it's provenance doesn't really matter just now]

The problem appears to relate to the argument coercing that is done
under the handle.invoke call. Clearly a String[] argument does not match
the Object[].class parameter type occurring in the parameters returned
by method.getParameterTypes() and passed to findStatic.

I have tried varying this by applying an explicit spreader

if (isStatic) {
MethodHandle h =
theLookup.findStaticmethod.getDeclaringClass(),
method.getName(),
methodType);
handle = h.asSpreader(0,
Object[].class,
method.getParameterCount());
} else {
. . .
}

and using handle.invokeWithArguments in place of handle.invoke but that
produces the same result (indeed looking at the code I think the
original invoke call is already treated as a spread call.

I'm not really sure why this is operating the way it does by wrapping
the String[] input in an Object[]. It seems that it may perhaps be an
artefact of trying to combine inexact invocation with duck typing for
generics. Am I doing something wrong in the method handle lookup or in
the invocation? Or is this a corner case where invoke is performing the
wrong type of coercion?

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
f***@univ-mlv.fr
2016-11-11 15:09:35 UTC
Permalink
Hi Andrew,
welcome to the jungle of the java.lang.invoke API !

----- Mail original -----
Envoyé: Vendredi 11 Novembre 2016 14:43:53
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Hi John,
I'm reviving an old (by 2-3 weeks) conversation here because it seems
the right place to ask about how to deal with a disparity I found
between the use of reflection vs method handles. I'm have retained the
cc to jigsaw-dev because i) the conversation started there and ii) it
relates to the question of how well Lookups and MethdoHandles can stand
in for reflection. Perhaps that's too tenuous to merit a CC but I'm
adopting the excuse that those who are only interested in more direct
Jigsaw content can simply ignore the rest of this thread.
The disparity I am seeing looks like it might just be an argument for
Maurizio's new reflection API more than anything else but it may just be
that I am doing something wrong. I'll present the problem in as reduced
form as I can manage to extract from the Byteman implementation.
Byteman includes a test to ensure that the type checker allows duck
typing when a compatible array type is passed as input to a method. So,
we have a test method
public class TestArrayArgTypeCheck extends Test {
. . .
public void testArrayCall(String[] args)
{
log("inside testArrayCall");
}
}
where log is defined by the parent class Test.
I use a Byteman rule as follows
RULE test array arg type check
CLASS TestArrayArgTypeCheck
METHOD testArrayCall
AT ENTRY
BIND test : Test = $this;
IF TRUE
DO test.log("args : " + java.util.Arrays.asList($args) );
ENDRULE
This rule is 'injected' at the start of the code for method
testArrayCall -- it's not actually executed as inline bytecode, rather a
callout to the rule interpreter executes the rule (out of line bytecode
execution is also an option but I don't [yet?] need to add that
complexity to the mix). In case you are unfamiliar with how Byteman
operates I'll summarise its operation.
$this and $args name bindings for values passed into the interpreter,
the instance of TestArrayArgTypeCheck fielding the call to the
method (which Byteman knows is of type TestArrayArgTypeCheck)
the argument to testArrayCall (which Byteman knows is of type String[])
the BIND clause binds a /rule-local/ variable test (of type Test) to
the instance of TestArrayArgTypeCheck fielding the call to the method
the DO clause
passes $this to Arrays.asList()
pastes the result into a String
passes the String to Test.log
So, the test ensures that Byteman's type checker accepts that the
String[] argument passed to Arrays.asList(Object[]) is a legitimate
argument without throwing a type exception. It doesn't really matter
what the log output is but the test does check for an expected output
and that is where the disparity arises.
The call to the target method is made by the test code as follows
. . .
String[] ordinals = { "first", "second", "third" };
log("calling testArrayCall");
testArrayCall(ordinals);
log("called testArrayCall");
. . .
<log>
calling testArrayCall
args : [first, second, third]
inside testArrayCall
called testArrayCall
</log>
When I run this on JDK9 using a modified version of Byteman that relies
<log>
calling testArrayCall
inside testArrayCall
called testArrayCall
</log>
So, the String[] array appears to have been wrapped in an Object[]
before being passed on to Arrays.asList().
On JDK8[-] I use reflection to execute the method call. So, essentially
the code looks like this
class MethodExpression {
Method method;
Expression recipient;
List<Expression> arguments;
. . .
Object interpret(...)
. . .
Object recipientValue =
(recipient != null
? recipient.interpret(...)
: null);
int argCount = arguments.size();
Object[] argValues = new Object[argCount];
for (int i = 0; i < argCount; i++) {
argValues[i] = arguments.get(i).interpret(...);
}
. . .
return method.invoke(recipientValue, argValues);
The essential differences in the code that gets executed on JDK9
(ignoring that I am inlining code here from different branches) are as
class MethodExpression {
Method method;
MethodHandle handle = getMethodHandle(method);
List<Expression> arguments;
. . .
Object interpret(...)
. . .
Object recipientValue =
(recipient != null
? recipient.interpret(...)
: null);
int argCount = arguments.size();
Object[] argValues = new Object[argCount];
for (int i = 0; i < argCount; i++) {
argValues[i] = arguments.get(i).interpret(helper);
}
. . .
if (recipient == null) {
handle.invoke(argValues);
} else {
handle.invokeWithArguments(recipientValue, argValues);
}
MH.invokeWithArguments doesn't work like Method.invoke,
a MethodHandle is a function so any invoke* on a method handle is a function call,
there is no method on a method handle that separates the receiver (what you call the recipientValue) from the arguments when performing a call.

MH.invokeWithArguments takes an array of arguments but because it is specified as a varargs you may think that it works like Method.invoke, but it is a trap,
it takes the receiver and the arguments altogether into the same array.


so with a Stream it's something like:
Object[] argValues = Stream.concat(
Optional.ofNullable(recipient).map(r -> r.interpret(helper)).stream(),
Arrays.stream(arguments).map(r -> r.interpret(helper))
).toArray(Object[]::new);

. . .
handle.invokeWithArguments(argValues);
regards,
Andrew Dinn
regards,
Rémi
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
John Rose
2016-11-12 03:23:13 UTC
Permalink
Post by f***@univ-mlv.fr
MH.invokeWithArguments takes an array of arguments but because it is specified as a varargs you may think that it works like Method.invoke, but it is a trap,
it takes the receiver and the arguments altogether into the same array.
Thanks, Remi. That should fix the problem.

We thought a little bit about adding more overloadings to invokeWithArguments,
such as one that works like Method.invoke (one prepended argument).
The general case would be making invokeWithArguments be signature-polymorphic,
with an on-the-fly asSpreader transform on the way through.

But, such extra generality would simplify only a few use cases, and on the
other hand it would probably create plenty of confusion whenever the target
method is *also* a varargs method.

— John
f***@univ-mlv.fr
2016-11-12 10:56:16 UTC
Permalink
Envoyé: Samedi 12 Novembre 2016 04:23:13
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules &
open packages
Post by f***@univ-mlv.fr
MH.invokeWithArguments takes an array of arguments but because it is specified
as a varargs you may think that it works like Method.invoke, but it is a trap,
it takes the receiver and the arguments altogether into the same array.
Thanks, Remi. That should fix the problem.
We thought a little bit about adding more overloadings to invokeWithArguments,
such as one that works like Method.invoke (one prepended argument).
The general case would be making invokeWithArguments be signature-polymorphic,
with an on-the-fly asSpreader transform on the way through.
But, such extra generality would simplify only a few use cases, and on the
other hand it would probably create plenty of confusion whenever the target
method is *also* a varargs method.
The other solution is to have another non-polymorphic invoke method, invokeVirtualWithArguments that takes an instance and an array of objects like Method.invoke. But in my opinion, Method.invoke is a weird legacy format for a method call, separating the receiver from the other arguments is not natural for the VM, so this format should not have its own invoker. That said, it may help to rewrite the implementation of the reflection API to use method handles under the hood in order to remove the dedicated bytecode generator used by the current implementation.
— John
Rémi
Stephen Felts
2016-11-12 13:42:20 UTC
Permalink
I've been trying to get a javassist that works in all environments.

For a while, I created a multi release jar file that used Javassist 3.20 for pre-jdk9 and Javassist 3.21 for JDK9/Jake. The JDK team wasn't happy about this use of multi release jar files but it worked great.
Then 3.22.0-CR1 was released and I thought the problem was solved. This is working for pre-JDK9 and JDK-9 but it's now failing on Jake (build 143).

java.lang.NoSuchMethodError: java.lang.reflect.Layer.parent()Ljava/util/Optional;
[exec] at javassist.ModuleClassPath.<init>(ModuleClassPath.java:54)
[exec] at javassist.ModuleClassPath.<init>(ModuleClassPath.java:40)
[exec] at javassist.LoaderClassPath.<init>(LoaderClassPath.java:82)
[exec] at javassist.ClassPoolTail.appendSystemPath(ClassPoolTail.java:248)
[exec] at javassist.ClassPool.appendSystemPath(ClassPool.java:944)
[exec] at javassist.ClassPool.<init>(ClassPool.java:179)
[exec] at org.glassfish.hk2.xml.tools.Hk2XmlGenerator.initializeHk2XmlGenerator(Hk2XmlGenerator.java:86)
[exec] at org.glassfish.hk2.xml.tools.Hk2XmlGenerator.process(Hk2XmlGenerator.java:140)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:959)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:875)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.access$2100(JavacProcessingEnvironment.java:106)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1182)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1290)
[exec] at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1260)
[exec] at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:939)
[exec] at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:304)
[exec] at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:160)
[exec] at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:55)
[exec] at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:41)

I guess I will need to go back to my 3.2021 for now.
Claes Redestad
2016-11-12 13:53:31 UTC
Permalink
Hi,

this appears to be due to the implementation of #NonHierarchicalLayers
proposal which is now in jake:
http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers

In short Optional<Layer> Layer.parent() has been replaced by List<Layer>
Layer.parents()

/Claes
Post by Stephen Felts
I've been trying to get a javassist that works in all environments.
For a while, I created a multi release jar file that used Javassist 3.20 for pre-jdk9 and Javassist 3.21 for JDK9/Jake. The JDK team wasn't happy about this use of multi release jar files but it worked great.
Then 3.22.0-CR1 was released and I thought the problem was solved. This is working for pre-JDK9 and JDK-9 but it's now failing on Jake (build 143).
java.lang.NoSuchMethodError: java.lang.reflect.Layer.parent()Ljava/util/Optional;
[exec] at javassist.ModuleClassPath.<init>(ModuleClassPath.java:54)
[exec] at javassist.ModuleClassPath.<init>(ModuleClassPath.java:40)
[exec] at javassist.LoaderClassPath.<init>(LoaderClassPath.java:82)
[exec] at javassist.ClassPoolTail.appendSystemPath(ClassPoolTail.java:248)
[exec] at javassist.ClassPool.appendSystemPath(ClassPool.java:944)
[exec] at javassist.ClassPool.<init>(ClassPool.java:179)
[exec] at org.glassfish.hk2.xml.tools.Hk2XmlGenerator.initializeHk2XmlGenerator(Hk2XmlGenerator.java:86)
[exec] at org.glassfish.hk2.xml.tools.Hk2XmlGenerator.process(Hk2XmlGenerator.java:140)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:959)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:875)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.access$2100(JavacProcessingEnvironment.java:106)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1182)
[exec] at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1290)
[exec] at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1260)
[exec] at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:939)
[exec] at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:304)
[exec] at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:160)
[exec] at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:55)
[exec] at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:41)
I guess I will need to go back to my 3.2021 for now.
Alan Bateman
2016-11-12 14:17:23 UTC
Permalink
Post by Stephen Felts
I've been trying to get a javassist that works in all environments.
For a while, I created a multi release jar file that used Javassist 3.20 for pre-jdk9 and Javassist 3.21 for JDK9/Jake. The JDK team wasn't happy about this use of multi release jar files but it worked great.
Then 3.22.0-CR1 was released and I thought the problem was solved. This is working for pre-JDK9 and JDK-9 but it's now failing on Jake (build 143).
As Claes noted, this is a new API in JDK 9 that is not final yet so some
churn is expected. It will be easier for Javassist once the API changes
in jake get to JDK 9. In the mean-time, here is the issue in Javassist
JIRA tracking this Mandy created it a few days ago):
https://issues.jboss.org/browse/JASSIST-264

-Alan.
Stephen Felts
2016-11-12 14:42:46 UTC
Permalink
It turns out that the trick of using 3.21 doesn't work for Jake any more. So I guess there is no work-around for me on that platform and I won't be working on it.



-----Original Message-----
From: Alan Bateman
Sent: Saturday, November 12, 2016 9:17 AM
To: Stephen Felts; jigsaw-***@openjdk.java.net
Subject: Re: javassist broken on Jake
Post by Stephen Felts
I've been trying to get a javassist that works in all environments.
For a while, I created a multi release jar file that used Javassist 3.20 for pre-jdk9 and Javassist 3.21 for JDK9/Jake. The JDK team wasn't happy about this use of multi release jar files but it worked great.
Then 3.22.0-CR1 was released and I thought the problem was solved. This is working for pre-JDK9 and JDK-9 but it's now failing on Jake (build 143).
As Claes noted, this is a new API in JDK 9 that is not final yet so some churn is expected. It will be easier for Javassist once the API changes in jake get to JDK 9. In the mean-time, here is the issue in Javassist JIRA tracking this Mandy created it a few days ago):
https://issues.jboss.org/browse/JASSIST-264

-Alan.
Alan Bateman
2016-11-12 18:07:21 UTC
Permalink
Post by Stephen Felts
It turns out that the trick of using 3.21 doesn't work for Jake any more. So I guess there is no work-around for me on that platform and I won't be working on it.
You may find that older versions (maybe 3.20 and older) will work okay
with the jake builds, it's just the recent code where they are using
APIs under development that will need updates until those APIs are
final. Probably a topic for the Javassist mailing list rather than here.

-Alan
Andrew Dinn
2016-11-18 16:43:46 UTC
Permalink
Post by John Rose
Post by f***@univ-mlv.fr
MH.invokeWithArguments takes an array of arguments but because it is
specified as a varargs you may think that it works like Method.invoke,
but it is a trap,
it takes the receiver and the arguments altogether into the same array.
Thanks, Remi. That should fix the problem.
I'm not sure Remi's answer exactly addresses my question exactly as he
explained what happens when there is a receiver for the call and my
problem case is a static call. Anyway, I think I have got the picture
and now know what is wrong.
Post by John Rose
We thought a little bit about adding more overloadings to
invokeWithArguments,
such as one that works like Method.invoke (one prepended argument).
The general case would be making invokeWithArguments be
signature-polymorphic,
with an on-the-fly asSpreader transform on the way through.
But, such extra generality would simplify only a few use cases, and on the
other hand it would probably create plenty of confusion whenever the target
method is *also* a varargs method.
My problem was that the String[] argument I was passing to Arrays.asList
got wrapped in an enclosing Object[] array. That's because the method
has a varargs signature and when you call the method handle returned by
findStatic or findVirtual the arguments are expected to be provided
unwrapped in the invoke call. Whereas a method handle for a method with
a trailing Object[] argument (naturally) expects the argument to contain
the relevant Objects wrapped in an Object[].

This is where the disparity with reflective invocation occurs where the
varargs arguments must be provided pre-wrapped to the reflective invoke
call.

So, (excusing the expository-but-fake array literals used to show
supplied arguments) where I reflectively call

method.invoke(["first", "second", third"])

the corresponding handle call would be expected to be

handle.invokeWithArguments("first", "second", "third")

Similary, with a varargs instance method (let's call it
Foo.foo(Object...) where the reflective call would be

method.invoke(foo, ["first", "second", third"])

the corresponding handle call would be either

handle.invokeWithArguments(foo, "first", "second", "third")

or

handle.bindTo(foo).invokeWithArguments("first", "second", "third")

(I think that was what Remi was getting at :-)

Of course, since I package the arguments in generci code and then invoke
the method in code specific to JDK8[-] or JDK9[+] I am stuck with an
Object for the receiver (or null for a static call) and an Object[] for
the arguments. Hence, I cannot actually supply the arguments one by one
in the method handle call.

The fix I found is to detect the case where a method is varargs and in
that case use method asfixedArity() to convert the method handle
retrieved by findStatic or findVirtual to one which accepts the
arguments pre-wrapped.

So, where I currently have the following reflective code:

Object receiver = ...
Object[] args = ...

Method method = ...
boolean isStatic = ...

if (isStatic) {
method.invoke(null, args);
} else {
method.invoke(receiver, args);
}

I simply need to replace the latter 5 lines as

. . .
MethodType methodType =
MethodType.methodType(method.getReturnType(),
method.getParameterTypes());
MethodHandle handle;

if (isStatic) {
handle = theLookup.findStatic(method.getDeclaringClass(),
method.getName(),
methodType);
} else {
handle = theLookup.findVirtual(method.getDeclaringClass(),
method.getName(),
methodType);
}

if (method.isVarArgs()) {
handle = handle.asfixedArity();
}

if (isStatic) {
handle.invokeWithArguments(args);
} else {
handle.bindTo(receiver).invokeWithArguments(args);
}

This appears to have resolved my problem.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Remi Forax
2016-11-01 15:17:05 UTC
Permalink
----- Mail original -----
Envoyé: Mardi 1 Novembre 2016 15:39:01
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Post by John Rose
Can we clarify "privileged code"? Privileged like in a SecurityManager in a
PrivilegedAction for example, for privileged like only jdk internal code? Just
to see it black on white ;)
Good question: I mean the basic JDK platform implementation. Something deep in
java.base. Like Unsafe.
I don't see why this can't be a "regular" API though, rather than a
super-user sledgehammer every single time. If user code can be
statically granted access, and that user code can deliberately acquire a
narrowly-scoped object which can access those Lookups/*Handles, then
isn't that better than using Unsafe, which not only represents
unrestricted system-wide access, but can undermine even the JVM's
integrity if leaked?
It's better than Unsafe because as a user you have to grant access by using by example an annotation,
and you can specifies friends and/or what you want to export (only private/only package private, etc).

But if the API returns a Lookup object and a client code with granted access exposes that lookup, all bets are off.
--
- DML
Rémi
Jochen Theodorou
2016-11-01 15:32:41 UTC
Permalink
I must say I am partially to the whole story.

My main problem is code I do not have under control, but suddenly need
"extended" rights for. If this comes from a another module I will have
by default no chance of doing this anymore because of
#AwkwardStrongEncapsulation.

This proposal shows a way around it, but only if I instrument or
otherwise transform the bytecode to add the annotation. If I have that
level of control I can do all sorts of things. My simple problem here
is, that this may work for a framework you program against, but not for
a library that happens to have to process the class without the class
knowing anything about that library. And normally you do not want to
have a bytecode/class transformer, just to use the library properly. Not
to mention, that the point of time in which you get a hold on the
object, it might be much too late to do any modifications to the class....

Well... it makes me ask the question: Does #AwkwardStrongEncapsulation
impose additional limitations for the redefinition of already loaded
classes?

bye Jochen
Alan Bateman
2016-11-01 15:44:31 UTC
Permalink
Post by Jochen Theodorou
Well... it makes me ask the question: Does #AwkwardStrongEncapsulation
impose additional limitations for the redefinition of already loaded
classes?
No but if the methods in the new class redefinition are doing
setAccessible then it's same checks as would happen if the original
class bytes did the same thing.

-Alan
Jochen Theodorou
2016-11-01 15:55:38 UTC
Permalink
Post by Alan Bateman
Post by Jochen Theodorou
Well... it makes me ask the question: Does #AwkwardStrongEncapsulation
impose additional limitations for the redefinition of already loaded
classes?
No but if the methods in the new class redefinition are doing
setAccessible then it's same checks as would happen if the original
class bytes did the same thing.
If I got Remi's proposal right, I would "only" have to add the
annotation on the class by instrumentation to gain access, no need for
setAccessible

bye Jochen
Andrew Dinn
2016-11-01 14:57:45 UTC
Permalink
Post by John Rose
Post by Jochen Theodorou
Can we clarify "privileged code"? Privileged like in a
SecurityManager in a PrivilegedAction for example, for privileged
like only jdk internal code? Just to see it black on white ;)
Good question: I mean the basic JDK platform implementation.
Something deep in java.base. Like Unsafe.
Or indeed you might only require enough privilege to call the public
MethodHandles.Lookup constructor -- n.b. although that constructor is
public its package is not exported to the default module. Of course, you
can always arrange for it to be exported to one of /your/ modules.

Note that whatever mechanism is used to obtain Lookup instances this has
several advantages.

1) Once a privileged module is able to create Lookups (with suitably
constrained access) arranging for it to safely hand them over to your
framework code (and to no other code) when needed and with no more
access than is actually merited is relatively simple. Better still
arrange to hand over the requisite MethodHandles and keep the Lookup
instances secured.

Whereas

If you want to use reflection safely then you have to be able to call
setAccessible from a module which is an export target for the package of
the class whose member you want to access. In general you don't want to
export all possible reflection targets to all your framework code
(especially when it is in the default module). So, you end up having to
set up some privileged module and arrange for it call setAccessible on
behalf of your framework code (and no other code). Configuring all
potential exports to a privileged module in advance on the command line
will be tricky/verbose at best to and may well not even be possible. If
you have an agent that is capable of editing exports lists then you can
rewrite module exports to your privileged module on the fly but that is
also a lot of work (I know because I have tried it).

2) A MethodHandle created by a Lookup is not opaque to the compiler. So
this creates opportunities for the compiler to optimize the code that
employs them. Accessible fields/methods/constructors are opaque to the
compiler which makes it much less likely that optimization will be possible.

regards,


Andrew Dinn
-----------
Remi Forax
2016-11-01 15:11:50 UTC
Permalink
----- Mail original -----
Envoyé: Mardi 1 Novembre 2016 15:04:33
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages
Post by David M. Lloyd
1. It requires the target class to be initialized
2. It requires the target class to proactively donate MethodHandles or a Lookup
to the lookup class
Both of these can be overcome, though only by privileged code.
The privileged code would forge (uh, "mint") a legitimate Lookup to the
not-yet-initialized class.
A "Vault" meta-framework doesn't need to inject a Lookup donation statement into
anybody's <clinit>.
Instead, it needs to do the super-user operation of making a trusted lookup.
It must also fulfill the super-user *responsibility* of not leaking such
lookups, just using them in a predictable, rule-based manner.
— John
yes, being in java.lang.invoke is enough,
https://gist.github.com/forax/cab12206323ca2e5866c9f8e20ee6d40

Rémi
Nicolai Parlog
2016-12-04 18:59:05 UTC
Permalink
Hi!
Intrusive access to arbitrary packages of arbitrary modules by,
e.g., serialization frameworks or debugging tools, will still
require the use of sharp knives such as `--add-exports` or
`--add-opens` command-line options, the legacy unsupported
`sun.misc.Unsafe` API and related APIs, or JVM TI.
Would it make sense to add a command line switch to open up an entire
module? I can not say whether the risk of misuse (is anybody going to
use `--add-opens` any more?) is worth the benefits (not necessary to
iterate over some sort of package list, e.g. from serialized types, to
open multiple packages) but I thought I'll throw the idea out there.

so long ... Nicolai
Thanks to everyone who commented on the weak-modules proposal.
I've just posted a third proposal for
#ReflectiveAccessToNonExportedTypes, along with an analysis of the
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/
000430.html
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/00
0431.html
Further comments most welcome, as usual!
- Mark
- --

PGP Key:
http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509

Web:
http://codefx.org
a blog about software development
https://www.sitepoint.com/java
high-quality Java/JVM content
http://do-foss.de
Free and Open Source Software for the City of Dortmund

Twitter:
https://twitter.com/nipafx

Continue reading on narkive:
Loading...