Reflection: how does one access a protected member in a superclass reflectively?

[Rony G. Flatscher]

Finally having gained enough time to start rewriting the reflection part for a bridge between a
scripting language (ooRexx) and Java 9.

From past discussions on this list my view upon accessing members in superclasses that are protected
in Java 9 is possible as such protected members are regarded to be public (they are accessible) from
their subclasses.

To test against the new module system I have created three simple modules (only what is relevant to
this problem is given):

* "mod_A": defines a package "mtest1" with an abstract public class "Class01A" that implements an
interface "I01A", its "module-info.java" reads:
        "module mod_A { exports mtest1; }"

* "mod_B": defines a package "mtest2" with a public "Class02A" which extends "mtest1.Class01A" and
implements an interface "I02A" which extends "mtest1.I01A", its "module-info.java" reads:
        "module mod_B {  requires mod_A;     exports mtest2 to mod_C;  }

* "mod_C": defines a package "mtest2" with a public "Class03A" which extends "mtest2.Class02A",
its "module-info.java" reads:
"module mod_C { exports mtest3; requires mod_B; }"

The code doing the reflection resides in the unnamed module for the time being (it eventually will
be part of a module).

Running the script code is done against the following Java settings:

-cp "%CLASSPATH%" --module-path F:\java9modules\out --add-modules mod_A,mod_B,mod_C

In the first round reflecting Fields is used as a testbed. The reflection code at this stage is able
to successfully skip over the closed "mod_B" module and arrive at "mod_A" classes. However,
reflecting for "Class03A" instance is not able to access the defined *protected* static field
"myClassName" in the superclass "Class01A" (the String value of that static field is:
"class-mtest1.Class01A")!

The debug output with the trailing stack trace for the runtime error is:

about to load class [mtest3.Class03A]
         loaded, clz~toString: [class mtest3.Class03A]
         [***@4973813a] package: [package mtest3] module: [module mod_C]
***@6fb0d3ed -> reflect(rru):
***@16f7c8c1, field values:
--->    rajo          =[***@24a35978]
        invocationType=[GET_FIELD_VALUE]
        reflectionType=[REFLECT_FIELD]
        bStrict       =[false]
        beanName      =[***@1563da5]
        bean          =[***@1563da5] instanceof Class? [false]
        beanClz       =[class mtest3.Class03A]
        memberName    =[MYCLASSNAME]
        rexxArgs[]    =[[Ljava.lang.String;@df27fae], rexxArgs.length   =[3]: [GETFIELDVALUE,
***@1563da5, MYCLASSNAME]
        tmpRexxArgs[] =[[Ljava.lang.String;@704921a5], tmpRexxArgs.length=[0]: []
        funcArgs[]    =[[Ljava.lang.Object;@727803de], funcArgs.length   =[0]: []
        bReturnJSO    =[false]
        bTryCoercions =[true]
<---
\\// RexxReflectJava9.reflectField: (1) in tmpClz.getSuperclass() loop:
tmpClz=[mtest2.Class02A], BEFORE isExported()
\\// RexxReflectJava9.reflectField: package of tmpClz not EXPORTED, hence SKIPPING tmpClz:
[mtest2.Class02A]
\\// RexxReflectJava9.reflectField: now checking                               --->
tmpClz=[mtest1.Class01A]
\\// RexxReflectJava9.reflectField: (1) in tmpClz.getSuperclass() loop:
tmpClz=[mtest1.Class01A], BEFORE isExported()
//\\ RexxReflectJava9.reflectField: (2) in tmpClz.getSuperclass() loop:
tmpClz=[mtest1.Class01A], AFTER  isExported()
RexxReflectJava9.processField(), arrived: -> [GET_FIELD_VALUE], tmpField=[protected static
java.lang.String mtest1.Class01A.myClassName]: field=[MYCLASSNAME] in
object=[***@1563da5]
RexxReflectJava9.processField(): => [GET_FIELD_VALUE]: found field=[MYCLASSNAME] in
object=[***@1563da5/***@1563da5]
oops GET-operation: tmpField "myClassName" caused exception "java.lang.IllegalAccessException:
class org.rexxla.bsf.engines.rexx.RexxReflectJava9 cannot access a member of class
mtest1.Class01A (in module mod_A) with modifiers "protected static""
java.lang.reflect.InaccessibleObjectException: Unable to make field protected static
java.lang.String mtest1.Class01A.myClassName accessible: module mod_A does not "opens  mtest1"
to unnamed module @16022d9d
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Unknown Source)
        at java.base/java.lang.reflect.Field.setAccessible(Unknown Source)
        at org.rexxla.bsf.engines.rexx.RexxReflectJava9.processField(RexxReflectJava9.java:294)
        at org.rexxla.bsf.engines.rexx.RexxReflectJava9.reflectField(RexxReflectJava9.java:113)
        at org.rexxla.bsf.engines.rexx.RexxReflectJava9.reflect(RexxReflectJava9.java:59)
        at org.rexxla.bsf.engines.rexx.RexxAndJava.javaCallBSF(RexxAndJava.java:3247)
        at org.rexxla.bsf.engines.rexx.RexxAndJava.javaCallBSF(RexxAndJava.java:4163)
        at org.rexxla.bsf.engines.rexx.RexxAndJava.jniRexxRunProgram(Native Method)
        at org.rexxla.bsf.engines.rexx.RexxEngine.apply(RexxEngine.java:1153)
        at org.rexxla.bsf.RexxDispatcher.main(RexxDispatcher.java:158)

Doing the comparable operation - accessing a field named "myClassName" in Java code from
"mtest3.Class03a" (in the main method) succeeds!

Here is the output of running "mtest3.Class3A" for comparison:

F:\java9modules>java --module-path out -m mod_C/mtest3.Class03A
class mtest3.Class03A.main() ...
       getMyClassNameStatic()=[class-mtest1.Class01A]
       myClassName           =[class-mtest1.Class01A]
       getMyName1()          =[from Class01A (static)]
     o.getMyName2()          =[from Class02A (instance)]
     o.myClassName           =[class-mtest1.Class01A]


So the question is, how can I reflectively access "mtest1.Class01A" static protected field
"myClassName" from "mtest3.Class03a" in Java 9?

The Java code for "mtest1.Class01A" (in "mod_A"), "mtest2.Class02A" (in "mod_B") and
"mtest3.Class03A" (in "mod_C") is given below.

---rony

"mod_A":

package mtest1;

abstract public class Class01A implements I01A
{
    protected static String myClassName = "class-mtest1.Class01A";
    protected static String myName1 = "from Class01A (static)";
    protected String myName2 = "from Class01A (instance)";

     public static void main (String args[]) {
         System.err.println(Class01A.class+".main() ...");
     }

     public static String getMyClassNameStatic() // static method in interface
     {
          return myClassName;
     }

     abstract public String getMyClassName() ; // default method in interface

     static protected String getMyName1()
     {
          return myName1;
     }

     abstract protected String getMyName2();
}


"mod_B":

package mtest2;

public class Class02A extends mtest1.Class01A implements I02A
{
     public static String myName1 = "from Class02A (static)";
     public String myName2 = "from Class02A (instance)";

public static void main (String args[]) {
          System.err.println(Class02A.class+".main() ...");

          System.err.println(" getMyClassNameStatic()=["+ getMyClassNameStatic()+"]");
          System.err.println(" getMyName1() =["+ getMyName1() +"]");
          Class02A o=new Class02A();
          System.err.println(" o.getMyName2() =["+o.getMyName2() +"]");
          System.err.println(" o.getMyClassName() =["+o.getMyClassName() +"]");
     }

     public String getMyClassName()
     {
          return myClassName;
     }

     protected String getMyName2()
     {
          return myName2;
     }
}

"mod_C":

package mtest3;

public class Class03A extends mtest2.Class02A
{
    public static void main (String args[]) {
        System.err.println(Class03A.class+".main() ...");

        System.err.println("       getMyClassNameStatic()=["+  getMyClassNameStatic()+"]");
        System.err.println("       myClassName           =["+  myClassName           +"]");
        System.err.println("       getMyName1()          =["+  getMyName1()          +"]");

        Class03A o=new Class03A();
        System.err.println("     o.getMyName2()          =["+o.getMyName2()          +"]");
        System.err.println("     o.myClassName           =["+o.myClassName           +"]");
    }

}

for completeness the Interface classes:

"mod_A":

 package mtest1;

public interface I01A
{
    static public String  getMyClassNameStatic()    // static method in interface
    {
        System.err.println("\t<<<(static public getMyClassNameStatic() method from
["+mtest1.I01A.class+"])>>>");
        return "interface-mtest1.I01A"; // myClassName;
    }

    default public String getMyClassName()          // default method in interface
    {
        System.err.println("\t<<<(default public getMyClassName() method from
["+mtest1.I01A.class+"])>>>");
        return "interface-mtest1.I01A"; // myClassName;
    }
}

"mod_B":

package mtest2;

public interface I02A extends mtest1.I01A
{
    static public String  getMyClassNameStatic()    // static method in interface
    {
        System.err.println("\t<<<(static public getMyClassNameStatic() method from
["+mtest2.I02A.class+"])>>>");
        return "interface-mtest2.I02A"; // myClassName;
    }

    default public String getMyClassName()          // default method in interface
    {
        System.err.println("\t<<<(default public getMyClassName() method from
["+mtest2.I02A.class+"])>>>");
        return "interface-mtest2.I02A"; // myClassName;
    }
}

[Alan Bateman]

Post by Rony G. Flatscher
So the question is, how can I reflectively access "mtest1.Class01A" static protected field
"myClassName" from "mtest3.Class03a" in Java 9?

The scenario in your mail is complicated but if I read it correctly then
I would expect this code in mtest3.Class03A (mod_C) to work:
Class.forName("mtest1.Class01A").getDeclaredField("myClassName").get(null);

I can't tell if the code in the unnamed module extends mtest1.Class01A
or not. If it does then the same code should work, if it's not a
sub-class then you should see IllegalAccessException being thrown.

Your traces show InaccessibleObjectException being thrown so I think you
are attempting to call setAccessible(true) to suppress the access check.
You can't do on that protected members when the package is not open for
deep reflection. In the example, mod_A would need to `opens mtest1` to
allow this.

-Alan.

[Rony G. Flatscher]

Post by Rony G. Flatscher
So the question is, how can I reflectively access "mtest1.Class01A" static protected field
"myClassName" from "mtest3.Class03a" in Java 9?

The scenario in your mail is complicated but if I read it correctly then I would expect this code
Class.forName("mtest1.Class01A").getDeclaredField("myClassName").get(null);

The problem here is, that no code from any of the modules is doing the reflection. Rather the
reflection comes from packages on the class path, hence from the unnamed module.

I can't tell if the code in the unnamed module extends mtest1.Class01A or not. If it does then the
same code should work, if it's not a sub-class then you should see IllegalAccessException being
thrown.

The excercise here is as follows:

* there are classes in "mod_A" which get extended in "mod_B" and once more extended in "mod_C". 
* Hence "mod_B" requires "mod_A", and "mod_C" requires "mod_B", where "mod_B" exports to "mod_C" only
* These are the two chains of class extensions, one (mtest1.Class01A) defines an abstract public
class with protected members, one (mtest1.Class01B) defines a public class with public members,
hence:
o one chain is: 
+ in "mod_A" there is the class "abstract public class mtest1.Class01A", which contains a
protected static field named "myClassName",
+ in "mod_B" there is the class "public class Class02A extends mtest1.Class01A", has no
static field named "myClassName",
+ in "mod_C" there is the class "publi class Class03A extends mtest2.Class02A", has no
static field named "myClassName.
o another chain is:
+ in "mod_A" there is the class "abstract public class mtest1.Class01B", which contains a
protected static field named "myClassName",
+ in "mod_B" there is the class "public class Class02B extends mtest1.Class01B", has no
static field named "myClassName",
+ in "mod_C" there is the class "public class Class03B extends mtest2.Class02B", has no
static field named "myClassName.

Two scenarios, one works, one causes the "IllegalAccessException":

* works: creating an instance of "mtest3.Class03B" reflectively and then getting the value of the
public static field "myClassName" in "mtest1.Class01B", works reflectively!

* "IllegalAccessException": creating an instance of "mtest3.Class03A" reflectively and then
getting the value of the public static field "myClassName" in "mtest1.Class01A", causes the
"IllegalAccessException";

o Interestingly,
+ the first error message is: "java.lang.IllegalAccessException: class
org.rexxla.bsf.engines.rexx.RexxReflectJava9 cannot access a member of class
mtest1.Class01A (in module mod_A) with modifiers "protected static"

+ in the catch block then trying a "setAccessible(true)" yields another
"IllegalAccessException" now with the following error message:
"java.lang.reflect.InaccessibleObjectException: Unable to make field protected static
java.lang.String mtest1.Class01A.myClassName accessible: module mod_A does not "opens"

Your traces show InaccessibleObjectException being thrown so I think you are attempting to call
setAccessible(true) to suppress the access check. You can't do on that protected members when the
package is not open for deep reflection. In the example, mod_A would need to `opens mtest1` to
allow this.

And that is exactly the point. "mtest1.Class01A" is a superclass of "mtest3.Class03A". Therefore the
protected members in the superclass "mtest1.Class01A" are public for its subclasses like
"mtest3.Class03A".

This is a very basic and important property of the Java language.

Therefore I assume that this is a bug in the current implementation of Java 9, but want to know for
sure.

---

I asked this very question ("are protected members in superclasses regarded to be public for
subclasses") almost nine months ago and it was agreed that "protected" in superclasses become
effectively "public" for subclasses in Java 9 as well, so it was clear that when adjusting the
bridge to Java 9 everything would work as in the past for the users. [For the past 17 years I have
always adhered to these Java semantics in the Rexx/ooRexx-Java bridge, users of the bridge have only
been able to access public classes and public members and protected classes and protected members of
superclasses. There has never been a need for any deep reflection (nor "desire" to do so) as package
private and private members have to be respected to be "private".]

---rony

[Alan Bateman]

Post by Rony G. Flatscher
* there are classes in "mod_A" which get extended in "mod_B" and
once more extended in "mod_C".
* Hence "mod_B" requires "mod_A", and "mod_C" requires "mod_B",
where "mod_B" exports to "mod_C" only
* These are the two chains of class extensions, one
(mtest1.Class01A) defines an abstract public class with protected
members, one (mtest1.Class01B) defines a public class with public
+ in "mod_A" there is the class "abstract public class
mtest1.Class01A", which contains a protected static field
named "myClassName",
+ in "mod_B" there is the class "public class Class02A
extends mtest1.Class01A", has no static field named
"myClassName",
+ in "mod_C" there is the class "publi class Class03A
extends mtest2.Class02A", has no static field named
"myClassName.
+ in "mod_A" there is the class "abstract public class
mtest1.Class01B", which contains a protected static field
named "myClassName",

If I read the earlier text correctly then Class01B.myClassName is public.

Post by Rony G. Flatscher
+ in "mod_B" there is the class "public class Class02B
extends mtest1.Class01B", has no static field named
"myClassName",
+ in "mod_C" there is the class "public class Class03B
extends mtest2.Class02B", has no static field named
"myClassName.
* works: creating an instance of "mtest3.Class03B" reflectively and
then getting the value of the public static field "myClassName" in
"mtest1.Class01B", works reflectively!

I assume the caller is org.rexxla.bsf.engines.rexx.RexxReflectJava9 on
the class path. Module mod_A exports mtest1, mtest1.Class1B is public,
code on the class path can reflect on Class1B's public members.

Post by Rony G. Flatscher
* "IllegalAccessException": creating an instance of
"mtest3.Class03A" reflectively and then getting the value of the
public static field "myClassName" in "mtest1.Class01A", causes the
"IllegalAccessException";

I think you mean "protected static".

Post by Rony G. Flatscher
*
o Interestingly,
"java.lang.IllegalAccessException: class
org.rexxla.bsf.engines.rexx.RexxReflectJava9 cannot access
a member of class mtest1.Class01A (in module mod_A) with
modifiers "protected static"

This looks correct, RexxReflectJava9 is not a sub-class of
mtest1.Class01A so it can't expect to reflect on Class01A's protected
members.

Post by Rony G. Flatscher
+ in the catch block then trying a "setAccessible(true)"
yields another "IllegalAccessException" now with the
"java.lang.reflect.InaccessibleObjectException: Unable to
make field protected static java.lang.String
mtest1.Class01A.myClassName accessible: module mod_A does

The InaccessibleObjectException looks correct too.

Post by Rony G. Flatscher
And that is exactly the point. "mtest1.Class01A" is a superclass of
"mtest3.Class03A". Therefore the protected members in the superclass
"mtest1.Class01A" are public for its subclasses like "mtest3.Class03A".

Do you have an issue here? If code is added to mod_C/mtest3.Class03A to
reflect on protected members of mod_A/mtest1.Class01A then I would
expect it to work. Furthermore, I would expect setAccessible(true) to
succeed on protected static members too (not instance members of course).

That said, the scenario that you've shown is code in RexxReflectJava9
doing the reflection. It's not in a sub-class of mtest1.Class01A so it
should not have any access to its protected members.

-Alan

[Rony G. Flatscher]

Post by Alan Bateman

Post by Rony G. Flatscher
* there are classes in "mod_A" which get extended in "mod_B" and once more extended in "mod_C". 
* Hence "mod_B" requires "mod_A", and "mod_C" requires "mod_B", where "mod_B" exports to
"mod_C" only
* These are the two chains of class extensions, one (mtest1.Class01A) defines an abstract
public class with protected members, one (mtest1.Class01B) defines a public class with public
o one chain is: 
+ in "mod_A" there is the class "abstract public class mtest1.Class01A", which contains
a protected static field named "myClassName",
+ in "mod_B" there is the class "public class Class02A extends mtest1.Class01A", has no
static field named "myClassName",
+ in "mod_C" there is the class "publi class Class03A extends mtest2.Class02A", has no
static field named "myClassName.
+ in "mod_A" there is the class "abstract public class mtest1.Class01B", which contains
a protected static field named "myClassName",

If I read the earlier text correctly then Class01B.myClassName is public.

Yes.

Post by Alan Bateman

Post by Rony G. Flatscher
+ in "mod_B" there is the class "public class Class02B extends mtest1.Class01B", has no
static field named "myClassName",
+ in "mod_C" there is the class "public class Class03B extends mtest2.Class02B", has no
static field named "myClassName.
* works: creating an instance of "mtest3.Class03B" reflectively and then getting the value of
the public static field "myClassName" in "mtest1.Class01B", works reflectively!

I assume the caller is org.rexxla.bsf.engines.rexx.RexxReflectJava9 on the class path. Module
mod_A exports mtest1, mtest1.Class1B is public, code on the class path can reflect on Class1B's
public members.

Post by Rony G. Flatscher
* "IllegalAccessException": creating an instance of "mtest3.Class03A" reflectively and then
getting the value of the public static field "myClassName" in "mtest1.Class01A", causes the
"IllegalAccessException";

I think you mean "protected static".

Yes, sorry.

Post by Alan Bateman

Post by Rony G. Flatscher
*
o Interestingly,
+ the first error message is: "java.lang.IllegalAccessException: class
org.rexxla.bsf.engines.rexx.RexxReflectJava9 cannot access a member of class
mtest1.Class01A (in module mod_A) with modifiers "protected static"

This looks correct, RexxReflectJava9 is not a sub-class of mtest1.Class01A so it can't expect to
reflect on Class01A's protected members.

Post by Rony G. Flatscher
+ in the catch block then trying a "setAccessible(true)" yields another
"java.lang.reflect.InaccessibleObjectException: Unable to make field protected static
java.lang.String mtest1.Class01A.myClassName accessible: module mod_A does not

The InaccessibleObjectException looks correct too.

Post by Rony G. Flatscher
And that is exactly the point. "mtest1.Class01A" is a superclass of "mtest3.Class03A". Therefore
the protected members in the superclass "mtest1.Class01A" are public for its subclasses like
"mtest3.Class03A".

Do you have an issue here? If code is added to mod_C/mtest3.Class03A to reflect on protected
members of mod_A/mtest1.Class01A then I would expect it to work. Furthermore, I would expect
setAccessible(true) to succeed on protected static members too (not instance members of course).
That said, the scenario that you've shown is code in RexxReflectJava9 doing the reflection. It's
not in a sub-class of mtest1.Class01A so it should not have any access to its protected members.

Well that is probably the core of the problem: who needs to be the subclass, the reflector or the
object to be reflected upon (whose inheritance tree is being walked up towards the root class, and
if public or protected members are found reflectively accessed)?

Or conceptually: the "mtest3.Class03A object" is clearly from a subclass of "mtest1.Class01A". Why
should it then not be allowed to access the public and protected members in the superclass
"mtest1.Class01A" via reflection on behalf of the "mtest3.Class03A object", which gets supplied in
the reflective access and which can be used to check whether the access would be legal or not?

---rony

[John Rose]

Post by Rony G. Flatscher
Well that is probably the core of the problem: who needs to be the subclass, the reflector or the
object to be reflected upon (whose inheritance tree is being walked up towards the root class, and
if public or protected members are found reflectively accessed)?

Quick comment:

With the core reflection API, the reflector's permissions
are derived from the reflector, not the class being investigated.

This works fine when a class reflects itself but is not so good for frameworks.

The reflection API in java.lang.invoke, though more limited, does
support rights delegation through the Lookup object. This means
that your "patient" class doesn't need to do the reflection work;
it can merely create a private lookup in itself and hand it to the
framework, which can do the reflection on behalf of the patient.

You can even mix the old and new mechanisms (a point I don't
think I've seen on this thread, although I haven't read it all).
Given a private lookup object in some patient P, a framework
class F can create a method handle on a core reflection API
point that is @CallerSensitive (like Class.forName, etc. etc.)
which allows F to call that API point with the same rights as
P, not F.

HTH
— John

[Jochen Theodorou]

Post by John Rose

Post by Rony G. Flatscher
Well that is probably the core of the problem: who needs to be the subclass, the reflector or the
object to be reflected upon (whose inheritance tree is being walked up towards the root class, and
if public or protected members are found reflectively accessed)?

With the core reflection API, the reflector's permissions
are derived from the reflector, not the class being investigated.
This works fine when a class reflects itself but is not so good for frameworks.

well said.

Post by John Rose
The reflection API in java.lang.invoke, though more limited, does
support rights delegation through the Lookup object. This means
that your "patient" class doesn't need to do the reflection work;
it can merely create a private lookup in itself and hand it to the
framework, which can do the reflection on behalf of the patient.

which is a big deal of a problem if the class, that is supposed to be
exposed to the framework is not under the control of the framework. And
there is no good solution (rewriting the bytecode for example is no good
solution)

Post by John Rose
You can even mix the old and new mechanisms (a point I don't
think I've seen on this thread, although I haven't read it all).
Given a private lookup object in some patient P, a framework
class F can create a method handle on a core reflection API
which allows F to call that API point with the same rights as
P, not F.

Groovy is using that kind of logic in the initial invokedynamic
implementation. The problem is that Reflection allowed much more.
Frameworks (and whole languages) did grow around these capabilities. In
Java9 they have no alternative implementation anymore. And I'd like to
especially mention @CallerSensitive... This makes it very very difficult
for frameworks. The more general, the more you have to work around that
kind of logic - up to the point where no workaround is possible anymore
(short of opening for deep reflection, living with thousands of warnings
and an in acceptable commandline usage).

bye Jochen

[John Rose]

which is a big deal of a problem if the class, that is supposed to be exposed to the framework is not under the control of the framework. And there is no good solution (rewriting the bytecode for example is no good solution)

Yep. We didn't solve all those problems in 9 since they are
very complicated trade-offs, and the "new kid on the block" of
enforced encapsulation is taking ground previously occupied
by the "neighborhood gang" of legitimate frameworks. So we
need a way to give those frameworks what they need, in a way
that still make encapsulation preservable and checkable.

(My mental model for this tradeoff is, how can I AOT-compile as
much as possible, while still leaving open some legitimate hooks
for frameworks to operate? I.e., enabling AOT and framework
intercession at the same time is what winning looks like.)

The privateLookup API will allow a framework to get full-power
"guest" status in an arbitrary uncooperative class. At least,
that's the advertisement. Will that help, or is there some bad
fine-print to that API?

[Rony G. Flatscher]

Post by John Rose

Post by Rony G. Flatscher
Well that is probably the core of the problem: who needs to be the subclass, the reflector or the
object to be reflected upon (whose inheritance tree is being walked up towards the root class, and
if public or protected members are found reflectively accessed)?

With the core reflection API, the reflector's permissions
are derived from the reflector, not the class being investigated.
This works fine when a class reflects itself but is not so good for frameworks.
The reflection API in java.lang.invoke, though more limited, does
support rights delegation through the Lookup object.  This means
that your "patient" class doesn't need to do the reflection work;
it can merely create a private lookup in itself and hand it to the
framework, which can do the reflection on behalf of the patient.
You can even mix the old and new mechanisms (a point I don't
think I've seen on this thread, although I haven't read it all).
Given a private lookup object in some patient P, a framework
class F can create a method handle on a core reflection API
which allows F to call that API point with the same rights as
P, not F.

Thank you for the pointer, reading through
<https://docs.oracle.com/javase/9/docs/api/java/lang/invoke/MethodHandles.Lookup.html> it seems that
the access problem cannot be solved with that package either.

The problem at hand is the fact that the framework in question ("BSF4ooRexx", a bridge between
ooRexx implemented in C++ and Java) is *not* part of any class hierarchy in the module(s) nor
related to any specific module.

A simple example ooRexx program using a Java class that gets intertwined with the three test modules
mod_A (exports mtest1), mod_B (requires mod_A, exports mtest2 to mod_C), mod_C (exports mtest3):

o=.bsf~new("mtest3.Class03A")             -- create Java object, get and assign proxy ooRexx object
say "o:" o "o~myClassName:" o~myClassName -- get (static) field value in "mtest1.Class01A",
accessible via inheritance rules

::requires BSF.CLS   -- direct interpreter to load the Java bridge

The basic logic is as follows:

* load the class "mtest3.Class03A", reflectively get a constructor and create a Java object, store
it in a Java repository return the key to allow looking up the object
* ooRexx creates an ooRexx proxy object which will forward all unknown messages to the bridge,
which in turn determines what is sought and uses reflection to carry out the desired operation
* the second statement shows the ooRexx object name (the key into the Java registry) and uses the
bridge to have it look up the field named "myClassName" (there is no message by that name on the
ooRexx side, so the bridge is used to lookup the proxied Java object; a message is indicated by
the message operator '~', the tilde, where the receiver is placed left of it and the message
name right of it)

The reflection logic in the bridge is (simplified) as follows:

* use the object's 'o' class and look up its declared methods or fields, analyze further, if
member is 'public', otherwise

o iterate over all of its superclasses looking up in each class all declared methods or
fields, analyze further, if member is 'public' *or* 'protected' (looking up a superclass
'protected' is regarded to be 'public' for the subclass) and, if a matching member is found,
carry out the operation (in this case a Field.get()) and return its value

This way it is assured that the users of the bridge are never able to get at private or package
private members (nor that they are able to invoke a protected member in the object's type itself).

This logic has been working since Java 1.1 (sic!) throughout all Java versions using Java's
reflection infrastructure.

Java 9 currently breaks this.

Is there currently (in the current implementation) any other means to achieve the same, mandatory
needed functionality in Java 9 that I could use?

---

The current implementation seems to do the following: it checks whether a class is from an  exported
package, if not it creates an exception. If the class is exported it then checks whether the
reflected member is public, if not it creates an exception.

What I would have expected is this: everything like the current implementation, but in the second
step, if the member is not public, check whether the object's class for which the reflection takes
place is a subclass of the class that is being checked and if so, allow protected members to be
accessed as well. Hence: in the example presented, the packages 'mtest1' and 'mtest3' are exported,
the class 'mtest3.Class03A' extends 'mtest2.Class02A' which extends 'mtest1.Class01A'. Therefore the
object of type 'mtest3.Class03A' is a subclass of 'mtest1.Class01A', hence as 'mtest1' is exported
all 'mtest1.Class01A' public *and* protected members should be accessible to the object of the
subtype 'mtest3.Class03A' via reflection in Java 9 as well.

---rony

[Alan Bateman]

Post by Rony G. Flatscher
* use the object's 'o' class and look up its declared methods or
fields, analyze further, if member is 'public', otherwise
o iterate over all of its superclasses looking up in each class
all declared methods or fields, analyze further, if member is
'public' *or* 'protected' (looking up a superclass 'protected'
is regarded to be 'public' for the subclass) and, if a
matching member is found, carry out the operation (in this
case a Field.get()) and return its value
This way it is assured that the users of the bridge are never able to
get at private or package private members (nor that they are able to
invoke a protected member in the object's type itself).

Accessibility has been significantly upgraded in Java SE 9. "public"
used to imply accessible to everyone, now we have "public to everyone",
"public to friend modules" or "public within a module". If you want the
above logic to work with modules then it will need to be updated to
handle classes in named modules. It will essentially amount to
overhauling the above to find the accessible members rather the public
members (in public classes).

BTW: On protected members then I assume they have never been accessible
to the bridge, maybe the bridge is using setAccessible to suppress the
access checks for those cases?

Post by Rony G. Flatscher
---
The current implementation seems to do the following: it checks
whether a class is from an  exported package, if not it creates an
exception. If the class is exported it then checks whether the
reflected member is public, if not it creates an exception.
What I would have expected is this: everything like the current
implementation, but in the second step, if the member is not public,
check whether the object's class for which the reflection takes place
is a subclass of the class that is being checked and if so, allow
protected members to be accessed as well. Hence: in the example
presented, the packages 'mtest1' and 'mtest3' are exported, the class
'mtest3.Class03A' extends 'mtest2.Class02A' which extends
'mtest1.Class01A'. Therefore the object of type 'mtest3.Class03A' is a
subclass of 'mtest1.Class01A', hence as 'mtest1' is exported all
'mtest1.Class01A' public *and* protected members should be accessible
to the object of the subtype 'mtest3.Class03A' via reflection in Java
9 as well.

You should find that the protected members are accessible in the
sub-class (mtest3.Class03A in this case). They won't be accessible to
the bridge code of course (as least not without some kind of rights
delegation as John mentioned).

-Alan

[jeffrey kutcher]

Disclaimer:
I don't have access to the full picture but wanted to throw this out there. Maybe it will be useful. Maybe you've already considered doing something like this and shot it down a long time ago. I don't know if there are show stoppers and I don't have the end goal in mind. However it seems like it might be a useful way to simplify what I've read so far. The following suggestion might untangle the web which appears to be growing though it may be too late to introduce.
Suggestion:
A closure might provide the plumbing you're looking for rather than using public/private/protected modifiers which seems to be gaining overloaded status. public/private/protected modifiers should go back to their original meaning and kept simple. A closure would allow classes by default, no or default access/authority/permission/security unless being owned by a container in which case would assume it's owner's access. This might eliminate the expanding access issues your finding and be more robust in the process, maybe even collapse the code. You'll have to provide an environment to accomplish this.
If a class "has a" or "is a" sub-class, the sub-class would take on the access of the parent class until such time as it deferred that access for more specific purposes. If not, any "has a" or "is a" sub-class would assume the role of its owner.
"has a" and "is a" sub-classes would have to reference their owner's environment to gain access permission otherwise no (or default) access would be granted.
Coming up with the right convention should prove to be elegant and introduce little to no overhead.

  * use the object's 'o' class and look up its declared methods or
    fields, analyze further, if member is 'public', otherwise
      o iterate over all of its superclasses looking up in each class
        all declared methods or fields, analyze further, if member is
        'public' *or* 'protected' (looking up a superclass 'protected'
        is regarded to be 'public' for the subclass) and, if a
        matching member is found, carry out the operation (in this
        case a Field.get()) and return its value
This way it is assured that the users of the bridge are never able to
get at private or package private members (nor that they are able to
invoke a protected member in the object's type itself).

Accessibility has been significantly upgraded in Java SE 9. "public"
used to imply accessible to everyone, now we have "public to everyone",
"public to friend modules" or "public within a module". If you want the
above logic to work with modules then it will need to be updated to
handle classes in named modules. It will essentially amount to
overhauling the above to find the accessible members rather the public
members (in public classes).

BTW: On protected members then I assume they have never been accessible
to the bridge, maybe the bridge is using setAccessible to suppress the
access checks for those cases?

---
The current implementation seems to do the following: it checks
whether a class is from an  exported package, if not it creates an
exception. If the class is exported it then checks whether the
reflected member is public, if not it creates an exception.
What I would have expected is this: everything like the current
implementation, but in the second step, if the member is not public,
check whether the object's class for which the reflection takes place
is a subclass of the class that is being checked and if so, allow
protected members to be accessed as well. Hence: in the example
presented, the packages 'mtest1' and 'mtest3' are exported, the class
'mtest3.Class03A' extends 'mtest2.Class02A' which extends
'mtest1.Class01A'. Therefore the object of type 'mtest3.Class03A' is a
subclass of 'mtest1.Class01A', hence as 'mtest1' is exported all
'mtest1.Class01A' public *and* protected members should be accessible
to the object of the subtype 'mtest3.Class03A' via reflection in Java
9 as well.

You should find that the protected members are accessible in the
sub-class (mtest3.Class03A in this case). They won't be accessible to
the bridge code of course (as least not without some kind of rights
delegation as John mentioned).

-Alan

[Rony G. Flatscher]

Post by Rony G. Flatscher
* use the object's 'o' class and look up its declared methods or fields, analyze further, if
member is 'public', otherwise
o iterate over all of its superclasses looking up in each class all declared methods or
fields, analyze further, if member is 'public' *or* 'protected' (looking up a superclass
'protected' is regarded to be 'public' for the subclass) and, if a matching member is
found, carry out the operation (in this case a Field.get()) and return its value
This way it is assured that the users of the bridge are never able to get at private or package
private members (nor that they are able to invoke a protected member in the object's type itself).

Accessibility has been significantly upgraded in Java SE 9. "public" used to imply accessible to
everyone, now we have "public to everyone", "public to friend modules" or "public within a
module".  If you want the above logic to work with modules then it will need to be updated to
handle classes in named modules. It will essentially amount to overhauling the above to find the
accessible members rather the public members (in public classes).

Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
module, but having a need to access public types from exported packages and get reflective access to
objects supertype's protected members?

This actually is an urgent need for my bridge. The BSF4ooRexx package comes with a wealth of samples
to show ooRexx programmers how easy it is to use Java as a huge external ooRexx class library. Some
of these samples in Java 9 now cause warnings, e.g. one example:

... cut ...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.rexxla.bsf.engines.rexx.RexxAndJava
(file:/C:/Program%20Files%20(x86)/BSF4ooRexx/bsf4ooRexx-v600-20180101-bin.jar)

to method sun.java2d.SunGraphics2D.setRenderingHint(java.awt.RenderingHints$Key,java.lang.Object)

WARNING: Please consider reporting this to the maintainers of
org.rexxla.bsf.engines.rexx.RexxAndJava
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access
operations
WARNING: All illegal access operations will be denied in a future release
... cut ...

So all of a sudden the reflection does not work anymore. The reason being that the graphic "g"
context in javax.swing.JComponent's paintComponent (java.awt.Graphics g) is actually a
"sun.java2d.SunGraphics2D" object, that according to this Java 9 warning is not exported and will be
an error in Java 10. (I have assumed, that if I would use a reflection Method object from
javax.swing.JComponent (which is protected and can only be accessed by ooRexx, if a dynamically
created subclass of JComponent is created from which the work is done) for setRenderingHint() that
invoking the Method object with the Sun object would succeed.)

There are other use-cases where such warnings occur.

No user of the ooRexx-Java bridge knows anything of the new module system in Java 9, nor should she
have a need to know if these sample programs have worked flawlessly in the past. Therefore I am
seriously interested to find a solution that adheres to the Java 9 module system.

BTW: On protected members then I assume they have never been accessible to the bridge, maybe the
bridge is using setAccessible to suppress the access checks for those cases?

Yes.
(Have been hoping that that need would be forgone eventually, when that reflection area gets
reworked, updated for this protected member case, if the object to be worked on stems from a
subclass to allow exactly the same thing the Java compiler has allowed for, this time for an
interpreted language.)

Post by Rony G. Flatscher
---
The current implementation seems to do the following: it checks whether a class is from an 
exported package, if not it creates an exception. If the class is exported it then checks whether
the reflected member is public, if not it creates an exception.
What I would have expected is this: everything like the current implementation, but in the second
step, if the member is not public, check whether the object's class for which the reflection
takes place is a subclass of the class that is being checked and if so, allow protected members
to be accessed as well. Hence: in the example presented, the packages 'mtest1' and 'mtest3' are
exported, the class 'mtest3.Class03A' extends 'mtest2.Class02A' which extends 'mtest1.Class01A'.
Therefore the object of type 'mtest3.Class03A' is a subclass of 'mtest1.Class01A', hence as
'mtest1' is exported all 'mtest1.Class01A' public *and* protected members should be accessible to
the object of the subtype 'mtest3.Class03A' via reflection in Java 9 as well.

You should find that the protected members are accessible in the sub-class (mtest3.Class03A in
this case). They won't be accessible to the bridge code of course (as least not without some kind
of rights delegation as John mentioned).

Not with Java 9 javac, the following Java program:

public class TestUse
{
    public static void main (String args[]) {
        mtest3.Class03A o=new mtest3.Class03A();
        System.out.println("o: "+o+", myClassName: "+o.myClassName);
    }
}

yields the error message:

F:\work\svn\bsf4oorexx\trunk\bsf4oorexx.dev\testUnits\bsf4rexx\java9modules>java  -cp
".;F:\work\svn\bsf4oorexx\trunk;.;C:\Program Files
(x86)\BSF4ooRexx\bsf4ooRexx-v600-20180101-bin.jar;C:\Program Files
(x86)\BSF4ooRexx\jni4net.j-0.8.8.0.jar;C:\Program Files (x86)\BSF4ooRexx\oorexx.net.jar;."
--module-path out --add-modules mod_A,mod_B,mod_C TestUse
Exception in thread "main" java.lang.IllegalAccessError: tried to access field
mtest1.Class01A.myClassName from class TestUse
        at TestUse.main(TestUse.java:5)

Note: the TestUse class will be part of the unnamed module.

---

Ad "rights delegation mechanism": would that really allow the bridge from the unnamed module to do
what is needed and therefore sought, ie. accessing protected members in exported superclasses? If
so, how? Would you have any links for further reading?

If you could tell me that I misunderstood the access rules of the java.lang.invoke package and that
it was possible to use it and/or the jdk.dynalink package to overcome, I would definitely try to
apply that to solve this issue, as this is really very important for the users of the bridge for
whom it is simply impossible to solve this.

---

Alternatively: what is the problem that you see for the Java 9 module system, if access checking is
done with the extension of also allowing protected members to be used, if the object on which the
reflective access is to be carried out is indeed from a subclass of the defining class of the
java.lang.reflect object being supplied?

---rony

[Alan Bateman]

Post by Rony G. Flatscher
public class TestUse
{
    public static void main (String args[]) {
        mtest3.Class03A o=new mtest3.Class03A();
        System.out.println("o: "+o+", myClassName: "+o.myClassName);
    }
}
F:\work\svn\bsf4oorexx\trunk\bsf4oorexx.dev\testUnits\bsf4rexx\java9modules>java
-cp ".;F:\work\svn\bsf4oorexx\trunk;.;C:\Program Files
(x86)\BSF4ooRexx\bsf4ooRexx-v600-20180101-bin.jar;C:\Program Files
(x86)\BSF4ooRexx\jni4net.j-0.8.8.0.jar;C:\Program Files
(x86)\BSF4ooRexx\oorexx.net.jar;." --module-path out --add-modules
mod_A,mod_B,mod_C TestUse
Exception in thread "main" java.lang.IllegalAccessError: tried to
access field mtest1.Class01A.myClassName from class TestUse
        at TestUse.main(TestUse.java:5)

If it compiles then it should run so something isn't right here. I just
tried this with JDK 9 and JDK 10 EA builds but couldn't duplicate it
(the compilation fails as expected because myClassName has protected
access in mod_A/mtest1.Class01A).

Would you mind zipping up the sources for mod_A, mod_B, and mod_C so we
can duplicate this?

-Alan

[Rony G. Flatscher]

Dear Alan:

tried to come up with a "cleaner" version to zip it up, however the error would not occur there.

---

As I have to get off for today, I just zipped up yesterday's version "as is" that exhibits that
compile error and attach it to this e-mail.

Just unzip it, go into "java9module" (this is on Windows):

* ./ contains the (Windows) batch files and TestUse.java
o to compile all sources:
+ 1_compile.cmd
o to compile TestUse.java:
+ 5_compile_and_run_TestUse.cmd
* src/ contains the sources
* out/ contains the compiled stuff

If you prefer and can wait until next week, I will try to find out the differences between the
"cleaned" version (removed all Interfaces for that version) and yesterday's version and report back.

---rony

Post by Rony G. Flatscher
public class TestUse
{
    public static void main (String args[]) {
        mtest3.Class03A o=new mtest3.Class03A();
        System.out.println("o: "+o+", myClassName: "+o.myClassName);
    }
}
F:\work\svn\bsf4oorexx\trunk\bsf4oorexx.dev\testUnits\bsf4rexx\java9modules>java  -cp
".;F:\work\svn\bsf4oorexx\trunk;.;C:\Program Files
(x86)\BSF4ooRexx\bsf4ooRexx-v600-20180101-bin.jar;C:\Program Files
(x86)\BSF4ooRexx\jni4net.j-0.8.8.0.jar;C:\Program Files (x86)\BSF4ooRexx\oorexx.net.jar;."
--module-path out --add-modules mod_A,mod_B,mod_C TestUse
Exception in thread "main" java.lang.IllegalAccessError: tried to access field
mtest1.Class01A.myClassName from class TestUse
        at TestUse.main(TestUse.java:5)

If it compiles then it should run so something isn't right here. I just tried this with JDK 9 and
JDK 10 EA builds but couldn't duplicate it (the compilation fails as expected because myClassName
has protected access in mod_A/mtest1.Class01A).
Would you mind zipping up the sources for mod_A, mod_B, and mod_C so we can duplicate this?
-Alan

[Alan Bateman]

An attachment in the email has been found to contain executable code
and has been removed.
File removed : java9modules.zip, zip,cmd
------------------------------------------------------------------------
tried to come up with a "cleaner" version to zip it up, however the
error would not occur there.

The attachment was dropped too.

When you say "the error would not occur there" then do you mean it won't
compile? I wouldn't expect the test to compile with a reference to the
protected member so maybe the issue was that the code was compiled in a
different way and so only found when you ran.

-Alan

[Rony G. Flatscher]

Post by Alan Bateman

An attachment in the email has been found to contain executable code and has been removed.
File removed : java9modules.zip, zip,cmd
----------------------------------------------------------------------------------------------------
tried to come up with a "cleaner" version to zip it up, however the error would not occur there.

The attachment was dropped too.

OK, next time, I will use Dropbox or the like instead.

Post by Alan Bateman
When you say "the error would not occur there" then do you mean it won't compile? I wouldn't
expect the test to compile with a reference to the protected member so maybe the issue was that
the code was compiled in a different way and so only found when you ran.

Looked into it and you are probably right, the class file was a few minutes older than the latest
version of the java file. Deleting the class file and trying to recompile yields the expected error
"...has protected access in Class01A..." and there is no public getter defined for that field in any
of the classes in the exported packages.

---rony

[Roger Riggs]

Hi,

Contributions need to be submitted using the OpenJDK infrastructure to
adhere to the IP requirements.
The mail lists shold pass attachments that are text and patches though
you may need to be
sure your mailer attaches them with the correct mime-types and/or
extensions.

Roger

p.s.
The current settings have pass_mime_types:

multipart/mixed
multipart/alternative
text/plain
text/x-diff
text/x-patch
message/rfc822
multipart/signed

Post by Rony G. Flatscher

Post by Alan Bateman

An attachment in the email has been found to contain executable code and has been removed.
File removed : java9modules.zip, zip,cmd
----------------------------------------------------------------------------------------------------
tried to come up with a "cleaner" version to zip it up, however the error would not occur there.

The attachment was dropped too.

OK, next time, I will use Dropbox or the like instead.

Sorry, only inline, attached or on cr.openjdk.java.net.

[Rony G. Flatscher]

Hi,

Contributions need to be submitted using the OpenJDK infrastructure to adhere to the IP requirements.

Would you have a link which OpenJFDK infrastructure to use in this case?

The mail lists shold pass attachments that are text and patches though you may need to be
sure your mailer attaches them with the correct mime-types and/or extensions.
Roger
p.s.
   multipart/mixed
   multipart/alternative
   text/plain
   text/x-diff
   text/x-patch
   message/rfc822
   multipart/signed

Thanks for this information!

Regards,

---rony

Post by Rony G. Flatscher

Post by Alan Bateman

An attachment in the email has been found to contain executable code and has been removed.
File removed : java9modules.zip, zip,cmd
----------------------------------------------------------------------------------------------------
tried to come up with a "cleaner" version to zip it up, however the error would not occur there.

The attachment was dropped too.

OK, next time, I will use Dropbox or the like instead.

Sorry, only inline, attached or on cr.openjdk.java.net.

[Roger Riggs]

Hi,

Attaching the patch as a .patch or .txt file should be sufficient for
small contributions and example code.

Roger

Post by Rony G. Flatscher
Hi,

Contributions need to be submitted using the OpenJDK infrastructure to adhere to the IP requirements.

Would you have a link which OpenJFDK infrastructure to use in this case?

The mail lists shold pass attachments that are text and patches though you may need to be
sure your mailer attaches them with the correct mime-types and/or extensions.
Roger
p.s.
   multipart/mixed
   multipart/alternative
   text/plain
   text/x-diff
   text/x-patch
   message/rfc822
   multipart/signed

Thanks for this information!
Regards,
---rony

Post by Rony G. Flatscher

Post by Alan Bateman

An attachment in the email has been found to contain executable code and has been removed.
File removed : java9modules.zip, zip,cmd
----------------------------------------------------------------------------------------------------
tried to come up with a "cleaner" version to zip it up, however the error would not occur there.

The attachment was dropped too.

OK, next time, I will use Dropbox or the like instead.

Sorry, only inline, attached or on cr.openjdk.java.net.

[Alan Bateman]

Post by Rony G. Flatscher
Would you have concrete suggestions for this use-case, i.e. a
framework that is not part of a module, but having a need to access
public types from exported packages and get reflective access to
objects supertype's protected members?

I think it would be better to start with public members as protected is
complicated (and hasn't changed with modules once you establish the
class declaring the member is accessible).

For your example, you've got a reference to a java.awt.Graphics2D
object, the actual implementation type is sun.java2d.SunGraphics2D. The
user is attempting to invoke one of the public setRenderingHint methods
that Graphics2D defines. You said in one of your mails that the bridge
"iterates over all its superclasses" which I take to mean that it
recursively looks at the superclass and interfaces to find a public
class or interface that defines the target setRenderingHint method. In
the example, I expect it would skip sun.java2d.SunGraphics2D if it were
non public.

Can you extend this check to test if the class is in a package exported
by its module. For the example, sun.java2d.SunGraphics2D is in the
java.desktop module and this module does not export sun.java2d to
everyone. Here is a code fragment to test this:

Class<?> clazz = graphicsObj.getClass();
boolean isExportedToAll =
clazz.getModule().isExported(clazz.getPackageName());

(I'm deliberately avoiding the 2-arg isExported to keep things simple
for this discussion).

If you can incorporate this check into the bridge then I suspect you'll
find most of the examples will work.

-Alan

[Rony G. Flatscher]

Post by Rony G. Flatscher
Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
module, but having a need to access public types from exported packages and get reflective access
to objects supertype's protected members?

I think it would be better to start with public members as protected is complicated (and hasn't
changed with modules once you establish the class declaring the member is accessible).
For your example, you've got a reference to a java.awt.Graphics2D object, the actual
implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
superclass and interfaces to find a public class or interface that defines the target
setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
were non public.
Can you extend this check to test if the class is in a package exported by its module. For the
example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
Class<?> clazz = graphicsObj.getClass();
boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
(I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
If you can incorporate this check into the bridge then I suspect you'll find most of the examples
will work.

Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
find an accessible member in a superclass, which means to have a need to also access protected
members in the superclass) and that is actually my current approach. However, I started out with
reflecting Fields first and see, whether I can reflectively get access.

The rewritten method resolution would follow next, which would allow me to tackle that warning and
see whether I can get rid of it. However, before going a wrong route I would like to learn what the
"official" Java 9 solution would be and try to implement that.

---rony

[Peter Levart]

Hi Rony,

Post by Rony G. Flatscher

Post by Rony G. Flatscher
Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
module, but having a need to access public types from exported packages and get reflective access
to objects supertype's protected members?

I think it would be better to start with public members as protected is complicated (and hasn't
changed with modules once you establish the class declaring the member is accessible).
For your example, you've got a reference to a java.awt.Graphics2D object, the actual
implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
superclass and interfaces to find a public class or interface that defines the target
setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
were non public.
Can you extend this check to test if the class is in a package exported by its module. For the
example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
Class<?> clazz = graphicsObj.getClass();
boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
(I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
If you can incorporate this check into the bridge then I suspect you'll find most of the examples
will work.

Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
find an accessible member in a superclass, which means to have a need to also access protected
members in the superclass) and that is actually my current approach. However, I started out with
reflecting Fields first and see, whether I can reflectively get access.
The rewritten method resolution would follow next, which would allow me to tackle that warning and
see whether I can get rid of it. However, before going a wrong route I would like to learn what the
"official" Java 9 solution would be and try to implement that.
---rony

Yes, I think you are dealing with two problems here which you have been
using the same solution for in the past.

The 1st thing you have been doing incorrectly for Java 9, as Alan
explained, is the idiom: o.getClass().getMethod(...) and the 2nd is that
you are trying to access protected members on behalf of some other class
which is a subclass of the protected member's declaring class.

The 1st problem has different solutions which are all doable in Java 9,
since you are dealing within the confines of public types, public
members and exported packages. One solution is to search for the most
specific member in the inheritance hierarchy which is also accessible
(declared in public type in exported package) which is what Alan suggests.

There might also be another elegant solution which requires some
re-design of your Rexx interpreter.  When you deal with reference values
in Rexx (the values that refer to objects in Java), you could track not
only the value itself but also the "static" type of that value. A
reference value is always obtained either by calling a constructor,
accessing a field (either static or instance), by calling a method
(static or instance) or by accessing an element of some array:

- calling constructor: the "static type" is the class upon which the
constructor has been called
- accessing a field: the "static type" is the type of the field (i.e.
Field.getDeclaringClass())
- calling a method: the "static type" is the return type of the method
(i.e. Method.getReturnType())
- accessing an element of some array: the "static type" is the array's
"static type"'s component type (i.e. Class.getComponentType() invoked on
array's "static type" Class).

When you take the "static" type as the starting Class when searching for
a public member with standard Class.getMethod() or Class.getField(), you
would then get the correct publicly accessible reflected member. With a
caveat that this only works when there's no generics involved. If
there's generics, the logic to compute the correct static type is more
involved and would sometimes require passing the generic type parameters
(when invoking constructors of generic classes or generic methods) in
the syntax of your Rexx language. So you may or may not want to do that.
Perhaps some library for deep resolving could be of help here (Google
Guava has some support for that). I guess searching for the most
specific member in the hierarchy that is also accessible is your best
bet currently if the goal is to be syntactically backwards compatible in
the Rexx language.

The 2nd problem is not trivial as you want to access a protected member
on behalf of some other sub-class of the member's declaring class which
is not cooperating (voluntarily handing you an instance of its Lookup
object). This currently requires the package containing the member's
declaring class to be opened at least to you (the Rexx interpreter) and
using the member.setAccessible(true) trick or
MethodHandles.privateLookupIn(declaringClass) equivalent for method
handles. Which is awkward because libraries packed as modules would
normally not specify that in their module descriptors and system modules
don't either. So you are left with either --add-opens command line
switches or deploying a javaagent to the JVM and using it's API point
java.lang.instrument.Instrumentation#redefineModule to add opens to
modules that way. Both approaches are not elegant, but that's what is
currently available, I think.

Regards, Peter

[Alan Bateman]

Post by Peter Levart
The 2nd problem is not trivial as you want to access a protected
member on behalf of some other sub-class of the member's declaring
class which is not cooperating (voluntarily handing you an instance of
its Lookup object). This currently requires the package containing the
member's declaring class to be opened at least to you (the Rexx
interpreter) and using the member.setAccessible(true) trick or
MethodHandles.privateLookupIn(declaringClass) equivalent for method
handles. Which is awkward because libraries packed as modules would
normally not specify that in their module descriptors and system
modules don't either. So you are left with either --add-opens command
line switches or deploying a javaagent to the JVM and using it's API
point java.lang.instrument.Instrumentation#redefineModule to add opens
to modules that way. Both approaches are not elegant, but that's what
is currently available, I think.

I suspect it may be just a misunderstanding. One of Rony's mails had
this example:

o=.bsf~new("mtest3.Class03A")             -- create Java object, get and
assign proxy ooRexx object
say "o:" o "o~myClassName:" o~myClassName -- get (static) field value in
"mtest1.Class01A", accessible via inheritance

I read this as the Rexx script doing the equivalent of "new
mtest3.Class03A()", in which case should be no expectation that
protected members are accessible to the Rexx code.

-Alan.

[Peter Levart]

Post by Alan Bateman

Post by Peter Levart
The 2nd problem is not trivial as you want to access a protected
member on behalf of some other sub-class of the member's declaring
class which is not cooperating (voluntarily handing you an instance
of its Lookup object). This currently requires the package containing
the member's declaring class to be opened at least to you (the Rexx
interpreter) and using the member.setAccessible(true) trick or
MethodHandles.privateLookupIn(declaringClass) equivalent for method
handles. Which is awkward because libraries packed as modules would
normally not specify that in their module descriptors and system
modules don't either. So you are left with either --add-opens command
line switches or deploying a javaagent to the JVM and using it's API
point java.lang.instrument.Instrumentation#redefineModule to add
opens to modules that way. Both approaches are not elegant, but
that's what is currently available, I think.

I suspect it may be just a misunderstanding. One of Rony's mails had
o=.bsf~new("mtest3.Class03A")             -- create Java object, get
and assign proxy ooRexx object
say "o:" o "o~myClassName:" o~myClassName -- get (static) field value
in "mtest1.Class01A", accessible via inheritance
I read this as the Rexx script doing the equivalent of "new
mtest3.Class03A()", in which case should be no expectation that
protected members are accessible to the Rexx code.
-Alan.

I was asking myself the same question, yes. If Rony wants to call
protected methods on "behalf" of subclasses, then Rexx runtime has to
have features to subclass existing Java classes.

@Rony: So why does Rexx script want to call protected members? Is that
because Rexx can extend existing Java classes?

Regards, Peter

[Rony G. Flatscher]

Post by Rony G. Flatscher

Post by Peter Levart
The 2nd problem is not trivial as you want to access a protected member on behalf of some other
sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
instance of its Lookup object). This currently requires the package containing the member's
declaring class to be opened at least to you (the Rexx interpreter) and using the
member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
method handles. Which is awkward because libraries packed as modules would normally not specify
that in their module descriptors and system modules don't either. So you are left with either
--add-opens command line switches or deploying a javaagent to the JVM and using it's API point
java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
approaches are not elegant, but that's what is currently available, I think.

o=.bsf~new("mtest3.Class03A")             -- create Java object, get and assign proxy ooRexx object
say "o:" o "o~myClassName:" o~myClassName -- get (static) field value in "mtest1.Class01A",
accessible via inheritance
I read this as the Rexx script doing the equivalent of "new mtest3.Class03A()", in which case
should be no expectation that protected members are accessible to the Rexx code.

Yes, you are right! There need to be a public member that is capable of accessing the protected ones
in this case.

Obviously I have used too many variants and mixed up the use cases, really sorry for the noise! :(

---rony

[Peter Levart]

Hi Rony,

Post by Peter Levart
The 2nd problem is not trivial as you want to access a protected
member on behalf of some other sub-class of the member's declaring
class which is not cooperating (voluntarily handing you an instance of
its Lookup object). This currently requires the package containing the
member's declaring class to be opened at least to you (the Rexx
interpreter) and using the member.setAccessible(true) trick or
MethodHandles.privateLookupIn(declaringClass) equivalent for method
handles. Which is awkward because libraries packed as modules would
normally not specify that in their module descriptors and system
modules don't either. So you are left with either --add-opens command
line switches or deploying a javaagent to the JVM and using it's API
point java.lang.instrument.Instrumentation#redefineModule to add opens
to modules that way. Both approaches are not elegant, but that's what
is currently available, I think.

Just one more thing... While solutions for tackling the 2nd problem
might seem attractive to use for solving the 1st problem too, I would
recommend not doing that. Opening all the packages of public API(s)
might inhibit possible optimizations John Rose has been talking about.
For reflective access to public API(s) you don't need to open the
packages because public API(s) are in exported packages and all the
"static" types that are needed to access them (field types, method
return an parameter types) are also guaranteed to be part of public
API(s) (at least good modules guarantee that). Public API(s) are
transitively public. For public API(s) it is just a matter of finding
the accessible member in the hierarchy where there will always be at
least one.

For the 2nd problem, the main difficulty seems to be how to open just
the packages that are involved in accessing the protected members on
behalf of subclasses hoping that those packages are in minority. Here's
one trick by using javaagent. Suppose your Rexx runtime had the
following nonpublic class in its heart:

package runtime;

import java.util.function.BiConsumer;

class Opener {
    private static class Holder {
        static BiConsumer<Class<?>, Module> opener;
    }

    static void openPackageOfTo(Class<?> clazz, Module module) {
        Holder.opener.accept(clazz, module);
    }
}


Now if you start the JVM by supplying the -javaagent:agent.jar command
line in addition to everything else and pack the following compiled code
into agent.jar with the following MANIFEST:

Manifest-Version: 1.0
Premain-Class: agent.Agent

---
package agent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.lang.reflect.Field;
import java.security.ProtectionDomain;
import java.util.Map;
import java.util.Set;
import java.util.function.BiConsumer;

public class Agent {
    private static final String OPENER_BINARY_CLASS_NAME =
"runtime/Opener";
    private static final String HOLDER_CLASS_NAME =
"runtime.Opener$Holder";
    private static final String OPENER_FIELD_NAME = "opener";

    private static Instrumentation instrumentation;

    public static void premain(String agentArgs, Instrumentation inst) {
        instrumentation = inst;
        inst.addTransformer(new ClassFileTransformer() {
            @Override
            public byte[] transform(Module module,
                                    ClassLoader loader,
                                    String className,
                                    Class<?> classBeingRedefined,
                                    ProtectionDomain protectionDomain,
                                    byte[] classfileBuffer) throws
IllegalClassFormatException {

                // when runtime.Opener starts loading...
                if (className.equals(OPENER_BINARY_CLASS_NAME) &&
classBeingRedefined == null) {
                    try {
                        // ...load runtime.Opener$Holder upfront using
the same classloader
                        Class<?> holderClass =
Class.forName(HOLDER_CLASS_NAME, true, loader);
                        // find the runtime.Opener$Holder#opener field
                        Field openerField =
holderClass.getDeclaredField(OPENER_FIELD_NAME);
                        // and make it accessible
                        openerField.setAccessible(true);
                        // inject the BiConsumer
                        openerField.set(null, (BiConsumer<Class<?>,
Module>) Agent::openPackageOfTo);
                    } catch (ReflectiveOperationException e) {
                        throw new InternalError(e);
                    }
                }

                // perform no actual transformation
                return null;
            }
        }, false);
    }

    static void openPackageOfTo(Class<?> clazz, Module module) {
        String pn = clazz.getPackageName();
        System.out.println("Opening package " + pn + " to " + module);
        instrumentation.redefineModule(
            clazz.getModule(),
            Set.of(),
            Map.of(),
            Map.of(pn, Set.of(module)), // extra opens
            Set.of(),
            Map.of()
        );
    }
}


With such Rexx runtime specific helper agent jar you can extend the
controlled power of java agent to your Rexx interpreter so you now have
the power to dynamically add just those opens that are absolutely
necessary for performing the accesses to protected members.


Regards, Peter

[Peter Levart]

@Potential implementer of below trick BEWARE!

While I tried to be smart by "injecting" special java agent powers into
designated trusted class, the presented mechanism is NOT SAFE as I
identify the class only by it's name. An attacker might create it's own
pair of classes with same names (runtime.Opener, runtime.Opener$Holder)
loaded in some isolated class loader and the presented agent will
happily inject the powers into the attacker's runtime.Opener$Holder. For
safe variant of such agent, the Opener$Holder class would have to prove
the authenticity to the Agent 1st. The mechanism of such authentication
will be left as an exercise to the reader...

Regards, Peter

Post by Peter Levart
Hi Rony,

Post by Peter Levart
The 2nd problem is not trivial as you want to access a protected
member on behalf of some other sub-class of the member's declaring
class which is not cooperating (voluntarily handing you an instance
of its Lookup object). This currently requires the package containing
the member's declaring class to be opened at least to you (the Rexx
interpreter) and using the member.setAccessible(true) trick or
MethodHandles.privateLookupIn(declaringClass) equivalent for method
handles. Which is awkward because libraries packed as modules would
normally not specify that in their module descriptors and system
modules don't either. So you are left with either --add-opens command
line switches or deploying a javaagent to the JVM and using it's API
point java.lang.instrument.Instrumentation#redefineModule to add
opens to modules that way. Both approaches are not elegant, but
that's what is currently available, I think.

Just one more thing... While solutions for tackling the 2nd problem
might seem attractive to use for solving the 1st problem too, I would
recommend not doing that. Opening all the packages of public API(s)
might inhibit possible optimizations John Rose has been talking about.
For reflective access to public API(s) you don't need to open the
packages because public API(s) are in exported packages and all the
"static" types that are needed to access them (field types, method
return an parameter types) are also guaranteed to be part of public
API(s) (at least good modules guarantee that). Public API(s) are
transitively public. For public API(s) it is just a matter of finding
the accessible member in the hierarchy where there will always be at
least one.
For the 2nd problem, the main difficulty seems to be how to open just
the packages that are involved in accessing the protected members on
behalf of subclasses hoping that those packages are in minority.
Here's one trick by using javaagent. Suppose your Rexx runtime had the
package runtime;
import java.util.function.BiConsumer;
class Opener {
    private static class Holder {
        static BiConsumer<Class<?>, Module> opener;
    }
    static void openPackageOfTo(Class<?> clazz, Module module) {
        Holder.opener.accept(clazz, module);
    }
}
Now if you start the JVM by supplying the -javaagent:agent.jar command
line in addition to everything else and pack the following compiled
Manifest-Version: 1.0
Premain-Class: agent.Agent
---
package agent;
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.lang.reflect.Field;
import java.security.ProtectionDomain;
import java.util.Map;
import java.util.Set;
import java.util.function.BiConsumer;
public class Agent {
    private static final String OPENER_BINARY_CLASS_NAME =
"runtime/Opener";
    private static final String HOLDER_CLASS_NAME =
"runtime.Opener$Holder";
    private static final String OPENER_FIELD_NAME = "opener";
    private static Instrumentation instrumentation;
    public static void premain(String agentArgs, Instrumentation inst) {
        instrumentation = inst;
        inst.addTransformer(new ClassFileTransformer() {
            public byte[] transform(Module module,
                                    ClassLoader loader,
                                    String className,
                                    Class<?> classBeingRedefined,
                                    ProtectionDomain protectionDomain,
                                    byte[] classfileBuffer) throws
IllegalClassFormatException {
                // when runtime.Opener starts loading...
                if (className.equals(OPENER_BINARY_CLASS_NAME) &&
classBeingRedefined == null) {
                    try {
                        // ...load runtime.Opener$Holder upfront using
the same classloader
                        Class<?> holderClass =
Class.forName(HOLDER_CLASS_NAME, true, loader);
                        // find the runtime.Opener$Holder#opener field
                        Field openerField =
holderClass.getDeclaredField(OPENER_FIELD_NAME);
                        // and make it accessible
                        openerField.setAccessible(true);
                        // inject the BiConsumer
                        openerField.set(null, (BiConsumer<Class<?>,
Module>) Agent::openPackageOfTo);
                    } catch (ReflectiveOperationException e) {
                        throw new InternalError(e);
                    }
                }
                // perform no actual transformation
                return null;
            }
        }, false);
    }
    static void openPackageOfTo(Class<?> clazz, Module module) {
        String pn = clazz.getPackageName();
        System.out.println("Opening package " + pn + " to " + module);
        instrumentation.redefineModule(
            clazz.getModule(),
            Set.of(),
            Map.of(),
            Map.of(pn, Set.of(module)), // extra opens
            Set.of(),
            Map.of()
        );
    }
}
With such Rexx runtime specific helper agent jar you can extend the
controlled power of java agent to your Rexx interpreter so you now
have the power to dynamically add just those opens that are absolutely
necessary for performing the accesses to protected members.
Regards, Peter

[Rony G. Flatscher]

Hi Peter,

thank you *very* much also for your kind explanations and even coming up with agent code to
demonstrate how one could use that approach!

In fact I have been doing a static analysis in the past (since Java 1.1) to determine whether
members should be accessible to Rexx, restricting access to public members and to protected
inherited members, if accessing them from a dynamically created subclass.

One thing I have to make sure is, that I keep compatible with the Java 1.6/6.0 baseline, as there
are some deployments where the shops are still employing that environment. This should not really be
a problem as I am rewriting the entire reflection part and will be able to employ different
approaches for pre-Java 9 deplyoments, where the reflection class employed is dependent on the Java
runtime version.

Currently I am exploring the implications of the new Java 9 module system, trying to adhere to its
rules. In essence the goal is to allow reflectively everything for Rexx peers that a compiled Java
program allows for in the Java 9 environment. For that the reflecting class (currently in the
unnamed module) goes up the inheritance tree until it finds an exported class and analyses it
reflectively.

Experimenting with variations of classes residing in different modules with different exports, it is
possible to mix-up the relationships, what gets exported to what, and when should protected members
in superclasses be accessible and when not (and yes, this part should belong to dynamically created
subclasses, which also need adjustments to the module system).

Best regards,

---rony

Post by Peter Levart
Hi Rony,

Post by Rony G. Flatscher

Post by Rony G. Flatscher
Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
module, but having a need to access public types from exported packages and get reflective access
to objects supertype's protected members?

I think it would be better to start with public members as protected is complicated (and hasn't
changed with modules once you establish the class declaring the member is accessible).
For your example, you've got a reference to a java.awt.Graphics2D object, the actual
implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
superclass and interfaces to find a public class or interface that defines the target
setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
were non public.
Can you extend this check to test if the class is in a package exported by its module. For the
example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
Class<?> clazz = graphicsObj.getClass();
boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
(I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
If you can incorporate this check into the bridge then I suspect you'll find most of the examples
will work.

Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
find an accessible member in a superclass, which means to have a need to also access protected
members in the superclass) and that is actually my current approach. However, I started out with
reflecting Fields first and see, whether I can reflectively get access.
The rewritten method resolution would follow next, which would allow me to tackle that warning and
see whether I can get rid of it. However, before going a wrong route I would like to learn what the
"official" Java 9 solution would be and try to implement that.
---rony

Yes, I think you are dealing with two problems here which you have been using the same solution
for in the past.
o.getClass().getMethod(...) and the 2nd is that you are trying to access protected members on
behalf of some other class which is a subclass of the protected member's declaring class.
The 1st problem has different solutions which are all doable in Java 9, since you are dealing
within the confines of public types, public members and exported packages. One solution is to
search for the most specific member in the inheritance hierarchy which is also accessible
(declared in public type in exported package) which is what Alan suggests.
There might also be another elegant solution which requires some re-design of your Rexx
interpreter.  When you deal with reference values in Rexx (the values that refer to objects in
Java), you could track not only the value itself but also the "static" type of that value. A
reference value is always obtained either by calling a constructor, accessing a field (either
static or instance), by calling a method (static or instance) or by accessing an element of some
- calling constructor: the "static type" is the class upon which the constructor has been called
- accessing a field: the "static type" is the type of the field (i.e. Field.getDeclaringClass())
- calling a method: the "static type" is the return type of the method (i.e. Method.getReturnType())
- accessing an element of some array: the "static type" is the array's "static type"'s component
type (i.e. Class.getComponentType() invoked on array's "static type" Class).
When you take the "static" type as the starting Class when searching for a public member with
standard Class.getMethod() or Class.getField(), you would then get the correct publicly accessible
reflected member. With a caveat that this only works when there's no generics involved. If there's
generics, the logic to compute the correct static type is more involved and would sometimes
require passing the generic type parameters (when invoking constructors of generic classes or
generic methods) in the syntax of your Rexx language. So you may or may not want to do that.
Perhaps some library for deep resolving could be of help here (Google Guava has some support for
that). I guess searching for the most specific member in the hierarchy that is also accessible is
your best bet currently if the goal is to be syntactically backwards compatible in the Rexx language.
The 2nd problem is not trivial as you want to access a protected member on behalf of some other
sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
instance of its Lookup object). This currently requires the package containing the member's
declaring class to be opened at least to you (the Rexx interpreter) and using the
member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
method handles. Which is awkward because libraries packed as modules would normally not specify
that in their module descriptors and system modules don't either. So you are left with either
--add-opens command line switches or deploying a javaagent to the JVM and using it's API point
java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
approaches are not elegant, but that's what is currently available, I think.
Regards, Peter

[Peter Levart]

Hi Rony,

Post by Rony G. Flatscher
Hi Peter,
thank you *very* much also for your kind explanations and even coming up with agent code to
demonstrate how one could use that approach!
In fact I have been doing a static analysis in the past (since Java 1.1) to determine whether
members should be accessible to Rexx, restricting access to public members and to protected
inherited members, if accessing them from a dynamically created subclass.

So you *are* loading dynamically generated subclasses. It's just that
their logic is implemented in Rexx scripting and therefore delegates
invocations to Rexx runtime and those invocations include invocations to
protected methods of their superclasses.

In that case, why don't you come up with some mechanism for dynamically
generated classes to hand over their Lookup(s) to Rexx runtime. For
example, the Rexx runtime could have the following public registration API:

---
package runtime;

import java.lang.invoke.MethodHandles;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class LookupRegistry {
    private static final Map<Class<?>, MethodHandles.Lookup> registry =
        new ConcurrentHashMap<>();

    public static void register(MethodHandles.Lookup lookup) {
        if (registry.putIfAbsent(lookup.lookupClass(), lookup) != null) {
            throw new IllegalStateException(
                "Lookup for " + lookup.lookupClass() + " is already
registered");
        }
    }

    /* non-public */ static MethodHandles.Lookup getLookup(Class<?>
clazz) {
        return registry.get(clazz);
    }
}


...then each Rexx dynamically generated class could include the
following static initialization block:


public class GeneratedClass {
    static {
        runtime.LookupRegistry.register(
            java.lang.invoke.MethodHandles.lookup()
        );
    }
...


That way Rexx runtime could have access to Lookup(s) of each of it's
dynamically generated classes as soon as they are loaded and initialized
and can use a particular class's Lookup to lookup a MethodHandle for the
protected methods of its superclass. It can even simulate a
"super.method()" invocation, etc...


Regards, Peter

Post by Rony G. Flatscher
One thing I have to make sure is, that I keep compatible with the Java 1.6/6.0 baseline, as there
are some deployments where the shops are still employing that environment. This should not really be
a problem as I am rewriting the entire reflection part and will be able to employ different
approaches for pre-Java 9 deplyoments, where the reflection class employed is dependent on the Java
runtime version.
Currently I am exploring the implications of the new Java 9 module system, trying to adhere to its
rules. In essence the goal is to allow reflectively everything for Rexx peers that a compiled Java
program allows for in the Java 9 environment. For that the reflecting class (currently in the
unnamed module) goes up the inheritance tree until it finds an exported class and analyses it
reflectively.
Experimenting with variations of classes residing in different modules with different exports, it is
possible to mix-up the relationships, what gets exported to what, and when should protected members
in superclasses be accessible and when not (and yes, this part should belong to dynamically created
subclasses, which also need adjustments to the module system).
Best regards,
---rony

Post by Peter Levart
Hi Rony,

Post by Rony G. Flatscher

Post by Rony G. Flatscher
Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
module, but having a need to access public types from exported packages and get reflective access
to objects supertype's protected members?

I think it would be better to start with public members as protected is complicated (and hasn't
changed with modules once you establish the class declaring the member is accessible).
For your example, you've got a reference to a java.awt.Graphics2D object, the actual
implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
superclass and interfaces to find a public class or interface that defines the target
setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
were non public.
Can you extend this check to test if the class is in a package exported by its module. For the
example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
Class<?> clazz = graphicsObj.getClass();
boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
(I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
If you can incorporate this check into the bridge then I suspect you'll find most of the examples
will work.

Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
find an accessible member in a superclass, which means to have a need to also access protected
members in the superclass) and that is actually my current approach. However, I started out with
reflecting Fields first and see, whether I can reflectively get access.
The rewritten method resolution would follow next, which would allow me to tackle that warning and
see whether I can get rid of it. However, before going a wrong route I would like to learn what the
"official" Java 9 solution would be and try to implement that.
---rony

Yes, I think you are dealing with two problems here which you have been using the same solution
for in the past.
o.getClass().getMethod(...) and the 2nd is that you are trying to access protected members on
behalf of some other class which is a subclass of the protected member's declaring class.
The 1st problem has different solutions which are all doable in Java 9, since you are dealing
within the confines of public types, public members and exported packages. One solution is to
search for the most specific member in the inheritance hierarchy which is also accessible
(declared in public type in exported package) which is what Alan suggests.
There might also be another elegant solution which requires some re-design of your Rexx
interpreter.  When you deal with reference values in Rexx (the values that refer to objects in
Java), you could track not only the value itself but also the "static" type of that value. A
reference value is always obtained either by calling a constructor, accessing a field (either
static or instance), by calling a method (static or instance) or by accessing an element of some
- calling constructor: the "static type" is the class upon which the constructor has been called
- accessing a field: the "static type" is the type of the field (i.e. Field.getDeclaringClass())
- calling a method: the "static type" is the return type of the method (i.e. Method.getReturnType())
- accessing an element of some array: the "static type" is the array's "static type"'s component
type (i.e. Class.getComponentType() invoked on array's "static type" Class).
When you take the "static" type as the starting Class when searching for a public member with
standard Class.getMethod() or Class.getField(), you would then get the correct publicly accessible
reflected member. With a caveat that this only works when there's no generics involved. If there's
generics, the logic to compute the correct static type is more involved and would sometimes
require passing the generic type parameters (when invoking constructors of generic classes or
generic methods) in the syntax of your Rexx language. So you may or may not want to do that.
Perhaps some library for deep resolving could be of help here (Google Guava has some support for
that). I guess searching for the most specific member in the hierarchy that is also accessible is
your best bet currently if the goal is to be syntactically backwards compatible in the Rexx language.
The 2nd problem is not trivial as you want to access a protected member on behalf of some other
sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
instance of its Lookup object). This currently requires the package containing the member's
declaring class to be opened at least to you (the Rexx interpreter) and using the
member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
method handles. Which is awkward because libraries packed as modules would normally not specify
that in their module descriptors and system modules don't either. So you are left with either
--add-opens command line switches or deploying a javaagent to the JVM and using it's API point
java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
approaches are not elegant, but that's what is currently available, I think.
Regards, Peter

[Rony G. Flatscher]

Hi Peter,

firstly: thank you *very* much again for your kind hints and help!

Ad Rexx and Java: the aim is to have Rexx (dynamically typed, caseless) behave 1:1 like a compilable
and runnable Java program. Over the course of time I looked into ASM (many, many years ago) to add
the ability to create dynamically subclasses. In the meantime Janino is at play as I really needed
just the ability to create dynamically simple subclasses (again quite some time ago). This support
was needed to allow abstract Java methods (and Java interface classes) to be implemented in ooRexx.
It was also implemented such, that concrete Java classes can be subclassed at runtime as well
(supplying a list of Java methods that should be interceptable by Rexx methods, but still allow Rexx
to invoke the Java methods in superclasses).

The reflection infrastructure in the bridge has been developed over the course of more than 15
years. It worked until Java 1.8/8 (introduction of new static and default methods) unchanged for
many years. Now with Java 9 clearly the reflection part needs to be redone to work with Java 9 and I
take the opportunity to rewrite it altogether in a way (hoping) that the module related problems can
be solved via reflection only, such that I can also create stripped down versions of the rewritten
reflection part for Java 1.8/8 and another for Java 1.6/6 and 1.7/7. Hence my desire to stick to
reflection for the time being.

If that is not possible with some of the needed functionality then I would immediately turn to
MethodHandles and try to follow your advice.

The task of the bridge is in principle is to allow Java programs to be written in Rexx and when the
Rexx program executes it should execute as if it was a Java program (if that can be compiled and
run)! :) [The bridge allows Java to employ Rexx as a scripting language as well, implementing
Apache's BSF and Java's 1.6/6 javax.script.ScriptEngine and the like.]

The Java objects to work with reflectively are received by invoking Java methods so are out of
control of the bridge. All interactions with Java have been done using the Java reflection mechanism
(including with the dynamically created Java classes at runtime) since Java 1.1.

Again, thank you very much for your kind help and advice!

---rony

P.S.: Currently I have been tracing down accessing public fields in a superclass of a type which
package is exported from the unnamed module (Rexx reflection at this stage works out of the unnamed
module), but the superclass resides in a non-exported package. A Java program from the unnamed
module is able to access those public fields, doing the same with reflection has not been successful
yet. Will double-check the example and post it under a new thread (javac 9 creates quite different
code for that Java program depending whether using the module or the classpath where the module
version uses MethodHandles; however both versions are able to access those public fields, which I
cannot access via reflection yet).

Post by Peter Levart
Hi Rony,

Post by Rony G. Flatscher
Hi Peter,
thank you *very* much also for your kind explanations and even coming up with agent code to
demonstrate how one could use that approach!
In fact I have been doing a static analysis in the past (since Java 1.1) to determine whether
members should be accessible to Rexx, restricting access to public members and to protected
inherited members, if accessing them from a dynamically created subclass.

So you *are* loading dynamically generated subclasses. It's just that their logic is implemented
in Rexx scripting and therefore delegates invocations to Rexx runtime and those invocations
include invocations to protected methods of their superclasses.
In that case, why don't you come up with some mechanism for dynamically generated classes to hand
over their Lookup(s) to Rexx runtime. For example, the Rexx runtime could have the following
---
package runtime;
import java.lang.invoke.MethodHandles;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class LookupRegistry {
    private static final Map<Class<?>, MethodHandles.Lookup> registry =
        new ConcurrentHashMap<>();
    public static void register(MethodHandles.Lookup lookup) {
        if (registry.putIfAbsent(lookup.lookupClass(), lookup) != null) {
            throw new IllegalStateException(
                "Lookup for " + lookup.lookupClass() + " is already registered");
        }
    }
    /* non-public */ static MethodHandles.Lookup getLookup(Class<?> clazz) {
        return registry.get(clazz);
    }
}
public class GeneratedClass {
    static {
        runtime.LookupRegistry.register(
            java.lang.invoke.MethodHandles.lookup()
        );
    }
...
That way Rexx runtime could have access to Lookup(s) of each of it's dynamically generated classes
as soon as they are loaded and initialized and can use a particular class's Lookup to lookup a
MethodHandle for the protected methods of its superclass. It can even simulate a "super.method()"
invocation, etc...
Regards, Peter

Post by Rony G. Flatscher
One thing I have to make sure is, that I keep compatible with the Java 1.6/6.0 baseline, as there
are some deployments where the shops are still employing that environment. This should not really be
a problem as I am rewriting the entire reflection part and will be able to employ different
approaches for pre-Java 9 deplyoments, where the reflection class employed is dependent on the Java
runtime version.
Currently I am exploring the implications of the new Java 9 module system, trying to adhere to its
rules. In essence the goal is to allow reflectively everything for Rexx peers that a compiled Java
program allows for in the Java 9 environment. For that the reflecting class (currently in the
unnamed module) goes up the inheritance tree until it finds an exported class and analyses it
reflectively.
Experimenting with variations of classes residing in different modules with different exports, it is
possible to mix-up the relationships, what gets exported to what, and when should protected members
in superclasses be accessible and when not (and yes, this part should belong to dynamically created
subclasses, which also need adjustments to the module system).
Best regards,
---rony

Post by Peter Levart
Hi Rony,

Post by Rony G. Flatscher

Post by Rony G. Flatscher
Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
module, but having a need to access public types from exported packages and get reflective access
to objects supertype's protected members?

I think it would be better to start with public members as protected is complicated (and hasn't
changed with modules once you establish the class declaring the member is accessible).
For your example, you've got a reference to a java.awt.Graphics2D object, the actual
implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
superclass and interfaces to find a public class or interface that defines the target
setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
were non public.
Can you extend this check to test if the class is in a package exported by its module. For the
example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
Class<?> clazz = graphicsObj.getClass();
boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
(I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
If you can incorporate this check into the bridge then I suspect you'll find most of the examples
will work.

Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
find an accessible member in a superclass, which means to have a need to also access protected
members in the superclass) and that is actually my current approach. However, I started out with
reflecting Fields first and see, whether I can reflectively get access.
The rewritten method resolution would follow next, which would allow me to tackle that warning and
see whether I can get rid of it. However, before going a wrong route I would like to learn what the
"official" Java 9 solution would be and try to implement that.
---rony

Yes, I think you are dealing with two problems here which you have been using the same solution
for in the past.
o.getClass().getMethod(...) and the 2nd is that you are trying to access protected members on
behalf of some other class which is a subclass of the protected member's declaring class.
The 1st problem has different solutions which are all doable in Java 9, since you are dealing
within the confines of public types, public members and exported packages. One solution is to
search for the most specific member in the inheritance hierarchy which is also accessible
(declared in public type in exported package) which is what Alan suggests.
There might also be another elegant solution which requires some re-design of your Rexx
interpreter.  When you deal with reference values in Rexx (the values that refer to objects in
Java), you could track not only the value itself but also the "static" type of that value. A
reference value is always obtained either by calling a constructor, accessing a field (either
static or instance), by calling a method (static or instance) or by accessing an element of some
- calling constructor: the "static type" is the class upon which the constructor has been called
- accessing a field: the "static type" is the type of the field (i.e. Field.getDeclaringClass())
- calling a method: the "static type" is the return type of the method (i.e. Method.getReturnType())
- accessing an element of some array: the "static type" is the array's "static type"'s component
type (i.e. Class.getComponentType() invoked on array's "static type" Class).
When you take the "static" type as the starting Class when searching for a public member with
standard Class.getMethod() or Class.getField(), you would then get the correct publicly accessible
reflected member. With a caveat that this only works when there's no generics involved. If there's
generics, the logic to compute the correct static type is more involved and would sometimes
require passing the generic type parameters (when invoking constructors of generic classes or
generic methods) in the syntax of your Rexx language. So you may or may not want to do that.
Perhaps some library for deep resolving could be of help here (Google Guava has some support for
that). I guess searching for the most specific member in the hierarchy that is also accessible is
your best bet currently if the goal is to be syntactically backwards compatible in the Rexx language.
The 2nd problem is not trivial as you want to access a protected member on behalf of some other
sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
instance of its Lookup object). This currently requires the package containing the member's
declaring class to be opened at least to you (the Rexx interpreter) and using the
member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
method handles. Which is awkward because libraries packed as modules would normally not specify
that in their module descriptors and system modules don't either. So you are left with either
--add-opens command line switches or deploying a javaagent to the JVM and using it's API point
java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
approaches are not elegant, but that's what is currently available, I think.
Regards, Peter