Strange observation: MethodHandle.invokeWithArguments() would not work, whereas Method.invoke() would with the very same arguments

[Rony G. Flatscher]

In the process of adapting pure reflective code (a Rexx-Java bridge) to use MethodHandles on Java 9
instead, everything seems to be working out so far.

In principle all invocations on the Java side are carried out by first using java.lang.reflect
(Field, Method, Constructor) using the supplied arguments (if the arguments can be coerced to the
respective parameterTypes it gets selected for invocation)  and if a candidate is found an
appropriate MethodHandle gets created, which then gets used to invoke the operation supplying the
coerced arguments, if any. Over the weekend I finalized the basic changes and started to test
against a set of sample/demo applications.

---

While testing a rather complex one (an adaption of the JavaFX address book example enhanced with a
BarChart, [1]), that exhibits a very strange behavior: when setting the values for the CategoryAxis
supplying an ObservableList of the month names in the current Locale, using a MethodHandle and
invoking it with invokeWithArguments() would yield (debug output):

// // // RexxReflectJava9.processMethod(), ARRIVED: -> [INVOKE], tmpMethod=[public final void
javafx.scene.chart.CategoryAxis.setCategories(javafx.collections.ObservableList)]:
method=[SETCATEGORIES] in
object=[rru.rexxArgs[1]="***@83278e1"/rru.bean="CategoryAxis[id=xAxis,
styleClass=axis]"]

// // // RexxReflectJava9.processMethod(),
coercedArgs=[[[Ljava.lang.String;@57cfe770]].getClass().toString()=class [Ljava.lang.Object;,
         parameterTypes=[interface
javafx.collections.ObservableList].getClass().toString()=class [Ljava.lang.Class;:,
         rru.funcArgs=[[[Ljava.lang.String;@57cfe770]].getClass().toString()=class
[Ljava.lang.Object;

// // :( RexxReflectJava9.processMethod(), MethodType for Method [public final void
javafx.scene.chart.CategoryAxis.setCategories(javafx.collections.ObservableList)]:
"(ObservableList)void"

// // :( RexxReflectJava9.processMethod(): INSTANCE, mh.bindTo("CategoryAxis[id=xAxis,
styleClass=axis]/class javafx.scene.chart.CategoryAxis").invokeWithArguments(...)

// :) :) RexxReflectJava9.processMethod(), MethodHandle
"MethodHandle(CategoryAxis,ObservableList)void" invocation caused a Throwable:
java.lang.ClassCastException: java.base/[Ljava.lang.String; cannot be cast to
java.base/java.lang.String


The supplied ObservableList argument represents the  month names and was created with the help of
"javafx.collections.FXCollections.observableList()" and then using its "addAll(monthNames)" method
to add the String array values returned by DateFormatSymbols.getMonths() to the list.

The supplied argument array "rru.funcArgs" will be coerced according to the reflected
"parameterTypes" array yielding the "coercedArgs" array; using java.util.Arrays.deepToString() gives:

// // // RexxReflectJava9.processMethod(),
coercedArgs=[[[Ljava.lang.String;@57cfe770]].getClass().toString()=class [Ljava.lang.Object;,
         parameterTypes=[interface
javafx.collections.ObservableList].getClass().toString()=class [Ljava.lang.Class;:,
         rru.funcArgs=[[[Ljava.lang.String;@57cfe770]].getClass().toString()=class
[Ljava.lang.Object;

---

The story is much longer but after quite long debugging sessions, I turned on reflective invoke via
tmpMethod instead of invoking the corresponding MethodHandle, which (surprisingly) works.

Then, in the next step doing the same invocation via the corresponding MethodHandle immediately
after the reflective invocation, allows that invocation to execute successfully as well!

Please note, the supplied coerced argument is in both cases the same! Coercion occurs according to
the "parameterTypes" returned by java.lang.reflect.Method which also is used for creating the
MethodType in order to use a publicLookup.findVirtual(...). Or with other words: the coerced
argument will be identical for both invocation types!


Another strange observation in the success case is as follows: when using reflective invocation by
default (and then only invoking the MethodHandle in the special case that the method "setCategories"
is to be executed) the coerced argument supplied to java.util.Arrays.deepToString() will list the
monthnames:

// // // RexxReflectJava9.processMethod(), ARRIVED: -> [INVOKE], tmpMethod=[public final void
javafx.scene.chart.CategoryAxis.setCategories(javafx.collections.ObservableList)]:
method=[SETCATEGORIES] in
object=[rru.rexxArgs[1]="***@2d809949"/rru.bean="CategoryAxis[id=xAxis,
styleClass=axis]"]

// // // RexxReflectJava9.processMethod(), coercedArgs=[[January, February, March, April, May,
June, July, August, September, October, November, December, ]].getClass().toString()=class
[Ljava.lang.Object;,
         parameterTypes=[interface
javafx.collections.ObservableList].getClass().toString()=class [Ljava.lang.Class;:,
         rru.funcArgs=[[January, February, March, April, May, June, July, August, September,
October, November, December, ]].getClass().toString()=class [Ljava.lang.Object;

// // // RexxReflectJava9.processMethod(), bean=[CategoryAxis[id=xAxis, styleClass=axis]],
methodName=[SETCATEGORIES], coercedArgs=[[January, February, March, April, May, June, July,
August, September, October, November, December, ]]

// // :( RexxReflectJava9.processMethod(), MethodType for Method [public final void
javafx.scene.chart.CategoryAxis.setCategories(javafx.collections.ObservableList)]:
"(ObservableList)void"

// // :( RexxReflectJava9.processMethod(): INSTANCE, mh.bindTo("CategoryAxis[id=xAxis,
styleClass=axis]/class javafx.scene.chart.CategoryAxis").invokeWithArguments(...)

... add2cachedFieldsOrMethods(): rru.memberName=[SETCATEGORIES] ->
rru.keyMemberName=[SETCATEGORIES], rru.invocationType=[INVOKE],
cfm=[CachedFieldOrMethod[mhk=METHOD,mh=MethodHandle(CategoryAxis,ObservableList)void,parameterTypes=[interface
javafx.collections.ObservableList]]]

I double checked that the only difference is in using java.lang.reflect.Method.invoke(...) which
makes the Throwable on the method handle invocation go away (and the monthnames to be shown by
Arrays.deepToString()).

Here is the excerpt of the code section in question that allows the MethodHandle mh to be invoked
successfully with the same coerced argument:

Class<?> parameterTypes[] = tmpMethod.getParameterTypes();
                rru.result=tmpMethod.invoke(rru.bean,coercedArgs);      // java.lang.reflect.Method
... cut ...
            Class <?> returnType=tmpMethod.getReturnType();
            MethodType mt = MethodType.methodType(returnType, parameterTypes);
... cut ...
           // access via MethodHandle with the rights of rru.firstExportedClz
                        mh=thisLookup.findVirtual(rru.firstExportedClz, methodName, mt);
                        // mh=thisLookup.unreflect(tmpMethod);  // same behaviour

                rru.result=mh.bindTo(rru.bean).invokeWithArguments(coercedArgs); // bind to
bean, invoke method

Just commenting out the reflective invocation in line 2 above
("rru.result=tmpMethod.invoke(rru.bean,coercedArgs); ") will cause the MethodHandle invocation to
fail with the reported Throwable!

--- 

Also it seems that java.util.Arrays.deepToString(...) is affected by this, if looking at the String
value it produces in both cases (different number of enclosing square brackets: the failing version
has another pair of enclosing square brackets).

Here the debug code for creating the above outputs ("coercedArgs" will be returned by a coerce
method and will be null, if the "rru.funcArgs" arguments from Rexx could not be coerced according to
the "parameterTypes"):

String tmpStrCoercedArgs= (coercedArgs==null ? null :
Arrays.deepToString(coercedArgs)+".getClass().toString()="+coercedArgs.getClass().toString() );

System.err.println("// // // RexxReflectJava9.processMethod(), coercedArgs="+tmpStrCoercedArgs
                                                               +",\n\t\t
parameterTypes="+Arrays.deepToString(parameterTypes)+".getClass().toString()="+parameterTypes.getClass().toString()
                                                               +":,\n\t\t
rru.funcArgs="+Arrays.deepToString(rru.funcArgs)+".getClass().toString()="+rru.funcArgs.getClass().toString()
                                                               );

---

This strange observation is on Windows 7:

F:\work\svn\bsf4oorexx\trunk\bsf4oorexx.dev\source_cc>java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

---

Would anyone have an idea what the reason could be or have any advice?

---rony

[1] Slides with the address book sample in question; look for the pictures in the section starting
at page: <http://www.rexxla.org/events/2017/presentations/AutoJava-BSF4ooRexx-07-JavaFx-201711.pdf>.

P.S.: Yes, the debug output could be cleaner (evolved from many permutations and formatting in the
past weeks), however this is right from the battle-field and tidying everything up is next on the
todo list.

[Rony G. Flatscher]

Just to conclude: this is something that has nothing to do with jigsaw per se.

If interested you could look further to mlvm-dev ("Da Vinci Machine Project"), here two respective
links: <http://mail.openjdk.java.net/pipermail/mlvm-dev/2018-March/006838.html> and a pure Java
example pointing at the problem:
<http://mail.openjdk.java.net/pipermail/mlvm-dev/2018-March/006846.html>.

---rony

Post by Rony G. Flatscher
In the process of adapting pure reflective code (a Rexx-Java bridge) to use MethodHandles on Java 9
instead, everything seems to be working out so far.
In principle all invocations on the Java side are carried out by first using java.lang.reflect
(Field, Method, Constructor) using the supplied arguments (if the arguments can be coerced to the
respective parameterTypes it gets selected for invocation)  and if a candidate is found an
appropriate MethodHandle gets created, which then gets used to invoke the operation supplying the
coerced arguments, if any. Over the weekend I finalized the basic changes and started to test
against a set of sample/demo applications.
---
While testing a rather complex one (an adaption of the JavaFX address book example enhanced with a
BarChart, [1]), that exhibits a very strange behavior: when setting the values for the CategoryAxis
supplying an ObservableList of the month names in the current Locale, using a MethodHandle and
// // // RexxReflectJava9.processMethod(), ARRIVED: -> [INVOKE], tmpMethod=[public final void
method=[SETCATEGORIES] in
styleClass=axis]"]
// // // RexxReflectJava9.processMethod(),
         parameterTypes=[interface
javafx.collections.ObservableList].getClass().toString()=class [Ljava.lang.Class;:,
[Ljava.lang.Object;
// // :( RexxReflectJava9.processMethod(), MethodType for Method [public final void
"(ObservableList)void"
// // :( RexxReflectJava9.processMethod(): INSTANCE, mh.bindTo("CategoryAxis[id=xAxis,
styleClass=axis]/class javafx.scene.chart.CategoryAxis").invokeWithArguments(...)
// :) :) RexxReflectJava9.processMethod(), MethodHandle
java.lang.ClassCastException: java.base/[Ljava.lang.String; cannot be cast to
java.base/java.lang.String
The supplied ObservableList argument represents the  month names and was created with the help of
"javafx.collections.FXCollections.observableList()" and then using its "addAll(monthNames)" method
to add the String array values returned by DateFormatSymbols.getMonths() to the list.
The supplied argument array "rru.funcArgs" will be coerced according to the reflected
// // // RexxReflectJava9.processMethod(),
         parameterTypes=[interface
javafx.collections.ObservableList].getClass().toString()=class [Ljava.lang.Class;:,
[Ljava.lang.Object;
---
The story is much longer but after quite long debugging sessions, I turned on reflective invoke via
tmpMethod instead of invoking the corresponding MethodHandle, which (surprisingly) works.
Then, in the next step doing the same invocation via the corresponding MethodHandle immediately
after the reflective invocation, allows that invocation to execute successfully as well!
Please note, the supplied coerced argument is in both cases the same! Coercion occurs according to
the "parameterTypes" returned by java.lang.reflect.Method which also is used for creating the
MethodType in order to use a publicLookup.findVirtual(...). Or with other words: the coerced
argument will be identical for both invocation types!
Another strange observation in the success case is as follows: when using reflective invocation by
default (and then only invoking the MethodHandle in the special case that the method "setCategories"
is to be executed) the coerced argument supplied to java.util.Arrays.deepToString() will list the
// // // RexxReflectJava9.processMethod(), ARRIVED: -> [INVOKE], tmpMethod=[public final void
method=[SETCATEGORIES] in
styleClass=axis]"]
// // // RexxReflectJava9.processMethod(), coercedArgs=[[January, February, March, April, May,
June, July, August, September, October, November, December, ]].getClass().toString()=class
[Ljava.lang.Object;,
         parameterTypes=[interface
javafx.collections.ObservableList].getClass().toString()=class [Ljava.lang.Class;:,
         rru.funcArgs=[[January, February, March, April, May, June, July, August, September,
October, November, December, ]].getClass().toString()=class [Ljava.lang.Object;
// // // RexxReflectJava9.processMethod(), bean=[CategoryAxis[id=xAxis, styleClass=axis]],
methodName=[SETCATEGORIES], coercedArgs=[[January, February, March, April, May, June, July,
August, September, October, November, December, ]]
// // :( RexxReflectJava9.processMethod(), MethodType for Method [public final void
"(ObservableList)void"
// // :( RexxReflectJava9.processMethod(): INSTANCE, mh.bindTo("CategoryAxis[id=xAxis,
styleClass=axis]/class javafx.scene.chart.CategoryAxis").invokeWithArguments(...)
... add2cachedFieldsOrMethods(): rru.memberName=[SETCATEGORIES] ->
rru.keyMemberName=[SETCATEGORIES], rru.invocationType=[INVOKE],
cfm=[CachedFieldOrMethod[mhk=METHOD,mh=MethodHandle(CategoryAxis,ObservableList)void,parameterTypes=[interface
javafx.collections.ObservableList]]]
I double checked that the only difference is in using java.lang.reflect.Method.invoke(...) which
makes the Throwable on the method handle invocation go away (and the monthnames to be shown by
Arrays.deepToString()).
Here is the excerpt of the code section in question that allows the MethodHandle mh to be invoked
Class<?> parameterTypes[] = tmpMethod.getParameterTypes();
                rru.result=tmpMethod.invoke(rru.bean,coercedArgs);      // java.lang.reflect.Method
... cut ...
            Class <?> returnType=tmpMethod.getReturnType();
            MethodType mt = MethodType.methodType(returnType, parameterTypes);
... cut ...
           // access via MethodHandle with the rights of rru.firstExportedClz
                        mh=thisLookup.findVirtual(rru.firstExportedClz, methodName, mt);
                        // mh=thisLookup.unreflect(tmpMethod);  // same behaviour
                rru.result=mh.bindTo(rru.bean).invokeWithArguments(coercedArgs); // bind to
bean, invoke method
Just commenting out the reflective invocation in line 2 above
("rru.result=tmpMethod.invoke(rru.bean,coercedArgs); ") will cause the MethodHandle invocation to
fail with the reported Throwable!
--- 
Also it seems that java.util.Arrays.deepToString(...) is affected by this, if looking at the String
value it produces in both cases (different number of enclosing square brackets: the failing version
has another pair of enclosing square brackets).
Here the debug code for creating the above outputs ("coercedArgs" will be returned by a coerce
method and will be null, if the "rru.funcArgs" arguments from Rexx could not be coerced according to
Arrays.deepToString(coercedArgs)+".getClass().toString()="+coercedArgs.getClass().toString() );
System.err.println("// // // RexxReflectJava9.processMethod(), coercedArgs="+tmpStrCoercedArgs
                                                               +",\n\t\t
parameterTypes="+Arrays.deepToString(parameterTypes)+".getClass().toString()="+parameterTypes.getClass().toString()
                                                               +":,\n\t\t
rru.funcArgs="+Arrays.deepToString(rru.funcArgs)+".getClass().toString()="+rru.funcArgs.getClass().toString()
                                                               );
---
F:\work\svn\bsf4oorexx\trunk\bsf4oorexx.dev\source_cc>java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
---
Would anyone have an idea what the reason could be or have any advice?
---rony
[1] Slides with the address book sample in question; look for the pictures in the section starting
at page: <http://www.rexxla.org/events/2017/presentations/AutoJava-BSF4ooRexx-07-JavaFx-201711.pdf>.
P.S.: Yes, the debug output could be cleaner (evolved from many permutations and formatting in the
past weeks), however this is right from the battle-field and tidying everything up is next on the
todo list.