ServiceLoader.load(Class, ClassLoader) does not load services exposed in modules and loaded by parent CL

[Peter Levart]

Hi,

I stumbled on a problem of a maven plugin that uses JDK's JavaScript
engine and doesn't work when maven is run with JDK 9 or 10. The code in
plugin to initialize the JavaScript engine is as follows:

                    jsEngine = new
ScriptEngineManager().getEngineByName("JavaScript");
                    if (jsEngine == null) {
                        throw new MojoExecutionException("Can't create
JavaScript engine");
                    }

ScriptEngineManager uses thread's context ClassLoader by default to
initialize script service providers:

    public ScriptEngineManager() {
        ClassLoader ctxtLoader =
Thread.currentThread().getContextClassLoader();
        init(ctxtLoader);
    }

...and delegates location of providers to ServiceLoader API:

    private ServiceLoader<ScriptEngineFactory> getServiceLoader(final
ClassLoader loader) {
        if (loader != null) {
            return ServiceLoader.load(ScriptEngineFactory.class, loader);
        } else {
            return ServiceLoader.loadInstalled(ScriptEngineFactory.class);
        }
    }

Maven seems to load plugins in a child class loader of the "system"
(application) class loader and also sets this class loader as thread
context class loader. ServiceLoader does not find any
ScriptEngineFactory services when using this class loader.

The question is: Is this maven plugin's fault? Should the plugin load
services using explicit "system" (application) class loader or should
the ServiceLoader lookup strategy climb the class loader delegation
chain and include service providers that are registered in system Layer
too if given class loader is a descendant of "system" (application)
class loader?

Before JDK 9, using an child class loader of "system" (application)
class loader would locate system services, but since JDK 9, they are
invisible to such child class loaders. Is this intentional? Would it be
wrong if the lookup strategy was more "backwards compatible" ?

Regards, Peter

[Alan Bateman]

Post by Peter Levart
Maven seems to load plugins in a child class loader of the "system"
(application) class loader and also sets this class loader as thread
context class loader. ServiceLoader does not find any
ScriptEngineFactory services when using this class loader.
The question is: Is this maven plugin's fault? Should the plugin load
services using explicit "system" (application) class loader or should
the ServiceLoader lookup strategy climb the class loader delegation
chain and include service providers that are registered in system
Layer too if given class loader is a descendant of "system"
(application) class loader?
Before JDK 9, using an child class loader of "system" (application)
class loader would locate system services, but since JDK 9, they are
invisible to such child class loaders. Is this intentional? Would it
be wrong if the lookup strategy was more "backwards compatible" ?

Are you sure the Maven plugin is setting the TCCL to a child of the
application class loader? Any chance that the parent is null which would
bypass the platform (formerly the extension) class loader?

-Alan

[Peter Levart]

Post by Alan Bateman

Post by Peter Levart
Maven seems to load plugins in a child class loader of the "system"
(application) class loader and also sets this class loader as thread
context class loader. ServiceLoader does not find any
ScriptEngineFactory services when using this class loader.
The question is: Is this maven plugin's fault? Should the plugin load
services using explicit "system" (application) class loader or should
the ServiceLoader lookup strategy climb the class loader delegation
chain and include service providers that are registered in system
Layer too if given class loader is a descendant of "system"
(application) class loader?
Before JDK 9, using an child class loader of "system" (application)
class loader would locate system services, but since JDK 9, they are
invisible to such child class loaders. Is this intentional? Would it
be wrong if the lookup strategy was more "backwards compatible" ?

Are you sure the Maven plugin is setting the TCCL to a child of the
application class loader? Any chance that the parent is null which
would bypass the platform (formerly the extension) class loader?
-Alan

I thought so. This is what debugger told be me about TCCL (debugger
calls .toString() on it by default):

ClassRealm[plugin>com.marand.misc:misc-mojo-git:1.24, parent:
jdk.internal.loader.ClassLoaders$***@1de0aca6]

But now that you asked specifically, I also checked the instance and
evaluated .getParent() on it and got null ?! toString is a damn liar!

It seems that Maven class loaders don't follow standard delegation model
and so JDK can't climb the chain. Sorry for not checking this before.
There's nothing JDK can do about it, I belive.

Thanks,

Peter

[Peter Levart]

Post by Peter Levart

Post by Alan Bateman

Post by Peter Levart
Maven seems to load plugins in a child class loader of the "system"
(application) class loader and also sets this class loader as thread
context class loader. ServiceLoader does not find any
ScriptEngineFactory services when using this class loader.
The question is: Is this maven plugin's fault? Should the plugin
load services using explicit "system" (application) class loader or
should the ServiceLoader lookup strategy climb the class loader
delegation chain and include service providers that are registered
in system Layer too if given class loader is a descendant of
"system" (application) class loader?
Before JDK 9, using an child class loader of "system" (application)
class loader would locate system services, but since JDK 9, they are
invisible to such child class loaders. Is this intentional? Would it
be wrong if the lookup strategy was more "backwards compatible" ?

Are you sure the Maven plugin is setting the TCCL to a child of the
application class loader? Any chance that the parent is null which
would bypass the platform (formerly the extension) class loader?
-Alan

I thought so. This is what debugger told be me about TCCL (debugger
But now that you asked specifically, I also checked the instance and
evaluated .getParent() on it and got null ?! toString is a damn liar!
It seems that Maven class loaders don't follow standard delegation
model and so JDK can't climb the chain. Sorry for not checking this
before. There's nothing JDK can do about it, I belive.

Well, there could be something. But I don't know whether it would be a
good idea. Platform class loader could present services registered in
the boot ModuleLayer as "virtual" resources at standard
META-INF/services resource path locations. ServiceLoader could then
locate them via the ClassLoader.getResources() call even if the class
loaders involved didn't follow standard delegation model, but providing
they would delegate the getResources() invocation in their non-standard
way. Maven class loader seems to delegate to application class loader,
but not via the standard getParent().

Would this be too messy?

Regards, Peter

[Alan Bateman]

Post by Peter Levart
I thought so. This is what debugger told be me about TCCL (debugger
But now that you asked specifically, I also checked the instance and
evaluated .getParent() on it and got null ?! toString is a damn liar!
It seems that Maven class loaders don't follow standard delegation
model and so JDK can't climb the chain. Sorry for not checking this
before. There's nothing JDK can do about it, I belive.

Which Maven plugin is this and is the latest version? The work to get
Maven working with JDK 9 got started in 2015 so I'm surprised that there
are still issues.

-Alan

[Peter Levart]

Post by Alan Bateman

Post by Peter Levart
I thought so. This is what debugger told be me about TCCL (debugger
But now that you asked specifically, I also checked the instance and
evaluated .getParent() on it and got null ?! toString is a damn liar!
It seems that Maven class loaders don't follow standard delegation
model and so JDK can't climb the chain. Sorry for not checking this
before. There's nothing JDK can do about it, I belive.

Which Maven plugin is this and is the latest version? The work to get
Maven working with JDK 9 got started in 2015 so I'm surprised that
there are still issues.

It's not an official plugin. And it seems that the Maven container is to
blame, not the plugin. The nonstandard ClassLoader is supplied by the
container. The plugin just uses the most direct and default API possible
to instantiate JavaScript engine:

jsEngine = new ScriptEngineManager().getEngineByName("JavaScript");

It is the environment the plugin is executing in that doesn't play well
with how system service providers are located from JDK 9 on - namely,
the nonstandard ClassLoader that delegates to system class loader, but
does not express this also in the .getParent() result. I don't know why
Maven choose this, but closer inspection reveals that its ClassLoader
does have a "parent", but it keeps it in its own field called
"parentClassLoader" and doesn't return it from .getParent(). There must
be a reason for this, but I don't know that it is.

Do other parts of the JDK also use TCCL to bootstrap service lookup by
default? Isn't it unusual that ScriptEngineManager uses TCCL by default?

Regards, Peter

Post by Alan Bateman
-Alan

[Alan Bateman]

Post by Peter Levart
It's not an official plugin. And it seems that the Maven container is
to blame, not the plugin.

Robert Scholte is on this mailing list and may be able to comment on this.

Post by Peter Levart
The nonstandard ClassLoader is supplied by the container. The plugin
just uses the most direct and default API possible to instantiate
jsEngine = new ScriptEngineManager().getEngineByName("JavaScript");
It is the environment the plugin is executing in that doesn't play
well with how system service providers are located from JDK 9 on -
namely, the nonstandard ClassLoader that delegates to system class
loader, but does not express this also in the .getParent() result. I
don't know why Maven choose this, but closer inspection reveals that
its ClassLoader does have a "parent", but it keeps it in its own field
called "parentClassLoader" and doesn't return it from .getParent().
There must be a reason for this, but I don't know that it is.
Do other parts of the JDK also use TCCL to bootstrap service lookup by
default? Isn't it unusual that ScriptEngineManager uses TCCL by default?

I wasn't involved in JSR 223 but it may have envisaged scenarios where
applications bundle scripting language implementations. This is not too
unusual and you'll find several APIs do this to allow for cases where an
application is launched in a container environment. Legacy applet and
Java EE containers have historically created a class loader per
"application" and this becomes the TCCL for the threads in that application.

-Alan

[Robert Scholte]

Hi Peter,

you've been hit by MNG-6371[1]. We've tried to solve this in Maven 3.5.1,
but we faced other unexpected classloader issues with maven-extensions and
maven-plugins with extensions. So far we haven't been able to fix it.
This is actually THE reason why Maven 3.5.1 was never released, we
reverted the classloader related changes and successfully released Maven
3.5.2.

If you like a challenge or a bit more info: the main issue is when we
create a new Realm with null[2]. This will create a new ClassLoader with
parent null[3], meaning no bootstrap classloader.

I can quote Alan:
Rhino used to be the JS engine in older releases and that may have been in
rt.jar and so loaded by the boot loader. When Nashorn replaced it (in JDK
8) then it was configured to be defined to the extension class loader so
this is why the code snippet doesn't find it.

In the Maven mailinglist are several threads trying to define how Maven
Classloading should work. So far only a few have mentioned this issue.

IIRC some have worked around it by initializing a new classloader.

thanks,
Robert


[1] https://issues.apache.org/jira/browse/MNG-6371
[2]
https://github.com/apache/maven/blob/4b95ad9fce6dfe7eec2be88f5837e96c7fbd7292/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmManager.java#L123
[3]
https://docs.oracle.com/javase/9/docs/api/java/lang/ClassLoader.html#ClassLoader-java.lang.String-java.lang.ClassLoader-

Post by Alan Bateman

Post by Peter Levart
It's not an official plugin. And it seems that the Maven container is
to blame, not the plugin.

Robert Scholte is on this mailing list and may be able to comment on this.

Post by Peter Levart
The nonstandard ClassLoader is supplied by the container. The plugin
just uses the most direct and default API possible to instantiate
jsEngine = new ScriptEngineManager().getEngineByName("JavaScript");
It is the environment the plugin is executing in that doesn't play well
with how system service providers are located from JDK 9 on - namely,
the nonstandard ClassLoader that delegates to system class loader, but
does not express this also in the .getParent() result. I don't know why
Maven choose this, but closer inspection reveals that its ClassLoader
does have a "parent", but it keeps it in its own field called
"parentClassLoader" and doesn't return it from .getParent(). There must
be a reason for this, but I don't know that it is.
Do other parts of the JDK also use TCCL to bootstrap service lookup by
default? Isn't it unusual that ScriptEngineManager uses TCCL by default?

I wasn't involved in JSR 223 but it may have envisaged scenarios where
applications bundle scripting language implementations. This is not too
unusual and you'll find several APIs do this to allow for cases where an
application is launched in a container environment. Legacy applet and
Java EE containers have historically created a class loader per
"application" and this becomes the TCCL for the threads in that application.
-Alan

[Jochen Theodorou]

Am 24.05.2018 um 19:33 schrieb Robert Scholte:
[...]

Post by Robert Scholte
Rhino used to be the JS engine in older releases and that may have been
in rt.jar and so loaded by the boot loader. When Nashorn replaced it (in
JDK 8) then it was configured to be defined to the extension class
loader so this is why the code snippet doesn't find it.

how is this handles in JDK 9 now?

bye Jochen

[Alan Bateman]

Post by Jochen Theodorou
[...]

Post by Robert Scholte
Rhino used to be the JS engine in older releases and that may have
been in rt.jar and so loaded by the boot loader. When Nashorn
replaced it (in JDK 8) then it was configured to be defined to the
extension class loader so this is why the code snippet doesn't find it.

how is this handles in JDK 9 now?

In the JDK 8, the Nashorn classes were loaded/defined by the extension
class loader. In JDK 9, the jdk.scripting.nashorn module is mapped to
the platform (formerly extension) class loader. So no real change and
the issue that Robert refers to must be an issue with JDK 8 too. I can't
quite tell if this is the same as what Peter is seeing but if Maven is
creating class loaders with null as the parent then some of the platform
classes, and the classes in the service provider modules, will not be
visible.

-Alan

[Peter Levart]

Post by Alan Bateman

Post by Jochen Theodorou
[...]

Post by Robert Scholte
Rhino used to be the JS engine in older releases and that may have
been in rt.jar and so loaded by the boot loader. When Nashorn
replaced it (in JDK 8) then it was configured to be defined to the
extension class loader so this is why the code snippet doesn't find it.

how is this handles in JDK 9 now?

In the JDK 8, the Nashorn classes were loaded/defined by the extension
class loader. In JDK 9, the jdk.scripting.nashorn module is mapped to
the platform (formerly extension) class loader. So no real change and
the issue that Robert refers to must be an issue with JDK 8 too. I
can't quite tell if this is the same as what Peter is seeing but if
Maven is creating class loaders with null as the parent then some of
the platform classes, and the classes in the service provider modules,
will not be visible.
-Alan

The reason why this works on JDK 8 and not on JDK 9+ is that in JDK 8,
the platform/system services are declared in the META-INF/services as
resources. While Maven class loader returns null from
ClassLoader.getParent() it still delegates to extension (platform) class
loader when searching for resources (ClassLoader.getResources()) and
loading classes (ClassLoader.loadClass()), so on JDK 8, the service is
found. Not so on JDK 9, which does not present system/platform services
to ServiceLoader which is initialized with a ClassLoader of which the
chain of parent(s) (traversing the ClassLoader.getParent()) does not
lead to platform class loader. On JDK 9, the platform/system services
are not declared as META-INF/services resources, but in the
module-info.class files...

Regards, Peter

[Peter Levart]

Hi Robert,

Post by Robert Scholte
Hi Peter,
you've been hit by MNG-6371[1]. We've tried to solve this in Maven
3.5.1, but we faced other unexpected classloader issues with
maven-extensions and maven-plugins with extensions. So far we haven't
been able to fix it.
This is actually THE reason why Maven 3.5.1 was never released, we
reverted the classloader related changes and successfully released
Maven 3.5.2.
If you like a challenge or a bit more info: the main issue is when we
create a new Realm with null[2]. This will create a new ClassLoader
with parent null[3], meaning no bootstrap classloader.

I haven't studied code much, so perhaps you could explain why you need
to create such ClassLoader (ClassRealm) that has a null parent (when
calling .getParent() upon it), but then you simulate the parent in a
non-compatibe way:

        ClassRealm classRealm = newRealm( baseRealmId );

        if ( parent != null )
        {
            classRealm.setParentClassLoader( parent );
        }

Such ClassRealm does eventually delegate to parent or any other realm
via strategy. It's just that such delegation is not via official
getParent() but via a non-standard getParentClassLoader().

Why do you need the official getParent() to be null? I see that you use
super.loadClass() in ClassRealm.loadClass() and perhaps you don't want
that super call to "see" anything else than current class path assigned
to current ClassRealm (a sublclass of URLClassLoader) and bootstrap
classes. Unfortunatelly ClassLoader.getParent() is final and you can't
just override it and return the parentClassLoader field...

    private Class<?> unsynchronizedLoadClass( String name, boolean
resolve )
        throws ClassNotFoundException
    {
        try
        {
            // first, try loading bootstrap classes
            return super.loadClass( name, resolve );
        }
        catch ( ClassNotFoundException e )
        {
            // next, try loading via imports, self and parent as
controlled by strategy
            return strategy.loadClass( name );
        }
    }

Instead of using super.loadClass() which 1st delegates to "official"
parent (which you would like it to be null), you could re-implement that
ClassLoader logic by "pretending" that official parent is null although
it would be non-null. Instead of calling super.localClass( name, resolve
), you could call loadClassWithoutParent( name, resolve ) which would be
something like:

    private static final ClassLoder bootstrapDelegator = new
ClassLoader(null);

    private Class<?> loadClassWithoutParent(String name, boolean resolve)
        throws ClassNotFoundException
    {
        synchronized (getClassLoadingLock(name)) {
            // First, check if the class has already been loaded
            Class<?> c = findLoadedClass(name);
            if (c == null) {
                try {
                    c = bootstrapDelegator.loadClass(name);
                } catch (ClassNotFoundException e) {
                    // ClassNotFoundException thrown if bootstrap class
not found
                }

                if (c == null) {
                    // If not found, then invoke findClass in order
                    // to find the class.
                    c = findClass(name);
                }
            }
            if (resolve) {
                resolveClass(c);
            }
            return c;
        }
    }


Would that work?


Regards, Peter

Post by Robert Scholte
Rhino used to be the JS engine in older releases and that may have
been in rt.jar and so loaded by the boot loader. When Nashorn replaced
it (in JDK 8) then it was configured to be defined to the extension
class loader so this is why the code snippet doesn't find it.
In the Maven mailinglist are several threads trying to define how
Maven Classloading should work. So far only a few have mentioned this
issue.
IIRC some have worked around it by initializing a new classloader.
thanks,
Robert
[1] https://issues.apache.org/jira/browse/MNG-6371
[2]
https://github.com/apache/maven/blob/4b95ad9fce6dfe7eec2be88f5837e96c7fbd7292/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmManager.java#L123
[3]
https://docs.oracle.com/javase/9/docs/api/java/lang/ClassLoader.html#ClassLoader-java.lang.String-java.lang.ClassLoader-
On Thu, 24 May 2018 12:29:33 +0200, Alan Bateman

Post by Alan Bateman

Post by Peter Levart
It's not an official plugin. And it seems that the Maven container
is to blame, not the plugin.

Robert Scholte is on this mailing list and may be able to comment on this.

Post by Peter Levart
The nonstandard ClassLoader is supplied by the container. The plugin
just uses the most direct and default API possible to instantiate
jsEngine = new ScriptEngineManager().getEngineByName("JavaScript");
It is the environment the plugin is executing in that doesn't play
well with how system service providers are located from JDK 9 on -
namely, the nonstandard ClassLoader that delegates to system class
loader, but does not express this also in the .getParent() result. I
don't know why Maven choose this, but closer inspection reveals that
its ClassLoader does have a "parent", but it keeps it in its own
field called "parentClassLoader" and doesn't return it from
.getParent(). There must be a reason for this, but I don't know that
it is.
Do other parts of the JDK also use TCCL to bootstrap service lookup
by default? Isn't it unusual that ScriptEngineManager uses TCCL by
default?

I wasn't involved in JSR 223 but it may have envisaged scenarios
where applications bundle scripting language implementations. This is
not too unusual and you'll find several APIs do this to allow for
cases where an application is launched in a container environment.
Legacy applet and Java EE containers have historically created a
class loader per "application" and this becomes the TCCL for the
threads in that application.
-Alan