ClassLoader.getResources(String)

[Stephen Colebourne]

I've been trying to use ClassLoader.getResources(String). The entire
application is in one named module, this includes the code that
invokes the ClassLoader method and the resource that it is trying to
find. The Javadoc says:

"Resources in named modules are subject to the encapsulation rules
specified by Module.getResourceAsStream. Additionally, and except for
the special case where the resource has a name ending with ".class",
this method will only find resources in packages of named modules when
the package is opened unconditionally (even if the caller of this
method is in the same module as the resource)."
https://docs.oracle.com/javase/9/docs/api/java/lang/ClassLoader.html#getResources-java.lang.String-

The call to ClassLoader.getResources(String) does not find the
resource. I assume that this is because of the last clause in the spec
"even if the caller of this method is in the same module as the
resource".

But I can't for the life of me think why such a difficult to meet
restriction has been added. The only way around it is to make the
package open, which is far from ideal. If its all within one module,
applying an access restriction like this is just unhelpful.

Stephen

[Alan Bateman]

Post by Stephen Colebourne
I've been trying to use ClassLoader.getResources(String). The entire
application is in one named module, this includes the code that
invokes the ClassLoader method and the resource that it is trying to
find.

Can you summarize what you are trying to do? If this is code in a module
trying to locate one of its own resources then Class.getResourceXXX or
Module.getResourcAsStream are the candidate APIs to use (not
ClassLoader.getResourceXXX as that can never locate resources that are
encapsulated).

-Alan

[Stephen Colebourne]

Post by Alan Bateman

Post by Stephen Colebourne
I've been trying to use ClassLoader.getResources(String). The entire
application is in one named module, this includes the code that
invokes the ClassLoader method and the resource that it is trying to
find.

Can you summarize what you are trying to do? If this is code in a module
trying to locate one of its own resources then Class.getResourceXXX or
Module.getResourcAsStream are the candidate APIs to use (not
ClassLoader.getResourceXXX as that can never locate resources that are
encapsulated).

I was using maven to create a jar-with-dependencies file, so I could
use jlink. With all the code in one jar file, there shouldn't be any
access barriers to worry about.

ClassLoader.getResources(String) worked just fine until Java 9. The
two APIs are not comparable - the ClassLoader one returns all URLs
found, whereas the Class one returns just one URL. Switching API would
change behaviour.

The code can be seen here:
https://github.com/OpenGamma/Strata/blob/master/modules/collect/src/main/java/com/opengamma/strata/collect/io/ResourceConfig.java#L242
It is a core part of the system that loads configuration at startup.

thanks
Stephen

[Alan Bateman]

Post by Stephen Colebourne
I was using maven to create a jar-with-dependencies file, so I could
use jlink. With all the code in one jar file, there shouldn't be any
access barriers to worry about.
ClassLoader.getResources(String) worked just fine until Java 9. The
two APIs are not comparable - the ClassLoader one returns all URLs
found, whereas the Class one returns just one URL. Switching API would
change behaviour.

ClassLoader.getResources searches the class path as it did in JDK 9 and
older, it it just can't locate non-".class" resources in modules when
they are encapsulated. Class loaders are oblivious as to who is
ultimately attempting to load a class or locate a resource (the
initiating and defining loader can be different, they can many class
loaders in the delegation chain).

With the uber modular JAR scenario then all classes for several
libraries are in the same module. This means that the names of resources
in that module are unique. If several libraries have the same resource
then I assume you drop all but one when you create this uber JAR (or
maybe you are merging some of the configuration files, I can't tell). So
I assume you could change this code to use Class.getResource and it will
locate at-most-one resource with a specific name.

To do a proper migration means re-examining ResourceConfig of course.
Using services is likely to be a lot cleaner and more robust than
scanning for configuration files.

-Alan

[Stephen Colebourne]

Following up on this, it does feel like the use case is now simply not
possible. I have a similar problem with
ClassLoader.getResources(String) in threeten-extra.

https://github.com/ThreeTen/threetenbp-extra/blob/master/src/main/java/org/threeten/extra/scale/SystemUTCRules.java#L202

The ThreeTen-Extra project defines a config file
org/threeten/extra/scale/LeapSecond.txt. The code uses
ClassLoader.getResources(String) to find the latest version of the
file, which may be in the threeten-extra jar file, or in any jar file
that uses threeten-extra.jar. ie. to replace the version from
threeten-extra.jar, a user simply has to add a file with the same
name/package to their jar file.

threeten-extra.jar contains org/threeten/extra/scale/LeapSecond.txt
application.jar also contains org/threeten/extra/scale/LeapSecond.txt

Under JPMS this fails, as the resource cannot be located in
org/threeten/extra/scale in a different jar file. But this appears
makes the whole design impossible to make work with JPMS.

The code in threeten-extra.jar cannot possibly know about the package
names of the application.jar, so there is no way for it to find the
config file.

There seem to be only two solutions to this
- ServiceLoader, but that is for code, not config files
- forcing the application to manually register their config file

Both of these provide a markedly worse outcome.

Am I missing something?

Stephen

Post by Alan Bateman

Post by Stephen Colebourne
I was using maven to create a jar-with-dependencies file, so I could
use jlink. With all the code in one jar file, there shouldn't be any
access barriers to worry about.
ClassLoader.getResources(String) worked just fine until Java 9. The
two APIs are not comparable - the ClassLoader one returns all URLs
found, whereas the Class one returns just one URL. Switching API would
change behaviour.

ClassLoader.getResources searches the class path as it did in JDK 9 and
older, it it just can't locate non-".class" resources in modules when they
are encapsulated. Class loaders are oblivious as to who is ultimately
attempting to load a class or locate a resource (the initiating and defining
loader can be different, they can many class loaders in the delegation
chain).
With the uber modular JAR scenario then all classes for several libraries
are in the same module. This means that the names of resources in that
module are unique. If several libraries have the same resource then I assume
you drop all but one when you create this uber JAR (or maybe you are merging
some of the configuration files, I can't tell). So I assume you could change
this code to use Class.getResource and it will locate at-most-one resource
with a specific name.
To do a proper migration means re-examining ResourceConfig of course. Using
services is likely to be a lot cleaner and more robust than scanning for
configuration files.
-Alan

[Alan Bateman]

Post by Stephen Colebourne
Following up on this, it does feel like the use case is now simply not
possible. I have a similar problem with
ClassLoader.getResources(String) in threeten-extra.
https://github.com/ThreeTen/threetenbp-extra/blob/master/src/main/java/org/threeten/extra/scale/SystemUTCRules.java#L202
The ThreeTen-Extra project defines a config file
org/threeten/extra/scale/LeapSecond.txt. The code uses
ClassLoader.getResources(String) to find the latest version of the
file, which may be in the threeten-extra jar file, or in any jar file
that uses threeten-extra.jar. ie. to replace the version from
threeten-extra.jar, a user simply has to add a file with the same
name/package to their jar file.
threeten-extra.jar contains org/threeten/extra/scale/LeapSecond.txt
application.jar also contains org/threeten/extra/scale/LeapSecond.txt
Under JPMS this fails, as the resource cannot be located in
org/threeten/extra/scale in a different jar file. But this appears
makes the whole design impossible to make work with JPMS.
The code in threeten-extra.jar cannot possibly know about the package
names of the application.jar, so there is no way for it to find the
config file.
There seem to be only two solutions to this
- ServiceLoader, but that is for code, not config files
- forcing the application to manually register their config file
Both of these provide a markedly worse outcome.
Am I missing something?

Resources can't be both encapsulated and not encapsulated at the same
time. If a module has a resource in a package that is intended to be
located by code in other modules using ClassLoader getResourceXXX then
it has to open the package.

In the above, then I assume the main issue isn't resource encapsulation,
it's that you've got two modules on the application module class
containing the same package so they can't both be mapped to the
application class loader.

You've dismissed services but I would expect it to provide a nice
solution. The service interface might be very simple, something like:

public interface LeapSecondDataProvider {
    LeapSecondData data();
}

or better still, define methods that allow SystemUTCRules select the
right version of the leap second data. Applications that ship their own
leap data second would ship an implementation of this class. Yes, it's
different to searching the class file for configuration files but a lot
more reliable.

-Alan

[Stephen Colebourne]

You've dismissed services but I would expect it to provide a nice solution.
public interface LeapSecondDataProvider {
LeapSecondData data();
}

Configuration and code are two very different things. Asking projects
and end users to write code for something that should be config is a
huge no-no.

My view is that JPMS has made using configuration files, especially
for libraries, a lot harder. This is a step back in usability. Just so
we are clear, for leap seconds I will now have to ask users to
manually register them using an API where previously they just added a
file. But for OpenGamma Strata, the configuration files are much more
complex and certainly unsuited to be code, even if the backwards
compatibility issues were acceptable. (This is a pattern I've used for
configuration for many years)

Effectively what is needed is another way for a library to be informed
of the presence of the calling application. One possible solution to
this would be to allow users to write module initialization code in
module-info.java. Then an application coder would have a solid
reliable place to put code that registers the additional configuration
files with the low-level library. Something like:

module com.foo.app {
requires org.threeten.extra;

init(ModuleInitContext context) {
UtcRules.registerLeapSecondFile("/com/foo/app/LeapSeconds.txt");
}
}

PS. ServiceLoader is a pain to use in Java 9 too. As a library doesn't
know whether it will run as a named module or on the classpath, I have
to duplicate the service loader configuration - once in
META-INF/services and once in module-info.java, which is horrible. It
also means that the provide() static method is a feature that cannot
be used by libraries.

Stephen

[Mark Raynsford]

On 2018-03-07T15:14:11 +0000

Post by Stephen Colebourne
Effectively what is needed is another way for a library to be informed
of the presence of the calling application. One possible solution to
this would be to allow users to write module initialization code in
module-info.java. Then an application coder would have a solid
reliable place to put code that registers the additional configuration
files with the low-level library.

There's a pattern I've seen used in OSGi that I've considered adapting
for my own use (exposing resource-only modules that don't contain any
code and yet need to be able to cause some other piece of code to
instantiate services on behalf of the module): They call it the
"extender pattern".

https://dzone.com/articles/osgi-42-extender-pattern-and

Briefly, what happens is that you subscribe to an interface that tells
you when bundles (OSGi terminology for artifacts containing modules,
more or less) are added to or removed from the system. In Jigsaw, this
would probably equate to publishing some sort of event that can be
observed whenever someone creates a new module layer. When you get
notified that a bundle has been added, you can scan the manifest of the
bundle (via the standard jar manifest APIs if you like) and can then,
for example, look for manifest fields that tell you where in the jar
file to find application-specific config files.

X-My-Extra-Config-File: /com/example/config.xml

The listening party can then read the config file, instantiate services
as necessary, etc.

It'd need to be handled in a way that ensured that you don't
essentially race the module resolution code; Just because your code
didn't get a chance to subscribe to "module became available" events
until after they'd appeared doesn't mean you should miss the events.
OSGi does handle this (it's written into the spec), but I don't know
quite how the implementations handle it.

[Alan Bateman]

Post by Stephen Colebourne
Configuration and code are two very different things. Asking projects
and end users to write code for something that should be config is a
huge no-no.
My view is that JPMS has made using configuration files, especially
for libraries, a lot harder. This is a step back in usability. Just so
we are clear, for leap seconds I will now have to ask users to
manually register them using an API where previously they just added a
file. But for OpenGamma Strata, the configuration files are much more
complex and certainly unsuited to be code, even if the backwards
compatibility issues were acceptable. (This is a pattern I've used for
configuration for many years)

It's usually cleaner to encapsulate that configuration but if you don't
want to change anything then you can continue to use
ClassLoader.getResources to search for resources as it works exactly as
it did before. Also if you move the configuration file to somewhere like
META-INF/config then it can never be encapsulated.

Post by Stephen Colebourne
PS. ServiceLoader is a pain to use in Java 9 too. As a library doesn't
know whether it will run as a named module or on the classpath, I have
to duplicate the service loader configuration - once in
META-INF/services and once in module-info.java, which is horrible.

Tooling should be able help with cases where you are creating a library
that may be deployed on the class path in some environments and the
module path in others. The `jar` tool does some sanity check in this
area but it could do more.

-Alan