Java Modules and Multi-Release Jar

[Ralph Goers]

Following advice I received on this list, the Log4j API jar is defined as a multi-release jar with the module-info.java file placed in META-INF/versions/9. This should work fine, yet we continue to have users who are complaining about various tools, such as Javadoc and Eclipse, that can’t handle it properly. One user has gone so far as to suggest we include the automatic module name in MANIFEST.MF in addition to the module-info.java file. When I look at https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path…- <https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path%E2%80%A6-> I can clearly see that what we are doing should be working.

My question is, if I follow this user’s suggestion it will mean that applications that correctly support multi-release jars will treat Log4j API as an explicit module while applications that don’t will treat it as an automatic module. I haven’t seen any documentation on what the consequences of this would be. To me it seems like a bad idea but I have nothing I can point to as to why it would be. Does anyone have any pointers?

FWIW, it seems like the bug I created for Javadoc is marked as fixed for Java 12, so I would assume when Eclipse fixes its problems this would cease to be an issue.

Ralph

[Alan Bateman]

Post by Ralph Goers
Following advice I received on this list, the Log4j API jar is defined as a multi-release jar with the module-info.java file placed in META-INF/versions/9. This should work fine, yet we continue to have users who are complaining about various tools, such as Javadoc and Eclipse, that can’t handle it properly. One user has gone so far as to suggest we include the automatic module name in MANIFEST.MF in addition to the module-info.java file. When I look at https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path…- <https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path%E2%80%A6-> I can clearly see that what we are doing should be working.
My question is, if I follow this user’s suggestion it will mean that applications that correctly support multi-release jars will treat Log4j API as an explicit module while applications that don’t will treat it as an automatic module. I haven’t seen any documentation on what the consequences of this would be. To me it seems like a bad idea but I have nothing I can point to as to why it would be. Does anyone have any pointers?
FWIW, it seems like the bug I created for Javadoc is marked as fixed for Java 12, so I would assume when Eclipse fixes its problems this would cease to be an issue.

I assume the issue with javadoc and modular multi-release JARs is
JDK-8208269 [1]. Are you able to test the Eclipse tool with a JDK 12 EA
build? If it indeed a javadoc only bug then we should try to get the fix
into a JDK 11 update.

I don't recall the discussion here that lead to the advice to put the
module-info.class in META-INF/versions/9 - if you can find the mail
thread or even the subject line of the discussion then it would help
provide the context. In general, the reason to use a MR JARs is where
you are targeting a range of JDK releases and you want to take advantage
of newer APIs on newer releases. Yes, it is possible to have the
module-info.class in the versioned section for the super advanced case
where code in META-INF/versions/10 or META-INF/versions/11 has
additional dependences or makes use of services that the code in the
base section doesn't but I suspect that isn't the case here. Whether the
module-info.class is in the top-level directory or META-INF/versions/9
shouldn't matter of course, except when running into libraries or tools
that can't handle MR JARs.

The Automatic-Module-Name attribute should be ignored if it present in
an explicit module. It may get you past some issue now but I assume will
cause confusion in another context.

-Alan

[1] https://bugs.openjdk.java.net/browse/JDK-8208269

[Luke Hutchison]

Post by Alan Bateman
I don't recall the discussion here that lead to the advice to put the
module-info.class in META-INF/versions/9 - if you can find the mail
thread or even the subject line of the discussion then it would help
provide the context. In general, the reason to use a MR JARs is where
you are targeting a range of JDK releases and you want to take advantage
of newer APIs on newer releases. Yes, it is possible to have the
module-info.class in the versioned section for the super advanced case
where code in META-INF/versions/10 or META-INF/versions/11 has
additional dependences or makes use of services that the code in the
base section doesn't but I suspect that isn't the case here. Whether the
module-info.class is in the top-level directory or META-INF/versions/9
shouldn't matter of course, except when running into libraries or tools
that can't handle MR JARs.

I have a related question about this specifically. I just added
multi-release jar support to ClassGraph (
https://github.com/classgraph/classgraph ). If one or more versioned
sections are present and the code is running on JRE9+, then the
highest-numbered section number less than or equal to the running JRE
version is scanned, and the other sections (including the base section) is
ignored by ClassGraph. Does this more or less match the JRE semantics, or
should I be rather using the versioned section to shadow/mask the base
section, but scan / read from both?

ie. does the JRE find resources and classes in both the versioned section
and the base section, or does the presence of a versioned section preclude
reading from the base section? Your comments seem to indicate the former,
since it sounds like module-info.class can be in either the versioned
section or the base section, and if it is in the base section, that means
module-info.class applies to all versions. That could mean that the base
section shadows the version sections, not the other way around (at least
for module-info.class).

[Alan Bateman]

Post by Luke Hutchison
I have a related question about this specifically. I just added
multi-release jar support to ClassGraph (
https://github.com/classgraph/classgraph ). If one or more versioned
sections are present and the code is running on JRE9+, then the
highest-numbered section number less than or equal to the running JRE
version is scanned, and the other sections (including the base
section) is ignored by ClassGraph. Does this more or less match the
JRE semantics, or should I be rather using the versioned section to
shadow/mask the base section, but scan / read from both?
ie. does the JRE find resources and classes in both the versioned
section and the base section, or does the presence of a versioned
section preclude reading from the base section? Your comments seem to
indicate the former, since it sounds like module-info.class can be in
either the versioned section or the base section, and if it is in the
base section, that means module-info.class applies to all versions.
That could mean that the base section shadows the version sections,
not the other way around (at least for module-info.class).

If you are running on JDK $N then an entry in META-INF/versions/$N will
override an entry of the same name in versioned sections < $N as well as
the base section. The JarFile javadoc and JEP 238 describe this in
detail. One mental model is to think of it as a search path. If you are
JDK 11 then search path for entries in the JAR file is:

    META-INF/versions/11:META-INF/versions/10:META-INF/versions/9:.

where "." is the top-level directory in the JAR file.

You can also verify your understanding by trying some examples with the
JarFile API, e.g. open the JarFile with the 4-arg constructor and use
the versionedStream method to obtain a stream of the JAR entries and
then map each entry to its real path with JarEntry::getRealPath. This
might be useful to verify that your library enumerates the correct set
of entries.

-Alan

[Luke Hutchison]

Thanks for the helpful explanation.

Post by Luke Hutchison
I have a related question about this specifically. I just added
multi-release jar support to ClassGraph (
https://github.com/classgraph/classgraph ). If one or more versioned
sections are present and the code is running on JRE9+, then the
highest-numbered section number less than or equal to the running JRE
version is scanned, and the other sections (including the base section) is
ignored by ClassGraph. Does this more or less match the JRE semantics, or
should I be rather using the versioned section to shadow/mask the base
section, but scan / read from both?
ie. does the JRE find resources and classes in both the versioned section
and the base section, or does the presence of a versioned section preclude
reading from the base section? Your comments seem to indicate the former,
since it sounds like module-info.class can be in either the versioned
section or the base section, and if it is in the base section, that means
module-info.class applies to all versions. That could mean that the base
section shadows the version sections, not the other way around (at least
for module-info.class).
If you are running on JDK $N then an entry in META-INF/versions/$N will
override an entry of the same name in versioned sections < $N as well as
the base section. The JarFile javadoc and JEP 238 describe this in detail.
One mental model is to think of it as a search path. If you are JDK 11 then
META-INF/versions/11:META-INF/versions/10:META-INF/versions/9:.
where "." is the top-level directory in the JAR file.
You can also verify your understanding by trying some examples with the
JarFile API, e.g. open the JarFile with the 4-arg constructor and use the
versionedStream method to obtain a stream of the JAR entries and then map
each entry to its real path with JarEntry::getRealPath. This might be
useful to verify that your library enumerates the correct set of entries.
-Alan

[Jochen Theodorou]

On 30.09.2018 10:33, Alan Bateman wrote:[...]

Post by Alan Bateman
One mental model is to think of it as a search path. If you are
    META-INF/versions/11:META-INF/versions/10:META-INF/versions/9:.
where "." is the top-level directory in the JAR file.

what are the plans if there is for example java 15? Will it search for
15, 14, 13, 12, 11, 10 and then 9? Or will 9 for example be removed from
the search path?

bye Jochen

[Luke Hutchison]

Post by Alan Bateman
If you are running on JDK $N then an entry in META-INF/versions/$N will
override an entry of the same name in versioned sections < $N as well as
the base section. The JarFile javadoc and JEP 238 describe this in detail.
One mental model is to think of it as a search path. If you are JDK 11 then
META-INF/versions/11:META-INF/versions/10:META-INF/versions/9:.
where "." is the top-level directory in the JAR file.
You can also verify your understanding by trying some examples with the
JarFile API, e.g. open the JarFile with the 4-arg constructor and use the
versionedStream method to obtain a stream of the JAR entries and then map
each entry to its real path with JarEntry::getRealPath. This might be
useful to verify that your library enumerates the correct set of entries.

I see in JEP 238 that multi-release jars "will contain" the attribute
"Multi-Release: true". Should this "will" be interpreted as "MUST"? i.e. if
a manifest file is missing this attribute, but the jar contains some
version layers in "META-INF/versions", will the JRE (possibly in some
future version) ignore the version layers, or will it detect the version
layers and use them anyway?

(I'm trying to match the semantics and the intent, including any possible
future behavior if multi-release jars become the norm, and I'm wondering if
I should ignore everything in "META-INF/versions" if the Multi-Release
attribute is missing.)

[Alan Bateman]

Post by Luke Hutchison
I see in JEP 238 that multi-release jars "will contain" the attribute
"Multi-Release: true". Should this "will" be interpreted as "MUST"?
i.e. if a manifest file is missing this attribute, but the jar
contains some version layers in "META-INF/versions", will the JRE
(possibly in some future version) ignore the version layers, or will
it detect the version layers and use them anyway?
(I'm trying to match the semantics and the intent, including any
possible future behavior if multi-release jars become the norm, and
I'm wondering if I should ignore everything in "META-INF/versions" if
the Multi-Release attribute is missing.)

If the JAR file does not contain "Multi-Release" attribute in its main
manifest then it is not a multi-release JAR. The wording in the JarFile
API spec might be a bit clearer.

-Alan.

[Ralph Goers]

Post by Ralph Goers
Following advice I received on this list, the Log4j API jar is defined as a multi-release jar with the module-info.java file placed in META-INF/versions/9. This should work fine, yet we continue to have users who are complaining about various tools, such as Javadoc and Eclipse, that can’t handle it properly. One user has gone so far as to suggest we include the automatic module name in MANIFEST.MF in addition to the module-info.java file. When I look at https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path…- <https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path%E2%80%A6-> I can clearly see that what we are doing should be working.
My question is, if I follow this user’s suggestion it will mean that applications that correctly support multi-release jars will treat Log4j API as an explicit module while applications that don’t will treat it as an automatic module. I haven’t seen any documentation on what the consequences of this would be. To me it seems like a bad idea but I have nothing I can point to as to why it would be. Does anyone have any pointers?
FWIW, it seems like the bug I created for Javadoc is marked as fixed for Java 12, so I would assume when Eclipse fixes its problems this would cease to be an issue.

I assume the issue with javadoc and modular multi-release JARs is JDK-8208269 [1]. Are you able to test the Eclipse tool with a JDK 12 EA build? If it indeed a javadoc only bug then we should try to get the fix into a JDK 11 update.
I don't recall the discussion here that lead to the advice to put the module-info.class in META-INF/versions/9 - if you can find the mail thread or even the subject line of the discussion then it would help provide the context. In general, the reason to use a MR JARs is where you are targeting a range of JDK releases and you want to take advantage of newer APIs on newer releases. Yes, it is possible to have the module-info.class in the versioned section for the super advanced case where code in META-INF/versions/10 or META-INF/versions/11 has additional dependences or makes use of services that the code in the base section doesn't but I suspect that isn't the case here. Whether the module-info.class is in the top-level directory or META-INF/versions/9 shouldn't matter of course, except when running into libraries or tools that can't handle MR JARs.
The Automatic-Module-Name attribute should be ignored if it present in an explicit module. It may get you past some issue now but I assume will cause confusion in another context.
-Alan
[1] https://bugs.openjdk.java.net/browse/JDK-8208269

Yes, that is the link. I am not sure what the problem is with Eclipse, but getting the fix into JDK 11 would be helpful as I suspect it is going to be more widely used than JDK 12, depending on how the various free support options play out.

As for the discussion on why the module info was placed into META-INF/versions/9, I brought up issues with tools that were trying to use Log4j and couldn’t handle finding a class compiled with Java 9 among other classes compiled with Java 7. The suggestion was made to move it into META-INF/version/9 so it would be ignored - which did help with some tools but not others. For example, I think Android still has not been fixed.

Ralph

[Alan Bateman]

Post by Ralph Goers
As for the discussion on why the module info was placed into META-INF/versions/9, I brought up issues with tools that were trying to use Log4j and couldn’t handle finding a class compiled with Java 9 among other classes compiled with Java 7. The suggestion was made to move it into META-INF/version/9 so it would be ignored - which did help with some tools but not others. For example, I think Android still has not been fixed.

There was a thread on core-libs-dev in Dec 2017 with subject line
"Android and Log4j". I see Martin Buchholz replied to say that the
issues with the Android tools were fixed:
http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-December/050382.html

In the thread I see Uwe Schindler suggested moving the module-info.class
META-INF/versions/9 but I don't think it was established if that worked
around the limitations of the tools or not. One of the linked issues in
the discussion suggests these tools weren't unpdaetd to ignore class
files in META-INF until recently:
  https://issuetracker.google.com/u/0/issues/77587908

-Alan.

[Ralph Goers]

Post by Ralph Goers
Following advice I received on this list, the Log4j API jar is defined as a multi-release jar with the module-info.java file placed in META-INF/versions/9. This should work fine, yet we continue to have users who are complaining about various tools, such as Javadoc and Eclipse, that can’t handle it properly. One user has gone so far as to suggest we include the automatic module name in MANIFEST.MF in addition to the module-info.java file. When I look at https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path…- <https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path%E2%80%A6-> I can clearly see that what we are doing should be working.
My question is, if I follow this user’s suggestion it will mean that applications that correctly support multi-release jars will treat Log4j API as an explicit module while applications that don’t will treat it as an automatic module. I haven’t seen any documentation on what the consequences of this would be. To me it seems like a bad idea but I have nothing I can point to as to why it would be. Does anyone have any pointers?
FWIW, it seems like the bug I created for Javadoc is marked as fixed for Java 12, so I would assume when Eclipse fixes its problems this would cease to be an issue.

I assume the issue with javadoc and modular multi-release JARs is JDK-8208269 [1]. Are you able to test the Eclipse tool with a JDK 12 EA build? If it indeed a javadoc only bug then we should try to get the fix into a JDK 11 update.
I don't recall the discussion here that lead to the advice to put the module-info.class in META-INF/versions/9 - if you can find the mail thread or even the subject line of the discussion then it would help provide the context. In general, the reason to use a MR JARs is where you are targeting a range of JDK releases and you want to take advantage of newer APIs on newer releases. Yes, it is possible to have the module-info.class in the versioned section for the super advanced case where code in META-INF/versions/10 or META-INF/versions/11 has additional dependences or makes use of services that the code in the base section doesn't but I suspect that isn't the case here. Whether the module-info.class is in the top-level directory or META-INF/versions/9 shouldn't matter of course, except when running into libraries or tools that can't handle MR JARs.
The Automatic-Module-Name attribute should be ignored if it present in an explicit module. It may get you past some issue now but I assume will cause confusion in another context.
-Alan
[1] https://bugs.openjdk.java.net/browse/JDK-8208269 <https://bugs.openjdk.java.net/browse/JDK-8208269>

To follow up with the main part of my question, if some tooling thinks it is an automatic module when it is an explicit module what problems might that cause? For example, the Log4j API uses the ServiceLoader to locate the Log4j implementation. That is explicitly declared in module-info.java. We also declare what packages are exposed. I am assuming the tool might think all the packages are exposed as an automatic module but then have it fail at runtime because they aren’t. I can see that ModuleFinder says it first checks for a module-info.class and only if it doesn’t find one looks for the manifest header. Is that the only code that cares about module-info or might something else handle it differently?

Ralph