Compiling module descriptors is not reproducible?

Discussion:

Mark Raynsford

2018-05-19 09:18:09 UTC

Hello!

I've been interested for a while in making all of my software builds
reproducible [0]. I don't think there's ever been any kind of
written guarantee that the output of javac will be completely
deterministic, but to date it seems like it actually has been. However,
I was a bit disappointed today to find that the output of one of the
builds at AdoptOpenJDK [1] seems to be inserting compiler version
information into compiled module descriptors. It doesn't seem to be
doing this for any other class files.

Here's the output of "openjdk version 10.0.1 2018-04-17" on Linux
(not an AdoptOpenJDK build) when compiling a module descriptor:

---
module com.io7m.jtensors.core
minor version: 0
major version: 53
flags: (0x8000) ACC_MODULE
this_class: #1 // "module-info"
super_class: #0
interfaces: 0, fields: 0, methods: 0, attributes: 2
Constant pool:
#1 = Class #17 // "module-info"
#2 = Utf8 SourceFile
#3 = Utf8 module-info.java
#4 = Utf8 Module
#5 = Module #18 // "com.io7m.jtensors.core"
#6 = Module #19 // "java.base"
#7 = Module #20 // "org.immutables.value"
#8 = Module #21 // "com.io7m.junreachable.core"
#9 = Package #22 // com/io7m/jtensors/core/determinants
#10 = Package #23 // com/io7m/jtensors/core/dotproducts
#11 = Package #24 // com/io7m/jtensors/core/parameterized/matrices
#12 = Package #25 // com/io7m/jtensors/core/parameterized/vectors
#13 = Package #26 // com/io7m/jtensors/core/quaternions
#14 = Package #27 // com/io7m/jtensors/core/unparameterized/matrices
#15 = Package #28 // com/io7m/jtensors/core/unparameterized/vectors
#16 = Package #29 // com/io7m/jtensors/core
#17 = Utf8 module-info
#18 = Utf8 com.io7m.jtensors.core
#19 = Utf8 java.base
#20 = Utf8 org.immutables.value
#21 = Utf8 com.io7m.junreachable.core
#22 = Utf8 com/io7m/jtensors/core/determinants
#23 = Utf8 com/io7m/jtensors/core/dotproducts
#24 = Utf8 com/io7m/jtensors/core/parameterized/matrices
#25 = Utf8 com/io7m/jtensors/core/parameterized/vectors
#26 = Utf8 com/io7m/jtensors/core/quaternions
#27 = Utf8 com/io7m/jtensors/core/unparameterized/matrices
#28 = Utf8 com/io7m/jtensors/core/unparameterized/vectors
#29 = Utf8 com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
Module:
#5,0 // "com.io7m.jtensors.core"
#0
3 // requires
#6,8000 // "java.base" ACC_MANDATED
#0
#7,40 // "org.immutables.value" ACC_STATIC_PHASE
#0
#8,0 // "com.io7m.junreachable.core"
#0
8 // exports
#9,0 // com/io7m/jtensors/core/determinants
#10,0 // com/io7m/jtensors/core/dotproducts
#11,0 // com/io7m/jtensors/core/parameterized/matrices
#12,0 // com/io7m/jtensors/core/parameterized/vectors
#13,0 // com/io7m/jtensors/core/quaternions
#14,0 // com/io7m/jtensors/core/unparameterized/matrices
#15,0 // com/io7m/jtensors/core/unparameterized/vectors
#16,0 // com/io7m/jtensors/core
0 // opens
0 // uses
0 // provides
---

Here's the same module descriptor compiled on "openjdk version
10-internal" on MacOS (this is actually OpenJDK 10+23):

---
module com.io7m.jtensors.core
minor version: 0
major version: 53
flags: (0x8000) ACC_MODULE
this_class: #1 // "module-info"
super_class: #0
interfaces: 0, fields: 0, methods: 0, attributes: 2
Constant pool:
#1 = Class #18 // "module-info"
#2 = Utf8 SourceFile
#3 = Utf8 module-info.java
#4 = Utf8 Module
#5 = Module #19 // "com.io7m.jtensors.core"
#6 = Module #20 // "java.base"
#7 = Utf8 10-internal
#8 = Module #21 // "org.immutables.value"
#9 = Module #22 // "com.io7m.junreachable.core"
#10 = Package #23 // com/io7m/jtensors/core/determinants
#11 = Package #24 // com/io7m/jtensors/core/dotproducts
#12 = Package #25 // com/io7m/jtensors/core/parameterized/matrices
#13 = Package #26 // com/io7m/jtensors/core/parameterized/vectors
#14 = Package #27 // com/io7m/jtensors/core/quaternions
#15 = Package #28 // com/io7m/jtensors/core/unparameterized/matrices
#16 = Package #29 // com/io7m/jtensors/core/unparameterized/vectors
#17 = Package #30 // com/io7m/jtensors/core
#18 = Utf8 module-info
#19 = Utf8 com.io7m.jtensors.core
#20 = Utf8 java.base
#21 = Utf8 org.immutables.value
#22 = Utf8 com.io7m.junreachable.core
#23 = Utf8 com/io7m/jtensors/core/determinants
#24 = Utf8 com/io7m/jtensors/core/dotproducts
#25 = Utf8 com/io7m/jtensors/core/parameterized/matrices
#26 = Utf8 com/io7m/jtensors/core/parameterized/vectors
#27 = Utf8 com/io7m/jtensors/core/quaternions
#28 = Utf8 com/io7m/jtensors/core/unparameterized/matrices
#29 = Utf8 com/io7m/jtensors/core/unparameterized/vectors
#30 = Utf8 com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
Module:
#5,0 // "com.io7m.jtensors.core"
#0
3 // requires
#6,8000 // "java.base" ACC_MANDATED
#7 // 10-internal
#8,40 // "org.immutables.value" ACC_STATIC_PHASE
#0
#9,0 // "com.io7m.junreachable.core"
#0
8 // exports
#10,0 // com/io7m/jtensors/core/determinants
#11,0 // com/io7m/jtensors/core/dotproducts
#12,0 // com/io7m/jtensors/core/parameterized/matrices
#13,0 // com/io7m/jtensors/core/parameterized/vectors
#14,0 // com/io7m/jtensors/core/quaternions
#15,0 // com/io7m/jtensors/core/unparameterized/matrices
#16,0 // com/io7m/jtensors/core/unparameterized/vectors
#17,0 // com/io7m/jtensors/core
0 // opens
0 // uses
0 // provides
---

Note the insertion of the "10-internal" string as constant pool entry
7.

Does anyone have any idea why this would happen? I'm considering filing
a bug at the AdoptOpenJDK project, but I'd like to be sure as to
whether or not this is actually intended behaviour by newer compilers.
Inserting version information obviously breaks reproducibility, so it'd
be nice if it could be turned off!

The sources to the above module descriptor are available [2] if anyone
wants to try compiling this for themselves. The SHA256 of the resulting
module-info.class should be
e7a94b5a2788a3c5cd6d7f586e70a0f2138a2eaa0e75144b6a16e75c4b297870 if
your compiler produces the same output as mine.

[0] https://reproducible-builds.org/
[1] https://adoptopenjdk.net/
[2] https://github.com/io7m/jtensors

--
Mark Raynsford | http://www.io7m.com

Alan Bateman

2018-05-19 11:26:55 UTC

Permalink

Post by Mark Raynsford
Hello!
I've been interested for a while in making all of my software builds
reproducible [0]. I don't think there's ever been any kind of
written guarantee that the output of javac will be completely
deterministic, but to date it seems like it actually has been. However,
I was a bit disappointed today to find that the output of one of the
builds at AdoptOpenJDK [1] seems to be inserting compiler version
information into compiled module descriptors. It doesn't seem to be
doing this for any other class files.

The requires table in the Module attribute can be used by compilers to
record version information about dependences. javac does record the
version information, I don't know if there is a way to disable that. In
any case, I assume your question will be partly answered by looking at
the output of `java --list-modules` on the different builds. If you
download OpenJDK builds from jdk.java.net/10 then you should see that
the standard and JDK-specific modules all report their version string as
"10" or "10.0.1" as they have been built with configure options that
make it so. At a guess, your mail may have included the output from an
exploded (as opposed to images) build.

-Alan

Remi Forax

2018-05-19 12:30:21 UTC

Permalink

----- Mail original -----

Envoyé: Samedi 19 Mai 2018 13:26:55
Objet: Re: Compiling module descriptors is not reproducible?

It seems there is no way:
http://hg.openjdk.java.net/jdk/jdk/file/5ec7380f671d/src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/ClassWriter.java#l986

In any case, I assume your question will be partly answered by looking at
the output of `java --list-modules` on the different builds. If you
download OpenJDK builds from jdk.java.net/10 then you should see that
the standard and JDK-specific modules all report their version string as
"10" or "10.0.1" as they have been built with configure options that
make it so. At a guess, your mail may have included the output from an
exploded (as opposed to images) build.
-Alan

Rémi

Mark Raynsford

2018-05-19 14:01:40 UTC

Permalink

On 2018-05-19T12:26:55 +0100

Post by Alan Bateman
The requires table in the Module attribute can be used by compilers to
record version information about dependences. javac does record the
version information, I don't know if there is a way to disable that. In
any case, I assume your question will be partly answered by looking at
the output of `java --list-modules` on the different builds. If you
download OpenJDK builds from jdk.java.net/10 then you should see that
the standard and JDK-specific modules all report their version string as
"10" or "10.0.1" as they have been built with configure options that
make it so. At a guess, your mail may have included the output from an
exploded (as opposed to images) build.

That's a good point: What I thought I was looking at was the compiler
inserting it's own version into the class file (as in "this class file
was compiled by compiler version 10-internal") as opposed to the
versions of the modules being inserted.

Having the platform module versions present still implies
non-reproducibility but perhaps that's workable in some way.

On 2018-05-19T13:49:40 +0200

Post by Alan Bateman
I think it's the opposite, by default the compiler add the version of the required module,
so the AdoptOpenJDK version does the right thing, you should have the module compiled version by default,
the real question is why the compiler of the openjdk version 10.0.1 doesn't generate the required module version.

That is another good point! This build is the one distributed on Arch
Linux. This was the procedure used to build it (no patches are applied):

https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/java10-openjdk#n58

--
Mark Raynsford | http://www.io7m.com

Remi Forax

2018-05-19 11:49:40 UTC

Permalink

----- Mail original -----

Envoyé: Samedi 19 Mai 2018 11:18:09
Objet: Compiling module descriptors is not reproducible?
Hello!

Hi Mark,

I've been interested for a while in making all of my software builds
reproducible [0]. I don't think there's ever been any kind of
written guarantee that the output of javac will be completely
deterministic, but to date it seems like it actually has been. However,
I was a bit disappointed today to find that the output of one of the
builds at AdoptOpenJDK [1] seems to be inserting compiler version
information into compiled module descriptors. It doesn't seem to be
doing this for any other class files.
Here's the output of "openjdk version 10.0.1 2018-04-17" on Linux
---
module com.io7m.jtensors.core
minor version: 0
major version: 53
flags: (0x8000) ACC_MODULE
this_class: #1 // "module-info"
super_class: #0
interfaces: 0, fields: 0, methods: 0, attributes: 2
#1 = Class #17 // "module-info"
#2 = Utf8 SourceFile
#3 = Utf8 module-info.java
#4 = Utf8 Module
#5 = Module #18 // "com.io7m.jtensors.core"
#6 = Module #19 // "java.base"
#7 = Module #20 // "org.immutables.value"
#8 = Module #21 // "com.io7m.junreachable.core"
#9 = Package #22 // com/io7m/jtensors/core/determinants
#10 = Package #23 // com/io7m/jtensors/core/dotproducts
#11 = Package #24 //
com/io7m/jtensors/core/parameterized/matrices
#12 = Package #25 //
com/io7m/jtensors/core/parameterized/vectors
#13 = Package #26 // com/io7m/jtensors/core/quaternions
#14 = Package #27 //
com/io7m/jtensors/core/unparameterized/matrices
#15 = Package #28 //
com/io7m/jtensors/core/unparameterized/vectors
#16 = Package #29 // com/io7m/jtensors/core
#17 = Utf8 module-info
#18 = Utf8 com.io7m.jtensors.core
#19 = Utf8 java.base
#20 = Utf8 org.immutables.value
#21 = Utf8 com.io7m.junreachable.core
#22 = Utf8 com/io7m/jtensors/core/determinants
#23 = Utf8 com/io7m/jtensors/core/dotproducts
#24 = Utf8 com/io7m/jtensors/core/parameterized/matrices
#25 = Utf8 com/io7m/jtensors/core/parameterized/vectors
#26 = Utf8 com/io7m/jtensors/core/quaternions
#27 = Utf8 com/io7m/jtensors/core/unparameterized/matrices
#28 = Utf8 com/io7m/jtensors/core/unparameterized/vectors
#29 = Utf8 com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
#5,0 // "com.io7m.jtensors.core"
#0
3 // requires
#6,8000 // "java.base" ACC_MANDATED
#0
#7,40 // "org.immutables.value" ACC_STATIC_PHASE
#0
#8,0 // "com.io7m.junreachable.core"
#0
8 // exports
#9,0 // com/io7m/jtensors/core/determinants
#10,0 // com/io7m/jtensors/core/dotproducts
#11,0 //
com/io7m/jtensors/core/parameterized/matrices
#12,0 //
com/io7m/jtensors/core/parameterized/vectors
#13,0 // com/io7m/jtensors/core/quaternions
#14,0 //
com/io7m/jtensors/core/unparameterized/matrices
#15,0 //
com/io7m/jtensors/core/unparameterized/vectors
#16,0 // com/io7m/jtensors/core
0 // opens
0 // uses
0 // provides
---
Here's the same module descriptor compiled on "openjdk version
---
module com.io7m.jtensors.core
minor version: 0
major version: 53
flags: (0x8000) ACC_MODULE
this_class: #1 // "module-info"
super_class: #0
interfaces: 0, fields: 0, methods: 0, attributes: 2
#1 = Class #18 // "module-info"
#2 = Utf8 SourceFile
#3 = Utf8 module-info.java
#4 = Utf8 Module
#5 = Module #19 // "com.io7m.jtensors.core"
#6 = Module #20 // "java.base"
#7 = Utf8 10-internal
#8 = Module #21 // "org.immutables.value"
#9 = Module #22 // "com.io7m.junreachable.core"
#10 = Package #23 // com/io7m/jtensors/core/determinants
#11 = Package #24 // com/io7m/jtensors/core/dotproducts
#12 = Package #25 //
com/io7m/jtensors/core/parameterized/matrices
#13 = Package #26 //
com/io7m/jtensors/core/parameterized/vectors
#14 = Package #27 // com/io7m/jtensors/core/quaternions
#15 = Package #28 //
com/io7m/jtensors/core/unparameterized/matrices
#16 = Package #29 //
com/io7m/jtensors/core/unparameterized/vectors
#17 = Package #30 // com/io7m/jtensors/core
#18 = Utf8 module-info
#19 = Utf8 com.io7m.jtensors.core
#20 = Utf8 java.base
#21 = Utf8 org.immutables.value
#22 = Utf8 com.io7m.junreachable.core
#23 = Utf8 com/io7m/jtensors/core/determinants
#24 = Utf8 com/io7m/jtensors/core/dotproducts
#25 = Utf8 com/io7m/jtensors/core/parameterized/matrices
#26 = Utf8 com/io7m/jtensors/core/parameterized/vectors
#27 = Utf8 com/io7m/jtensors/core/quaternions
#28 = Utf8 com/io7m/jtensors/core/unparameterized/matrices
#29 = Utf8 com/io7m/jtensors/core/unparameterized/vectors
#30 = Utf8 com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
#5,0 // "com.io7m.jtensors.core"
#0
3 // requires
#6,8000 // "java.base" ACC_MANDATED
#7 // 10-internal
#8,40 // "org.immutables.value" ACC_STATIC_PHASE
#0
#9,0 // "com.io7m.junreachable.core"
#0
8 // exports
#10,0 // com/io7m/jtensors/core/determinants
#11,0 // com/io7m/jtensors/core/dotproducts
#12,0 //
com/io7m/jtensors/core/parameterized/matrices
#13,0 //
com/io7m/jtensors/core/parameterized/vectors
#14,0 // com/io7m/jtensors/core/quaternions
#15,0 //
com/io7m/jtensors/core/unparameterized/matrices
#16,0 //
com/io7m/jtensors/core/unparameterized/vectors
#17,0 // com/io7m/jtensors/core
0 // opens
0 // uses
0 // provides
---
Note the insertion of the "10-internal" string as constant pool entry
7.
Does anyone have any idea why this would happen? I'm considering filing
a bug at the AdoptOpenJDK project, but I'd like to be sure as to
whether or not this is actually intended behaviour by newer compilers.
Inserting version information obviously breaks reproducibility, so it'd
be nice if it could be turned off!

I think it's the opposite, by default the compiler add the version of the required module,
so the AdoptOpenJDK version does the right thing, you should have the module compiled version by default,
the real question is why the compiler of the openjdk version 10.0.1 doesn't generate the required module version.

The sources to the above module descriptor are available [2] if anyone
wants to try compiling this for themselves. The SHA256 of the resulting
module-info.class should be
e7a94b5a2788a3c5cd6d7f586e70a0f2138a2eaa0e75144b6a16e75c4b297870 if
your compiler produces the same output as mine.
[0] https://reproducible-builds.org/
[1] https://adoptopenjdk.net/
[2] https://github.com/io7m/jtensors
--
Mark Raynsford | http://www.io7m.com

Rémi