DRAFT: Project Jigsaw: The Big Picture (part 1)

I've started drafting an overview of the current state of Jigsaw.
http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01
Comments, questions, and suggestions welcome. Flames will be ignored.

The terms 'library' and 'module path' could use a definition in that
section.

It should maybe be somewhat more clear that the installation of modules
into a module repository (library?) requires special tooling, and that
modules are not editable in-place; these are both things which will
break most developers' assumptions going in.

The role of Jigsaw's versioning scheme at the different phases could be
clarified somewhat. It could be made more clear that Jigsaw imposes a
single version scheme and thus a single module repository format (that's
how it appears at least).

It is worth clarifying that in the Jigsaw model, nothing is exported by
default. There seems to be no provision for filtering the set of
packages, classes, and/or resources which are imported from a dependency.

The disposition of META-INF and resource (non-package) directories is
not clear, especially in terms of export and import behavior.

Negative dependencies would add a great deal of complexity; I would
advise against it (think of Windows NT ACLs and their mixed allow/deny
rule sets; experts knew the best practices but most people just screwed
them up).

One key guiding principle seems to be missing: keep it simple...

--
- DML

Julien Ponge

2011-12-21 09:31:03 UTC

Mark,

I have a bunch of those except flames. So here they are, for what they're worth.

Most applications do not need to add or remove modules dynamically at
run time, nor do they need to use multiple versions of the same
module simultaneously. The module system should be optimized for
common scenarios but also support narrowly-scoped forms of dynamic
multi-version resolution motivated by actual use cases such as, e.g.,
application servers, IDEs, and test harnesses.

Granted, a significant share of applications simply rely on avoiding ClassNotFoundException. Nevertheless there is also a significant share of applications that require some form of side-by-side versioning, dynamic (un)loading and the ability to access functional units through a service locator of some form. This is basically any kind of application that needs to manage some form of lifecycle. This is not just containers: applications deployed to containers often have to deal with dynamic, e.g., a dependent module appearing or disappearing.

While I agree that optimizing for static modules matches a reality, you should not be scared of providing a sensible solution to the dynamic cases. Otherwise people will necessarily hack on top of your APIs and come up with incompatible and discussable solutions.

People have adopted systems like OSGi, or came up with solutions on their own involving classloader wizardry. They may not be perfect but at the very least they do work now.

Narrowly-scoped solutions may be advisable, but at the very least they must not reject prior art that was validated by real needs, especially if you put the long adoption cycles of Java releases into the balance.

The module system does not support general dynamic run-time resolution;
i.e., it is not possible to add or remove dependences or modules after an
application has started running. Sophisticated container-type programs
such as application servers, IDEs, and test harnesses can achieve the
effect of run-time resolution in a limited way by using the module-system
API to install modules into a temporary module library and then run them
from that library.

TODO: Finish implementing run-time module-path support.
TODO: Design and implement container support.

No flame here, but again, this is where Jigsaw will either win or become yet another java.util.logging.

Implementing a container within the JDK would be a mistake, but this does not prevent from adding the support for adding/removing modules at runtime. Do you have any public document discussing the possible approaches for your APIs contracts?

I guess that you will be able to dynamically load a module like you can load a class from a classloader, but without the ability to unload then people will fall in the traps where others (OSGi?) fell: reliance of the GC to clean up, hoping that no stale references exists, etc.

A module's `exports` declarations govern the [accessibility][acc] of the
public types declared in the named packages. It is thus enforced at both
compile time, by the Java compiler, and at run time, by the virtual
machine.

Correct me if I'm wrong, but this is the same as OSGi export clauses, right?

I always felt like it was weird to have public classes that in reality are not being made visible at the package level. In OSGi this leads to JARs / bundles / modules that have public types not being really public depending on the runtime context (classpath, OSGi, etc).

Why not rely on the compilation unit visibility? Like introducing a "module protected visibility" without managing it at the module metadata level? I tend to think that "module protected class Foo { }" is cleaner than "public class Foo { }" only to be made "hidden" in module-info.java by not having a corresponding exports clause.

The `public` modifier makes the types imported into `bar` from `foo`
available to any other module that depends directly upon `bar`.

It may be just me, but I don't find it explicit to have "requires public foo" meaning that the module re-exports from foo. 'reexports foo" as a separate clause may be more readable, although slightly less concise.

But isn't the seminal "The Feel of Java" article all about explicit/readable over implicit/concise? :-)

In this case any other module that depends upon either `bar` or `baz`
will be able to use public types exported by `foo` without depending upon
`foo` itself

Can't this lead to unexpected types visibility at runtime depending on which module was actually resolved as a dependency? Wouldn't it be useful to be defensive regarding what imports bring you in crappy modules by having the possibility of filtering?

Or maybe the "permits" clause could be used just for that?

ISSUE: Should aliases have version numbers? The syntax currently

allows them. They appear to be necessary to support refactoring by
aggregation. In popular native packaging systems, however, the natural
mapping of a module alias is to a virtual package, and virtual packages
don't have version numbers.

ISSUE: Should each module in a set of modules related by local

dependence be required explicitly to permit all the other modules?
That is not the case today, but it is arguably safer.

I think so.

A non-default view can, finally, also declare an entry point different
from that of its containing module's default view, so a single module can
define multiple related entry points. For example, the declaration
module commands {
view cat {
class org.foo.commands.Cat;
}
view find {
class org.foo.commands.Find;
}
view ls {
class org.foo.commands.List;
}
}
defines three entry points: `cat`, `find`, and `ls`.

You may want to add a sentence and/or example to say how "java -m Foo" can pick one view or the other.

Services

What you have here sounds good in principle, especially using ServiceLoader, but we again get to the point of dynamics.

On one hand you have a nominal static module system and a mechanism to bind to services provided by modules in a decoupled fashion. Great. On the other hand you seem not to be willing to have full dynamic modules + services + lifecycle notifications although you seem to intend that there will be an API to still load them dynamically? which means that Jigsaw may likely end up being half-baked here? meaning that people will hack on top of that or resort to solutions like OSGi which will most likely not be 100% 2-ways compatible and have their own issues.

Today is the chance to provide a clean language + JVM solution to address dynamics instead of letting people come up with hacks. Balkanization is what the Java community would rather avoid I suppose.

I'm sure an open discussion such as the ones you are initiating on this list can only lead to pragmatic consensus? and eventually a much needed JSR once it stabilizes.

Cheers

--
Julien Ponge
http://julien.ponge.info/

Glyn Normington

2011-12-21 10:35:44 UTC

Here's a reply I didn't manage to send to the list...

Regards,
Glyn

From: Glyn Normington <gnormington at vmware.com>
Subject: Re: DRAFT: Project Jigsaw: The Big Picture (part 1)
Date: 21 December 2011 09:17:15 GMT
To: mark.reinhold at oracle.com

The draft does not mention interoperation ([1]), with other module systems in general or OSGi in particular, and there is no mention of this in the "Still to come" section. If these requirements have been dropped, it would be good to note that explicitly.
Regards,
Glyn
[1] See section 4 of http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12

Eric Johnson

2011-12-21 13:01:01 UTC

Given the amount of discussion about resolution throughout the entire
document (which I'm glad to see), it sure seems like it should be
mentioned as one of the design principles.

Perhaps something like:
Requirements for resolution change depending on context - For compile
time, resolution should favor the earliest possible version of a module.
At installation time, favoring the latest possible version is likely
appropriate, and further, persisting this configuration for later reuse
and faster load times may be an essential characteristic. For testing
scenarios the need to select a variety of combinations for compatibility
checks will require recomputing resolution with each execution, possibly
with additional constraints added.

-Eric.

mark.reinhold

2011-12-21 18:07:27 UTC

The terms 'library' and 'module path' could use a definition in that section.

Those terms are given brief inline definitions when introduced. They'll
be explained more fully later on, in sections still being written.

It should maybe be somewhat more clear that the installation of modules into a
module repository (library?) requires special tooling, and that modules are not
editable in-place; these are both things which will break most developers'
assumptions going in.

These points will be made in a forthcoming section.

The role of Jigsaw's versioning scheme at the different phases could be
clarified somewhat. It could be made more clear that Jigsaw imposes a single
version scheme and thus a single module repository format (that's how it
appears at least).
It is worth clarifying that in the Jigsaw model, nothing is exported by
default. There seems to be no provision for filtering the set of packages,
classes, and/or resources which are imported from a dependency.
The disposition of META-INF and resource (non-package) directories is not
clear, especially in terms of export and import behavior.

I'll make a note to clarify these points.

Re. single library format: Yes, there's only one.

Re. filtering: You're right, there is at present no way to do that.
We haven't yet seen any use cases for it.

Re. META-INF and resources: Our current thinking is that these are not
subject to import or export; they're just part of a module's private
data. If a module needs to export resources then it can do so via the
ResourceBundle facility, which will be enhanced to export resources in
terms of services (though the details of that remain to be worked out).

Negative dependencies would add a great deal of complexity; I would advise
against it (think of Windows NT ACLs and their mixed allow/deny rule sets;
experts knew the best practices but most people just screwed them up).

I agree they'd add complexity, and I'd prefer to avoid them unless there
are strong use cases in their favor.

One key guiding principle seems to be missing: keep it simple...

Well, we've tried to keep it as simple as possible given our goals and
constraints. Suggestions for ways to make it simpler are welcome.

Bear in mind that most developers will never need to think about permits
clauses, local dependences, or views. I'll see if I can reorganize the
next draft to separate the fundamental features from those that are for
more advanced scenarios.

- Mark

mark.reinhold

2011-12-21 18:38:37 UTC

Those requirements have not been dropped. The features intended to
address them, principally the module-system API, will be covered in
forthcoming sections.

- Mark

Papick Garcia Taboada

2011-12-21 18:55:36 UTC

Hello Mark, hello all others reading here.

Sorry for my bad english, I am not a native speaker. I am just a brazilian java developer (living in old germany for 20 years now) that did in fact have to implement some modular applications/ architectures - sometimes at compiletime, sometimes at runtime. I did some OSGi, but mostly I got away with some maven or ivy magic on compile/ build time and spring components/ configurations at runtime. I am trying to follow news on jigsaw for a while now, but I was never involved in building something any close to a jvm. I even managed to build and showcase jigsaw lately here in a conference, but again, I am not a java rock star.

Unfortunately, I can't reply directly to the mail because I joined the mailing list too late. It was my understanding, that the jigsaw mailing list was for implementation matters only, not for discussing proposals/ requirements/ documents. So my reply comes out of thread.

Reading the "big picture" is see more questions than answers. I simply don't understand what this draft is for. It is titled big picture, but it is draft and part 1. There is no way to address such a topic in so few lines, so my apology if I am not getting it or reading wrongly between the lines. Besides, you have a draft 12 (I did miss the 11 other ones) requirement document online that would better fit the title "big picture".

The whole "we don't have a JSR" and we have some bits of information here and there is - IMHO - doing more damage than helping the cause. I share the 2008 visions (wouldn't it be cool(*), there is not a moment to loose, Mark, do you remember?). But meanwhile the world around java shifted surprisingly, and my focus is now on "there is not a moment to loose" and lesser on "wouldn't it be cool".

Here some thoughts.

Big picture, the design principles:

(1) Modularity is a language construct
(2) Module boundaries should be strongly enforced
(3) Static, single-version module resolution is usually sufficient

Sounds good, but (1) is the whole point of adding modularization to the jdk, or we could just stick to OSGi. A modularization that is not enforced is not (read enforcement as explained here(**) ), so we all agree on (2). That we are going to need split packages loaded in a common class loader because of the mess in the jdk (see java.util package) goes against (2). From the requirements document, draft 12:

In support of platform modularization, it must be possible to declare that the types defined in a specific set of modules must be loaded by the same class loader.

But (3) is something that I really do not get. Again, from the requirements document:

Multi-version support should only be enabled when type collisions can be detected and reported as errors. This most likely means that it cannot be enabled for run-time linking since detecting collisions is nontrivial.

This most likely means?!? Seriously ? just one version at runtime? The whole point of versioning and modularization is that we HAVE version collisions, and we are witnessing repackaging of all sorts circumventing our version conflicts. This is why we started using something like OSGi, despite of all the doing, tooling and architectural pain.

If we can have only one version resolved, we aren?t getting any better than with ivy or maven dependency management and dependency mediation. By the way, the new Big picture is messing around with the terminology we are used to: from maven, spring ? and somehow from common sense ? we know ?scopes? as where to use the artifacts/ components (runtime, compile-time, test-time, etc.). Then we have different life cycles: maven defines a concise project lifecicle. Every single component model defines its own component life cycle, and regarding modules (at the same level of granularity) we have a bundle life cycle defined by OSGi. In this context, take a look at ?Phases? in the big picture?

(*) http://mreinhold.org/blog/cool
(**) http://www.theserverside.com/feature/OSGi-Because-Sharing-Shouldnt-Be-Painful

mark.reinhold

2011-12-21 19:18:53 UTC

Given the amount of discussion about resolution throughout the entire document
(which I'm glad to see), it sure seems like it should be mentioned as one of
the design principles.
Requirements for resolution change depending on context - For compile time,
resolution should favor the earliest possible version of a module. At
installation time, favoring the latest possible version is likely appropriate,
and further, ...

I don't think this is worth elevating to the level of a design principle.
Phase-dependent resolution behavior is nothing new (e.g., Maven does it),
so it's not a distinguishing feature of the Jigsaw module system.

- Mark

mark.reinhold

2011-12-21 21:29:42 UTC

Post by Julien Ponge
I have a bunch of those except flames. So here they are, for what they're
worth.

Thanks for your comments -- replies below.

Post by Julien Ponge
...
Granted, a significant share of applications simply rely on avoiding
ClassNotFoundException. Nevertheless there is also a significant share of
applications that require some form of side-by-side versioning, dynamic (un)
loading and the ability to access functional units through a service locator of
some form. ...

There are definitely important classes of applications that require
fully-dynamic multi-version module resolution and service lookup with
a rich lifecycle API. That's a pretty complicated programming model,
however, and it's not one that most Java developers need, nor is it
required in order to modularize the platform itself. In SE 8 we're
therefore proposing just to solve the simpler, more-common problem,
and to make sure that developers who actually need to use frameworks
like OSGi can do so in a way that works well with the base platform.

Post by Julien Ponge
...
Implementing a container within the JDK would be a mistake, but this does not
prevent from adding the support for adding/removing modules at runtime. Do you
have any public document discussing the possible approaches for your APIs
contracts?

I completely agree that adding a container to the JDK would be a mistake,
and we aren't proposing to do that. You suggest that we could still
support the general dynamic loading and unloading of modules at runtime,
but that would add significant complexity to both the programming model
(i.e., the specification) and the implementation. The Jigsaw design is,
so far, very much simpler than OSGi, and that's largely because we chose
early on not to try to solve all the big problems that OSGi addresses.

As to API documentation, we have some (admittedly sketchy) Javadoc right
now; a forthcoming section of the "big picture" document will have more
details, and we'll be fleshing out the Javadoc as we go.

Post by Julien Ponge
A module's `exports` declarations govern the [accessibility][acc] of the
public types declared in the named packages. It is thus enforced at both
compile time, by the Java compiler, and at run time, by the virtual
machine.
Correct me if I'm wrong, but this is the same as OSGi export clauses, right?

No, it's not. Export declarations in Jigsaw are meaningful in all
phases, whereas in OSGi they're pretty much just a run-time concept.
They're also much stronger than in OSGi, since access to non-exported
types is specifically disallowed at run time by the JVM.

Post by Julien Ponge
I always felt like it was weird to have public classes that in reality are not
being made visible at the package level. In OSGi this leads to JARs / bundles /
modules that have public types not being really public depending on the runtime
context (classpath, OSGi, etc).
Why not rely on the compilation unit visibility? Like introducing a "module
protected visibility" without managing it at the module metadata level? I tend
to think that "module protected class Foo { }" is cleaner than "public class
Foo { }" only to be made "hidden" in module-info.java by not having a
corresponding exports clause.

We've considered that. If we were starting from scratch today, such a
general module-accessibility modifier could well be the way to go. With
countless lines of existing Java code out in the wild, however, our take
is that if developers have to modify all their source code and rebuild
their libraries and systems in order to take advantage of modularity (or,
equivalently but less robustly, run tools over their existing binaries),
then that would be a significant barrier to adoption.

Post by Julien Ponge
The `public` modifier makes the types imported into `bar` from `foo`
available to any other module that depends directly upon `bar`.
It may be just me, but I don't find it explicit to have "requires public foo"
meaning that the module re-exports from foo. 'reexports foo" as a separate
clause may be more readable, although slightly less concise.

I agree that that's arguably more Java-like. It's really a matter of
syntax, so we're going to go with what we have for now; this can easily
be revisited later on.

Post by Julien Ponge
In this case any other module that depends upon either `bar` or `baz`
will be able to use public types exported by `foo` without depending upon
`foo` itself
Can't this lead to unexpected types visibility at runtime depending on which
module was actually resolved as a dependency? Wouldn't it be useful to be
defensive regarding what imports bring you in crappy modules by having the
possibility of filtering?
Or maybe the "permits" clause could be used just for that?

No, `permits` wouldn't work for that.

Whether a module should be able to filter the types that it re-exports is
an interesting question; I'll make a note of it.

Post by Julien Ponge
A non-default view can, finally, also declare an entry point different
from that of its containing module's default view, ...
You may want to add a sentence and/or example to say how "java -m Foo" can pick
one view or the other.

Good point; I'll do that.

Post by Julien Ponge
Services
What you have here sounds good in principle, especially using ServiceLoader,
but we again get to the point of dynamics.
On one hand you have a nominal static module system and a mechanism to bind to
services provided by modules in a decoupled fashion. Great. On the other hand
you seem not to be willing to have full dynamic modules + services + lifecycle
notifications although you seem to intend that there will be an API to still
load them dynamically? which means that Jigsaw may likely end up being
half-baked here? meaning that people will hack on top of that or resort to
solutions like OSGi which will most likely not be 100% 2-ways compatible and
have their own issues.

Our aim is that Jigsaw be "baked enough" that container-type applications
can be built on top of it, and can load and unload independent modular
components or applications. Developers who really need rich dynamism
should go use OSGi or a similar framework.

- Mark

Daniel Siegmann

2011-12-22 01:56:52 UTC

We've considered that. ?If we were starting from scratch today, such a
general module-accessibility modifier could well be the way to go. ?With
countless lines of existing Java code out in the wild, however, our take
is that if developers have to modify all their source code and rebuild
their libraries and systems in order to take advantage of modularity (or,
equivalently but less robustly, run tools over their existing binaries),
then that would be a significant barrier to adoption.

This doesn't seem a very good justification to me. It is likely this
system will be in place for a very long time, and eventually the
amount of new code being written for JDK 8 and later will outweigh the
legacy code.

Why not support both approaches? A "module protected" visibility could
be added and could be the recommended approach for new code which
targets JDK 8 or later. Meanwhile, the visibility of public types
could default to being visible outside the module, but could be
restricted by package in module-info.java.

Or am I overlooking something?

~Daniel

David M. Lloyd

2011-12-22 02:29:36 UTC

Post by Daniel Siegmann

This doesn't seem a very good justification to me. It is likely this
system will be in place for a very long time, and eventually the
amount of new code being written for JDK 8 and later will outweigh the
legacy code.
Why not support both approaches? A "module protected" visibility could
be added and could be the recommended approach for new code which
targets JDK 8 or later. Meanwhile, the visibility of public types
could default to being visible outside the module, but could be
restricted by package in module-info.java.
Or am I overlooking something?

Be sure you're not confusing visibility with accessibility. I think
Mark was talking about accessibility - you seem to be talking about
both. All the major module implementations seem to implement visibility
in terms of packages primarily, and classes secondarily if at all, so
it's not perfectly clear how the accessibility of a class can apply to
its visibility.

As far as accessibility goes, I still believe that the a simple approach
is best: Expand package/default access or protected access (or both) to
mean "module-wide" instead of "package-wide" if the source/class file
version is Java 8 or later (this can be done a number of ways; it is not
my intent to expand on these details at this point though I do have a
few ideas). Because the accessibility setting is already confined to a
module there's no need to retroactively change source code. The package
accessibility limit is not meaningful when a package is confined to a
single module (as it very definitely should be) so this becomes
conceptual dead weight post Java 8 anyways. Why not clean up the trash
while adding a new concept in an intuitive way?

This way you don't have to change a single line of code to take
advantage of the feature... indeed you don't even have to take advantage
of the feature at all if you're just repackaging old class files because
they obviously already work with the more constrained visibility that
they originally shipped with.

--
- DML

Jesse Glick

2011-12-22 11:03:06 UTC

Expand package/default access or protected access (or both) to mean "module-wide"
instead of "package-wide" if the source/class file version is Java 8 or later

For code targeted to 8+ and not using any (nondefault) views, I like this a lot:

1. "public" would mean what it says - to javac, to javadoc, to java; and most importantly, to someone reading an individual source file.

2. Refactoring implementation code inside a module into (or out of, or across) nested packages for clarity would become simpler because you would not need to fiddle with
modifiers - all the impl code would use default access regardless of package location (keeping the code shorter too), with the obvious exception of overridden public methods.

3. The well-known problem of adding an internal-only method (called from other packages) to an exported class would disappear - just use default access. (In current
Jigsaw I think you have to work around this by adding an unexported type in the exported package which delegates to a default-access method; in systems which only allow
whole packages to be exported, the workaround is more intricate, involving a callback in a nonexported package and static initializers.)

4. The module-info.java semantics could permit you to omit "exports" clauses, or perhaps explicitly says "exports *", to mean that any class in any package should be
accessible iff it is marked public. This avoids the bottleneck of editing module-info.java every time you place a new package into the API (thus risking merge conflicts
in an SCM etc.).

If you are using views to export certain packages to certain callers only, then I guess module-info.java could use explicit "exports" clauses as in the current proposal,
while still taking advantage of #3 and perhaps #2.

you don't even have to take advantage of the feature at all if you're
just repackaging old class files because they obviously already work with the more constrained visibility that they originally shipped with.

They would still work, but you may want to ban outside access to implementation packages despite their containing public members. For this case also, an explicit list of
"exports" clauses would work if recompiling with -source 8 or using 'jar uvf *.jar module-info.class'.

David M. Lloyd

2011-12-22 16:07:21 UTC

Post by Jesse Glick
They would still work, but you may want to ban outside access to
implementation packages despite their containing public members. For
this case also, an explicit list of "exports" clauses would work if
recompiling with -source 8 or using 'jar uvf *.jar module-info.class'.

Just a quick note on this - it's not really clear what banning outside
access would entail here. Normal accessibility rules would be a
sufficient check at compile-time for sure. But at run-time, if you have
a module's class loader, unless you use slow/clunky/possibly error-prone
call stack introspection on every class load request you can't really
enforce this visibility constraint. So pretty much all you can do to
enforce this rule is:

1. Only link exported paths (which is already the case, for us at least)
2. Prevent unauthorized access to a module's class loader (via
Permission) when a security manager is present (we use a general
RuntimePermission for all modules but I think for the JDK usage we
could/should be more fine-grained)

I view module ClassLoader access as analogous to having a
java.lang.reflect.Method which has its "accessible" attribute set to
true. Basically anyone who has the reference the the ClassLoader has
full access to the module. So yeah this means that any non-public class
in a public and imported package can be *loaded* by, say,
Class.forName("org.blah.Name",false,moduleClassLoader) but the normal
accessibility checks would apply.

The alternative is to add more run-time checks; the performance penalty
could be substantial.

--
- DML

Jesse Glick

2011-12-22 17:11:52 UTC

at run-time, if you have a module's class loader [...] you
can't really enforce this visibility constraint.

The class in the unexported package might be visible, but the Jigsaw-enhanced VM would ensure that it is not accessible. Just as with reflection generally, Class.forName
might return a Class yet Class.newInstance (for example) would fail with an IllegalAccessException. This is in contrast to existing module systems without VM support,
where you might not be able to link statically against a class in an unexported package, but you could still call it reflectively without special permissions.

I view module ClassLoader access as analogous to having a java.lang.reflect.Method which has its "accessible" attribute set to true.

Again this is a description of current module systems built on older JVMs and only able to control visibility by overriding ClassLoader.loadClass to throw
ClassNotFoundException even when a parent loader can load the class. As I understand it, Jigsaw would enforce the same accessibility on reflective calls as in the
compiler and linker.

Jesse Glick

2011-12-22 19:08:32 UTC

Post by Jesse Glick
3. The well-known problem of adding an internal-only method (called from other packages) to an exported class would disappear

See sun.misc.SharedSecrets as an example.

Julien Ponge

2011-12-22 17:37:55 UTC

Thanks Mark for your feedback. Minor comments below. Hope they are constructive.

Post by mark.reinhold
There are definitely important classes of applications that require
fully-dynamic multi-version module resolution and service lookup with
a rich lifecycle API. ?That's a pretty complicated programming model,
however, and it's not one that most Java developers need, nor is it
required in order to modularize the platform itself. ?In SE 8 we're
therefore proposing just to solve the simpler, more-common problem,
and to make sure that developers who actually need to use frameworks
like OSGi can do so in a way that works well with the base platform.

Granted, such programming models are inherently complicated and most
problems are reminiscent of those encountered in concurrent
programming.

My point which may be hidden in my previous reply is that while I
agree that focusing on the most common problems is absolutely right,
there may be a chance to evolve the language+platform to also cope
with dynamics in a cleaner way that say, OSGi.

I have been using OSGi for some time. I can't blame it for making ways
around Java and the platform so as to provide dynamic modules. It's a
pragmatic solution for now, but it requires such a rigorous discipline
from the developer that it easily and often bites. However if you use
OSGi just as a static module system then it's quite easy to use and it
already does what Jigsaw promises.

Post by mark.reinhold
I completely agree that adding a container to the JDK would be a mistake,
and we aren't proposing to do that. ?You suggest that we could still
support the general dynamic loading and unloading of modules at runtime,
but that would add significant complexity to both the programming model
(i.e., the specification) and the implementation. ?The Jigsaw design is,
so far, very much simpler than OSGi, and that's largely because we chose
early on not to try to solve all the big problems that OSGi addresses.

I still wonder if we couldn't come up with a clean and simple solution
as long as we have some latitude on evolving the specs. This is an
open question, really.

Jigsaw is indeed quite simple in its current form, but things like
views and local requires tend not to be that simple at first... plus
they somehow try to solve OSGI-esque problems (breaking isolation for
some "friend" packages, etc) ;-)

Post by mark.reinhold
As to API documentation, we have some (admittedly sketchy) Javadoc right
now; a forthcoming section of the "big picture" document will have more
details, and we'll be fleshing out the Javadoc as we go.

Nice

Post by mark.reinhold
No, it's not. ?Export declarations in Jigsaw are meaningful in all
phases, whereas in OSGi they're pretty much just a run-time concept.

Post by mark.reinhold
They're also much stronger than in OSGi, since access to non-exported
types is specifically disallowed at run time by the JVM.

I don't get it. Do you mean that you cannot pass a reference of a
non-exported class instance from a module to another module? In this
case it would indeed be stronger than OSGi where you just cannot load
such classes, but you can still pass a reference that was allocated
from the originating module.

Cheers

--
Julien Ponge
http://julien.ponge.info/

mark.reinhold

2011-12-21 21:54:33 UTC

Given the amount of discussion about resolution throughout the entire document
(which I'm glad to see), it sure seems like it should be mentioned as one of
the design principles.
...

On further thought, and prompted by a hallway conversation with Alex,
perhaps the essence of what you're getting at is that modules are
relevant to all phases of development rather than just, say, to run
time. That's definitely a distinguishing aspect of the Jigsaw module
system; I'll work it into the design-principles section in the next
draft.

Thanks,
- Mark

Arnaud Héritier

2011-12-21 23:56:55 UTC

Hi,

After having participating in Maven for years I think it was a wrong
choice to not have a stronger definition/enforcement of versions.
It is in the central place of the resolution system and it becomes a
nightmare when you have to compare Alpha, beta, RC, CR, ... and many others
versions that we may not imagine.
Nowadays we have also a lot of issues because of the conflict between
OSGi versions and Maven SNAPSHOTs.
I agree to relax some rules like having the ability to not use versions
as far as you don't share modules with others, but after that when modules
have to be shared I think that you need to have something really strict to
propose an efficient resolution mechanism.

About modules definitions it isn't clear for me if sources are/can be
considered as resources and thus shared within modules. We have the case
for example to share sources for GWT but in many others cases (from IDEs,
....).
Related to that I see nothing maven classifiers. The major part of time
this is useful especially at compile/development time to reuse some
variations of a same maven artifact (sources, javadocs, ...) but when
jigsaw will be out I would like to have the ability in Maven (and others
build systems) to reuse jigsaw repositories thus they'll need to be
extensible (the repository layout and the identification/resolution system)

Thanks.

Arnaud

David Bosschaert

2011-12-22 11:51:15 UTC

The module definition doesn't look like Java at all to me, take this example
module bar @ 1.0 {
requires foo;
exports bar;
view bar.internal {
permits baz;
exports bar.private;
}
}
so I really don't think it should be part of the java language itself.
I think it will require a lot of extensions to the language and
introduce 'pseudo' keywords - keywords that are only keywords inside
the module section, but not elsewhere.
Besides, I still like to see how this can be made extensible as required in [1].

Additionally, you say that the module declaration is by convention in
a module-info.java file. I think we need something stronger than a
convention here as I don't think you'd want 2 files both declaring the
module definitions inside a single module.
I think we should mandate a single module definition file in a
well-known location such as /META-INF/module-info.jmod where the
format of that module-info.jmod is either a text-based DSL (maybe
similar to the module declaration above, but then extensible) or using
XML.

I have made this point before, but I don't feel it has been taken into
consideration appropriately in the current design.

Best regards,

David

[1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#extensible-module-declarations

I've started drafting an overview of the current state of Jigsaw.
?http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01
Comments, questions, and suggestions welcome. ?Flames will be ignored.
- Mark

Alan Bateman

2011-12-23 11:48:22 UTC

Post by David Bosschaert
Besides, I still like to see how this can be made extensible as required in [1].
[1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#extensible-module-declarations

The compiler just passes it through as the ModuleData attribute [1].

-Alan.

[1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-2.5

Sebastian Sickelmann

2011-12-22 15:17:23 UTC

Hi,

thank you for the good big picture. I know have a somewhat better
feeling what jigsaw will/can be.

the top syntactic

level of a module declaration defines the module?s /default view/.

Further

views of a module?s bindings can be defined using the |view| construct,
which specifies a view name together with a bracketed list of exports

|>module bar {

requires foo;
exports bar;
view bar.internal {
permits baz;
exports bar.private;
}
}

The |bar| module now defines two views. The default view, available
by referencing the module name |bar|, is the same as before?it?s
as if the declaration also said |view bar { exports bar; }|. The new

view,

public

What happens in the following example?

|module bar {
requires foo;
|| permits baz;
|| exports barpackage;
|||
| view bar.publicview {
exports barpackage.public;
}
}

Module bar only allows baz to access the module. Through the view bar.publicview every module can access barpackage and barpackage.public.
|
Isn't here a discrepancy not to inherit the permits clauses of the containing module?

|What happens to the local-dependencies? baz can access|package-private types in barpackage. Can baz also localy depend on barpackage.public?

My next question goes to reexported and version numbers:

|>module foo {

exports foo;
}
module bar {
requires public foo;
}
module baz {
requires public bar;
}
module buz {
requires baz; // Can also use foo's exported types
}|

What would happen if definition of buz and bar are the following:

|module bar {
requires public foo @ 1.0;
}
|

|module buz {
requires baz; // Can also use foo's exported types
requires foo @>= 2.0;
}

The use case is: buz want to use functionality of baz.
baz also provides some functionality for which foo must be reexported. But buz is not using this functionality.
But buz uses some functionlity in foo that is only supported since version 2.0 of foo.
Can buz override the reexported dependency of foo at 1.0 to foo@>=2.0?

We can get in the the same situation if an old version of bar doen't need foo at all. buz already uses foo @>= 2.0
Now buz bar starts using foo @ 1.0 and unintentionally reexportes foo.
||Can buz override the reexported dependency of foo at 1.0 to foo@>=2.0?
|

|-- Sebastian
|

Jesse Glick

2011-12-22 17:51:05 UTC

module baz { // [you wrote bar initially but I think you meant baz]
}
module buz {
requires baz; // Can also use foo's exported types
}
The use case is: buz want to use functionality of baz.
baz also provides some functionality for which foo must be reexported. But buz is not using this functionality.
But buz uses some functionality in foo that is only supported since version 2.0 of foo.

I guess this cannot work and the module system would just refuse to load (or even install) this version of buz. The module system does not "know" that buz is not using
the part of baz referring to foo, so it must assume that it is. And baz has declared that it cannot be used with foo at 2.0, so you are out of luck. This is why the author
of baz needed to be as lenient as possible about the acceptable versions of foo - or at least offer to publish a new version of baz based on foo at 2.0.

The situation is no different if baz does not use the 'public' modifier on its import; buz still explicitly imports foo, and may legally call those parts of baz referring
to foo, with the expectation that the class space is consistent. In other words, the "reexport" feature is a mere convenience, and the underlying problem would exist
without it.

Since Jigsaw does not normally support loading multiple versions of a single module at once, the situation is no different even if baz's public signature makes no mention
of foo's types - if it merely uses foo internally. A module system supporting parallel loading of old and new versions of a single module could fix this if it could
ensure that all static linkages between modules passed consistent versions of imported types. (An OSGi expert should discuss whether it can handle this case and how.)

Jigsaw could also of course offer a mechanism for manually overriding unsatisfied dependencies, in which case someone - either the author of buz, or the author of the
root module (the final app using buz), or the user running the app - has to investigate _why_ the author of baz did not advertise foo at 2.0 compatibility, what the actual
compatibility impact will be in the context of this app, and whether it is safe to proceed anyway.

Sebastian Sickelmann

2011-12-25 10:16:57 UTC

Post by Jesse Glick

module baz { // [you wrote bar initially but I think you meant baz]
}
module buz {
requires baz; // Can also use foo's exported types
}
The use case is: buz want to use functionality of baz.
baz also provides some functionality for which foo must be
reexported. But buz is not using this functionality.
But buz uses some functionality in foo that is only supported since version 2.0 of foo.

I'am knowing that trying to resolve that buz doesn't not need foo at 1.0
cannot be made automatically. I see the reexport more than a "service"
of the implementor/publisher/programmer of buz. If buz uses types of
foo at 1.0 in it's public API than an reexport is a nice feature, so that
the users of buz are not forced to require foo explicit.

Post by Jesse Glick
The situation is no different if baz does not use the 'public'
modifier on its import; buz still explicitly imports foo, and may
legally call those parts of baz referring to foo, with the expectation
that the class space is consistent. In other words, the "reexport"
feature is a mere convenience, and the underlying problem would exist
without it.
Since Jigsaw does not normally support loading multiple versions of a
single module at once, the situation is no different even if baz's
public signature makes no mention of foo's types - if it merely uses
foo internally. A module system supporting parallel loading of old and
new versions of a single module could fix this if it could ensure that
all static linkages between modules passed consistent versions of
imported types. (An OSGi expert should discuss whether it can handle
this case and how.)

But what is the following design principle mean?

/>Static, single-version module resolution is usually sufficient/ ? Most

Post by Jesse Glick
applications do not need to add or remove modules dynamically
at run time, nor do they need to use multiple versions of the same
module simultaneously. The module system should be optimized
for common scenarios but also support narrowly-scoped forms of
dynamic multi-version resolution motivated by actual use cases
such as, /e.g./, application servers, IDEs, and test harnesses.

How can i achieve multi-version resolution?

Post by Jesse Glick
Jigsaw could also of course offer a mechanism for manually overriding
unsatisfied dependencies, in which case someone - either the author of
buz, or the author of the root module (the final app using buz), or
the user running the app - has to investigate _why_ the author of baz
did not advertise foo at 2.0 compatibility, what the actual compatibility
impact will be in the context of this app, and whether it is safe to
proceed anyway.

This is the minimum i would expect. As a user i need a way to override
wrong decisions of the module author.

-- Sebastian

mark.reinhold

2011-12-23 21:34:25 UTC

Post by David Bosschaert
The module definition doesn't look like Java at all to me, take this example
...
I have made this point before, but I don't feel it has been taken into
consideration appropriately in the current design.

You're right, it hasn't. That's intentional, because we've chosen for
now to explore the option of treating module declarations as part of
the language. The ultimate design and specification could well turn
out differently, but going into the JSR process I want to make sure
that we have a good understanding of the linguistic approach.

- Mark

mark.reinhold

2011-12-23 23:36:54 UTC

As you say later on, this is an open question. I'm highly skeptical
that a "clean and simple" design supporting fully-dynamic resolution
is possible, but I'd be happy to be proved wrong.

Post by Julien Ponge
I have been using OSGi for some time. I can't blame it for making ways
around Java and the platform so as to provide dynamic modules.

Me neither -- in fact it's impressive what OSGi can do given that
it's not deeply integrated with the underlying platform.

Post by Julien Ponge
It's a
pragmatic solution for now, but it requires such a rigorous discipline
from the developer that it easily and often bites. However if you use
OSGi just as a static module system then it's quite easy to use and it
already does what Jigsaw promises.

Perhaps, but only for the simplest of scenarios.

Post by Julien Ponge
Correct me if I'm wrong, but this is the same as OSGi export clauses,
right?

No, it's not. Export declarations in Jigsaw are meaningful in all
phases, whereas in OSGi they're pretty much just a run-time concept.

Post by mark.reinhold
They're also much stronger than in OSGi, since access to non-exported
types is specifically disallowed at run time by the JVM.

No, I mean that if you somehow do get a reference to such an object
you won't be able to do anything with it. The JVM's access-checking
machinery won't let you, in the same way and for the same reason that
it won't let you invoke a private method.

- Mark

David M. Lloyd

2011-12-23 23:54:43 UTC

As you say later on, this is an open question. I'm highly skeptical
that a "clean and simple" design supporting fully-dynamic resolution
is possible, but I'd be happy to be proved wrong.

Post by Julien Ponge
I have been using OSGi for some time. I can't blame it for making ways
around Java and the platform so as to provide dynamic modules.

Me neither -- in fact it's impressive what OSGi can do given that
it's not deeply integrated with the underlying platform.

Perhaps, but only for the simplest of scenarios.

Post by Julien Ponge
Correct me if I'm wrong, but this is the same as OSGi export clauses,
right?

No, it's not. Export declarations in Jigsaw are meaningful in all
phases, whereas in OSGi they're pretty much just a run-time concept.

Post by mark.reinhold
They're also much stronger than in OSGi, since access to non-exported
types is specifically disallowed at run time by the JVM.

There needs to be a release valve for this equivalent to
setAccessible(), else frameworks will be much more difficult to
implement. Anyway I'm not really sure there is a good reason to
introduce this level of access checking. It seems like feature creep to
me - there isn't an obvious use case which is being fulfilled here which
isn't adequately met today.

--
- DML

Jesse Glick

2011-12-27 15:22:48 UTC

access to non-exported types is specifically disallowed at run time by the JVM.

Do you mean that you cannot pass a reference of a
non-exported class instance from a module to another module?

No, I mean that if you somehow do get a reference to such an object
you won't be able to do anything with it. The JVM's access-checking
machinery won't let you

There needs to be a release valve for this equivalent to setAccessible()

To make a potentially confusing subject more concrete, let us say you have

module m1 {exports m1;}
package m1;
public abstract class SomeAPI {
public abstract void run();
public static SomeAPI instance() {
return new m1.internal.SomeImpl();
}
}
package m1.internal; // not exported!
public class SomeImpl extends SomeAPI {
@Override public void run() {...}
public void other() {...}
}
module m2 {requires m1;}
package m2;
public class APIClient {
void use() {
m1.SomeAPI obj = m1.SomeAPI.instance();
// ...now see below
}
}

Obviously APIClient.use can call "obj.run()". Almost as obviously, it cannot refer statically to m1.internal.SomeImpl: using modular javac that would not even compile;
and if you used nonmodular javac or an assembler to force it to do so, the module system would throw some sort of linkage error when loading or running APIClient. (A
module system based on a current JRE would throw NoClassDefFoundError, wrapping a ClassNotFoundException intentionally thrown from a ClassLoader.loadClass override on
m2's loader.)

The subtlety is whether APIClient can use reflection to bypass the intent of package exports. In the analogous situation in the NetBeans module system, and I guess in
OSGi as well, the following will run:

obj.getClass().getMethod("other").invoke(obj);

since this bypasses the trigger - attempting to load "m1.internal.SomeImpl" from m2's ClassLoader.

Under Jigsaw I would expect the call to Method.invoke to throw IllegalAccessException, unless you rewrite to

Method m = obj.getClass().getMethod("other");
m.setAccessible(true);
m.invoke(obj);

and the SecurityManager does not complain.

By the way - since Class does not extend AccessibleObject, it is unclear whether

obj.getClass().newInstance();

should also throw IllegalAccessException, and if so, how you could suppress that (given adequate permissions). Since Class.newInstance seems to essentially be a
convenience method, I suppose you could rewrite as

obj.getClass().getConstructor().newInstance();

and if necessary make it accessible:

Constructor<?> c = obj.getClass().getConstructor();
c.setAccessible(true);
c.newInstance();

Here I am assuming that for a "public" class in a package not exported to the caller, every member defined in that class is considered inaccessible, just as if the class
had default access and the caller was in another package. On that topic, a minor nit: the IllegalAccessException detail message needs to explain that the entire class,
not just one member of it, is inaccessible; the following in JDK 7:

package p1;
class Private implements Runnable {
public Private() {}
@Override public void run() {
System.out.println("run");
}
public void other() {
System.out.println("other");
}
}
package p1;
public class Access {
public static Runnable r() {
return new Private();
}
}
package p2;
public class Main {
public static void main(String[] args) throws Exception {
Runnable r = p1.Access.r();
r.run();
r.getClass().getMethod("other").invoke(r);
}
}

prints

run
Exception in thread "main" java.lang.IllegalAccessException: Class p2.Main can not access a member of class p1.Private with modifiers "public"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:95)
at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:261)
at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:253)
at java.lang.reflect.Method.invoke(Method.java:594)
at p2.Main.main(Main.java:6)

which is a little misleading - the modifiers of the member are irrelevant.

David M. Lloyd

2011-12-27 17:48:41 UTC

Post by Jesse Glick

access to non-exported types is specifically disallowed at run time by the JVM.

Do you mean that you cannot pass a reference of a
non-exported class instance from a module to another module?

No, I mean that if you somehow do get a reference to such an object
you won't be able to do anything with it. The JVM's access-checking
machinery won't let you

There needs to be a release valve for this equivalent to setAccessible()

I disagree. I don't see a practical application for this kind of
accessibility constraint. It is redundant with respect to accessibility
modifiers on the class. If we have a module-level accessibility
modifier, then providing an additional check for non-exported public
classes is redundant at best. At worst it makes it impossible for users
to provide a public class under a non-exported package (think again of
frameworks, JavaBeans spec etc.).

I say let visibility be controlled by exports, and let accessibility be
controlled by modifiers, and we sidestep this whole business. It's
cleaner, and frankly less surprising in my opinion.

--
- DML

Jesse Glick

2011-12-27 19:31:40 UTC