[jsr294-modularity-eg] EG response to 294 comments

[Alex Buckley]

Dear experts,

I would like to welcome BJ Hargrave as IBM's representative on 294,
replacing Glyn Normington who recently left IBM.

Prior to joining the EG, BJ submitted comments during the EDR period. It
seems appropriate to discuss them first, and the handful of comments
from other people afterwards. BJ has given permission for me to quote
his text here on the public EG list; I have numbered them 1..6.


1) Superpackage membership

> The design in the current JSR 294 proposal is overly rigid (e.g. 
> double-link membership, baked-in superpackage membership) and will be 
> fraught with practical issues when used in the real world. There exist and 
> are proposed modularity solutions based upon class loaders and deployment 
> artifacts (JSR 277, OSGi). Membership in a module is more flexibly decided 
> by the way one organizes types into artifacts. Then all that is really 
> needed is a new access modifier for module privacy (to add to the existing 
> public, protected, package private and private access modifiers). This 
> will allow the developer to declare that the type or member is accessible 
> outside of its package but is not accessible outside its module (its 
> module being determined by the artifact in which it is contained; that is, 
> its class loader.). The combination of a module privacy access modifier 
> and artifact-based modularity provide a flexible and simple enhancement to 
> the current environment without the complexity and brittleness of the 
> current JSR 294 proposal.

There is a design space where one extreme makes supercompilation units
supreme:

   A) Superpackage membership is determined by supercompilation units.
      Exports are determined by supercompilation units.

The other extreme makes supercompilation units irrelevant:

   Z) Superpackage membership is determined by compilation units.
      Exports are neither documented by nor determined by
      supercompilation units.
      (All public types are automatically exported, so exports are
       explicit; 'module' types are never exported.)

Currently 294 is somewhere in the middle:

   M) Superpackage membership is determined by compilation units and/or
      supercompilation units, which must be in sync if both are present.
      Exports are determined by supercompilation units.

All points from (A) thru (M) to (Z) recognize that a type's membership
of a superpackage is declared by a static artifact of the Java language. 
This is essential because it makes reasoning about membership 
straightforward. Any compile-time module system aimed at millions of 
programmers cannot have module membership depend on artifacts outside 
the language or on evaluation of arbitrary code.

However, we could try to reduce the rigidity of a supercompilation unit. 
(Z) is appealing but probably goes too far in dropping the concept of an 
export list, which is useful for documentation. Also, the member list 
allows legacy programs to be recompiled without modifying large numbers 
of compilation units. So let's try this point in the design space, 
somewhere between (M) and (Z):

   R) Superpackage membership is determined by compilation units and/or
      supercompilation units, which must be in sync if both are present.
      Exports are documented by, but not determined by, supercompilation
      units.
      (All public types are accessible even without being exported;
       'module' types are never exported.)

Here are the details of point (R):

1) 'public' types continue to be globally accessible in general.
    They can be exported from a supercompilation unit.
    If not exported, their use generates a warning at compile-time.
    This warning indicates that a public type might not be accessible
    at run-time, for example if the type is packaged within a 277 module.
    (This warning is analogous to an unchecked warnings in generics,
     where the compiler can prove code is statically safe but not
     necessarily dynamically safe.)

2) Introduce a 'module' qualifier for types and members.
    'module'-level types cannot be exported from a supercompilation unit.

3) Since module-level types cannot be exported, and since public types
    are accessible even without an export, there is a case for dropping
    nested superpackages.

Here is the development story:

- Given a set of existing public types, write a simple supercompilation 
unit to describe membership and export your.package.*. No changes to 
existing source code are required; just recompile everything. Legacy 
clients will continue to work.

- As time goes on, refine the export list to export fewer public types. 
Old clients will continue to execute, but recompiling them will give 
warnings. Now you have a middle ground between actively supporting old 
clients and actively breaking them.

- You can force clients to migrate to your new API by requalifying types 
from 'public' to 'module'. You can also add 'module foo;' to compilation 
units at your leisure.

- New "internal" types to be shared between packages should be 
'module'-qualified. Once you start using 'module' widely, your 
supercompilation unit doesn't need to change or even be recompiled.

- If you're starting a new project, put 'module foo;' in compilation
units and use 'public' only for types which are exported. Use 'module' 
as necessary for types ahared internally. Write a supercompilation unit 
to document the API and to assist a deployment module system.

Comments?

(Maybe we also have to bite the bullet and do a global rename from 
"superpackage" to "module"...)


2) Classloader restrictions

> Another major problem with the current JSR 294 proposal which 
> significantly decreases its value it the limitation that superpackage 
> cannot span class loaders. A large subsystem deployed in a JSR 277 or OSGi 
> environment will be decomposed into many modules. The modules within the 
> subsystem will need to share types between them but will want to limit 
> access to some of these types to other (3rd party) modules outside the 
> subsystem. So superpackages are in a sense subservient to these modules, 
> which is a shame given that superpackages are a strong concept (in terms 
> of VM enforcement) and multi-class loader superpackages would be necessary 
> to make superpackage useful in a practical sense.

In conversation, BJ expanded on the "practical sense": a deployer wishes
to declare a single superpackage to encapsulate multiple independent
superpackages. This is not possible with the EDR version of nested 
superpackages, since independent superpackages will not declare 
themselves as members of the appropriate enclosing superpackage. To 
support this deployment scenario, a nested superpackage would have to 
declare its enclosing superpackage as 'open' in some way.

Crucially, this would allow a nested superpackage to be a member of
multiple superpackages at once. Some options are:

- Design a complicated scheme to duplicate the types of the nested 
superpackage across multiple classloaders; or

- Drop the single-classloader restriction of a nest and satisfy 
ourselves that the VM's loading constraints will ensure a consistent 
definition of a type in a nested superpackage when loaded by different 
classloaders; or

- Drop nested superpackages entirely, including 'open' nested 
superpackages. (See (6) below)

Comments?



3) Member accessibility

> The current JSR 294 proposal only addresses type access and not member 
> access. It would be very useful to export a type from a superpackage that 
> has members which are only accessible to members of the superpackage. 
> Given the current design, this would vastly complicate the already complex 
> super-package.java file. But using a new module private access modifier in 
> the type source code could easily enable this.

Yes, and point (R) supports this.


4) Wrapping classfiles in superpackages

> The JSR doesn't provide any support for people wanting to wrap existing 
> class files in superpackages. Two use cases are wrapping 3rd party classes 
> that don't allow modification and producing 'bilingual' OSGi bundles that 
> exploit superpackages on Java 7 and later platforms. The excuse is that 
> JSR 294 can't modify the semantics of existing class files but that you 
> can use bytecode rewriting to achieve the necessary effect, but this is 
> not very convincing. If bytecode rewriting is acceptable, then a VM 
> provided mechanism should also be acceptable.

Java SE currently does not include supported APIs or tools to create or
manipulate class files. There are a number of third-party class file
manipulation libraries available, such as ASM. There is already a
version of the ASM library available that supports superpackages, see
http://www.jroller.com/eu/entry/asm_and_jsr_294

Another option is to include some mechanism that avoids class file
rewriting and incorporates some special case logic directly in the VM.
If such an API was added, it would have to be supported indefinitely.
That means that all Java SE VM implementations on all platforms for all
time would need to include additional code to support this special
mechanism, whether it would ever be used by applications or not. Given
that the use cases presented are migration aids used only until the code
is transitioned to superpackages, it seems much preferable to use
libraries such as the ASM library mentioned above.


5) Definition of exports

> JSR 294 doesn't enable exporting all the classes of a named package which 
> is a maintenance headache in an OSGi environment as it would be necessary 
> to do some kind of scan of JAR contents to determine exported classes from 
> an exported package.

The spec allows the 'export com.foo.*' form which exports all public 
types (including inner classes) of a named package.


6) Nested superpackages

> The concept of nested superpackages is not worth the extra specification 
> overhead and complexity that they introduce.

The concept of 'open' nested superpackages from (2) conflicts with (6). 
I like nesting (perhaps without the classloader restriction) because 
large programs should be able to put "internal" packages inside 
"internal" superpackages and then refactor the internals of "internal" 
superpackages without disturbing the top-level superpackage. That's the 
essence of encapsulation.

But I accept that nesting is a generalization which makes the spec much 
more complicated. Also, nesting cannot express arbitrary friendship 
between types, since the friendly types would have to have the same 
enclosing superpackage. Overall, I could live with dropping nesting at 
this time, while knowing that long-term it is desirable.


> Finally, I support all the comments made by Peter Kriens in 
> http://www.osgi.org/blog/2007/11/jsr-294-superpackages.html

FYI, I had some correspondence with Peter about that entry, but we can
talk about that later.

Alex