[jsr294-modularity-eg] Module boundaries

Wed Feb 4 04:32:35 EST 2009

Yes, setting module membership at runtime effectively simulates 
'internal', where the .NET assembly-build operation is done lazily.

Module nesting sounds interesting. Could you please send some source 
code which exhibits how it would impact module-private accessibility?

We can certainly have a phone call. I am travelling internationally for 
the next week and, in practice, will not be back in the office until 
Tuesday 17th.

EG members, please indicate when on 17-20th February you could be 
available. Good times would be 9am West Coast, 12pm East Coast, 6pm France.

Alex

Peter Kriens wrote:
> Ok, so you allow a module boundary to be set in runtime, this will make 
> the module keyword in unbound modules reflect the semantics of the .NET 
> 'internal' concept? I agree we should then have a discussion later about 
> how the compiler gets the unbound module boundaries.(Do we have an 
> action list for these things?)
> 
> If I look at software projects than there will be a tendency to put a 
> set of related packages into a bound module, where a number of these 
> bound modules are packaged in a JAR (the unbound module). This seems to 
> scream for nesting of bound modules?
> 
>   VM -> unbound module -> bound module +-> package -> type -+
>                         ^              |           ^        |
>                         +--------------+           +--------+
> 
> And, can we please, pretty please, have a kick-off phone conference now 
> we're a group starting to work on this?
> 
> Kind regards,
> 
>     Peter Kriens
> 
> On 3 feb 2009, at 01:47, Alex Buckley wrote:
> 
>> .NET is not Java.
>>
>> Compiled types in .NET live in non-executable modules (.netmodule 
>> files). To be executed, modules must be packaged into assemblies 
>> (.dll/.exe files).
>> This packaging makes "internal" a bit tricky. "internal" is specified 
>> as "accessible from the same program", where a program is merely a 
>> bunch of source files. A C# compiler must take it on trust that the 
>> .netmodule which accesses an "internal" artifact will be packaged in 
>> the same assembly as the .netmodule which declares the artifact.
>>
>> This is not unreasonable, since .netmodules MUST be packaged. But if 
>> packaged "wrongly", code accessing an "internal" artifact which the 
>> compiler promised was accessible will fail at runtime. You do not want 
>> language features to behave like this. There is effectively a hole in 
>> the C# spec caused by the greater CLI spec stratifying loadable code 
>> into non-executable and executable.
>>
>> Java is different. A JVM does not require classfiles to be packaged in 
>> any way before executing them. Nor does the reference implementation 
>> of a Java compiler require classfiles to be packaged before compiling 
>> against them. These things are not going to change. Since 
>> module-private accessibility is a language and VM construct, it cannot 
>> depend on the future packaging of code still being compiled or 
>> compiled against.
>>
>> (What it can depend on is an API consulted by a Java compiler to 
>> discover whether module M version X and module M version Y have 
>> "equal" versions. The API implementation would be honor-bound to give 
>> consistent answers, similar to classloaders. We'll get to that API 
>> soon, I hope.)
>>
>> This boils down to saying the module-private accessibility and module 
>> membership go hand-in-hand for me. In any case, the first thing a 
>> programmer will ask when seeing a module-private member is: "What 
>> module does this member belong to?". The answer should NOT be "it 
>> depends on a compiler flag or invocation of jar".
>>
>> I will shortly send a Java language grammar and classfile format which 
>> reflect these views.
>>
>> Now, I know this issue is not going to go away. I therefore propose 
>> that IN ADDITION to encoding module membership in source, it should be 
>> possible for the host system to determine module membership. If you 
>> REALLY want the answer above to be "it depends on a compiler flag or 
>> invocation of jar", you can do that. But a Java compiler/VM must defer 
>> to the membership in a source/class file if it exists.
>>
>> Is that enough of a weakening of my requirements #1 and #2?
>>
>> Alex
>>
>> Peter Kriens wrote:
>>> Hmm. .NET shows that you do not have to know all the module 
>>> boundaries during compilation. 'internal' indicates inside the 
>>> assembly and the source assembly is clearly not known during 
>>> compilation. This concept looks attractively simple and is in line 
>>> with common practice.
>>> For 294, during compilation, the JAR files and project source can 
>>> easily be used inside the compiler for the scope. During runtime, the 
>>> module system can set this boundary.
>>> However, I also like a module concept that is defined in the 
>>> language. This will be more complex (and probably should be 
>>> hierarchical) but it should be not attempt to define the deployment 
>>> artifact nor dependencies. That is, it should be more like namespaces.
>>> Kind regards,
>>>    Peter Kriens
>>>    On 28 jan 2009, at 03:56, Alex Buckley wrote:
>>>> I am happy to discuss module-private accessibility first, in that it 
>>>> addresses the limitations of packages on which we all agree.
>>>>
>>>> The issue we cannot ignore is that to check module-private 
>>>> accessibility at compile-time means knowing the modules to which the 
>>>> requesting and requested types belong. If I invoke:
>>>>
>>>> javac Foo.java Bar.java
>>>>
>>>> where Foo.java refers to Bar, then a compiler must be able to 
>>>> compute at least the module name to which Foo belongs as Foo is 
>>>> compiled, if Bar is found to be module-private as it is compiled. 
>>>> There are no deployment artifacts around yet. It shouldn't be a 
>>>> surprise that Foo.class and Bar.class are subsequently emitted to 
>>>> encode their module membership.
>>>>
>>>> So with module accessibility comes module membership. Module 
>>>> dependencies are a quite different topic which we'll get to soon 
>>>> enough, and they are the place where minimizing new concepts is 
>>>> essential.
>>>>
>>>> Alex
>>>>
>>>> Peter Kriens wrote:
>>>>> I think it would make sense to work from first principles
>>>>>  1. What problem are we trying to solve?
>>>>>  2. How do we scope this problem to an acceptable area for all?
>>>>>  3. What requirements should the solution fulfill
>>>>> I would like to have a short discussion about the problem so we are 
>>>>> sure we are all on the same page. I can start off with the problem 
>>>>> I think we are trying to solve.
>>>>> Modularity
>>>>> Modularity is the art of encapsulation and hiding. A module limits 
>>>>> the amount of information outside its boundaries. This reduces the 
>>>>> overall complexity of a system because it becomes possible to 
>>>>> reason (and change) locally about a module instead of understanding 
>>>>> the whole system, and it provides a local namespace that is easier 
>>>>> to work with for humans than a global namespace.
>>>>> Java provides modularity in many places. A type encapsulates its 
>>>>> fields and methods and selectively exposes them to other types with 
>>>>> the access modifies: private, protected and public. A package 
>>>>> encapsulates a number of types and resources. And last, but not 
>>>>> least, the class loader provides a class space that can be distinct 
>>>>> from other class spaces.
>>>>> The modularity that is enabled with class loaders has been 
>>>>> exploited by many. However, the Java Language has little to say 
>>>>> about class loaders; it is outside the scope of the language.
>>>>> So what problem do we need to solve, seeing that Java provides 
>>>>> already so many forms of modularity? There is a sense in the 
>>>>> community that packages are too small for modules. Though the 
>>>>> package names hint at a hierarchy (and this is how they are found 
>>>>> in a file system), this hierarchy does not provide a preferential 
>>>>> treatment for children. Many programs consist of hundreds of 
>>>>> packages. However, except for the limited package concept, the 
>>>>> programmer cannot indicate that the visibility of an artifact 
>>>>> should be limited to the program, or part of the program.
>>>>> A large number of delivery formats have been created over the 
>>>>> years, most of the based on the JAR format (which is also not a 
>>>>> part of the language): WAR, EAR, OSGi bundles, midlets, xlets, and 
>>>>> likely many more proprietary formats. Usually, the modularity of 
>>>>> these modules is based on a rather simplistic hierarchical class 
>>>>> loader model. That is, all classes in ancestors are visible, but 
>>>>> not any siblings or their children. The language is moot on the 
>>>>> point of this type of encapsulation.
>>>>> Delivery modules contain types that depend on types in other 
>>>>> delivery modules. There is no uniformly agreed Java standard to 
>>>>> model these dependencies. During build time, the compiler is 
>>>>> provided with a linear list of JARs and the compiler picks the 
>>>>> first type that matches a name. Only the name is encoded in the 
>>>>> class file, not the originating delivery module. During runtime, 
>>>>> the same process is used to find classes, albeit in a hierarchical 
>>>>> class loader model. This has all so far been outside the Java 
>>>>> Language.
>>>>> It is crucial that we distinguish the language/logic modularity and 
>>>>> the modularity based on the deployment artifact. Interestingly, in 
>>>>> my understanding, .NET makes such a distinction between the 
>>>>> modularity of a delivery unit (called an assembly) and the finer 
>>>>> grained modularity inside a delivery unit, represented by 
>>>>> namespaces. The "internal" keyword indicates that the artifact is 
>>>>> visible only inside an assembly and the "namespace" keyword 
>>>>> provides a hierarchical namespace.
>>>>> Therefore, one can identify two problems in the Java platform 
>>>>> concerning modularity
>>>>>  1. Packages are too limited for proper language/logic modularity
>>>>>  2. Lack of a runtime module concept that is related to deployment
>>>>>     artifacts (physical modularity)
>>>>> Looking at the proposed time frame (EDR by mid-April), attacking 
>>>>> both problems simultaneously seems rather unrealistic based on my 
>>>>> limited experience. I therefore suggest to start with problem #1: 
>>>>> Packages are too limited for proper language/logic modularity. What 
>>>>> do you think?
>>>>> Kind regards,
>>>>> Peter Kriens
>>>>> On 27 jan 2009, at 05:31, Alex Buckley wrote:
>>>>>> Comments welcome.
>>>>>>
>>>>>> 1) Packages are typically arranged in hierarchies, but types can 
>>>>>> only be
>>>>>> used across different branches of the hierarchy by being made public,
>>>>>> which exposes implementation details too widely. Information 
>>>>>> hiding is
>>>>>> further reduced by interface members always being public.
>>>>>>
>>>>>> Non-solution: hierarchical package membership. Redefining existing
>>>>>> well-known semantics is always a bad idea, and this one is especially
>>>>>> complicated. There would need to be a way to stop package-private
>>>>>> artifacts from being accessible to subpackages, or else we would be
>>>>>> strengthening information hiding in one place only to weaken it
>>>>>> elsewhere. Also, there would need to be a way to configure the 
>>>>>> depth of
>>>>>> exposure of package-private artifacts, since artifacts in package A
>>>>>> should sometimes be accessible to A.B and not to A.B.C, and so on.
>>>>>>
>>>>>> 2) Dependencies of one type on another are expressed in source and
>>>>>> classfiles, but there is no standard way for a Java compiler or 
>>>>>> JVM to
>>>>>> interact with its environment to read and resolve dependencies. This
>>>>>> causes compile-time and runtime environments to differ, and 
>>>>>> complicates
>>>>>> any effort to version types available in the (compile-time or 
>>>>>> runtime)
>>>>>> environment.
>>>>>>
>>>>>> Non-solution: standardize the CLASSPATH. One list for all programs 
>>>>>> and
>>>>>> libraries in the JVM makes versioning almost meaningless, and 
>>>>>> packages
>>>>>> whose types occur in multiple CLASSPATH entries can behave poorly. 
>>>>>> (See
>>>>>> Peter Kriens' presentation at Devoxx 2008.)
>>>>>>
>>>>>> 3) Many packages have grown large over the years. They are 
>>>>>> effectively
>>>>>> impossible to refactor now, since it is binary-incompatible to rename
>>>>>> types and dangerous to "split" packages. The next-best option is 
>>>>>> to let
>>>>>> subsets be taken of existing packages, and deliver subsets
>>>>>> independently, but there is no mechanism in the Java language or 
>>>>>> JVM for
>>>>>> that.
>>>>>>
>>>>>> Non-solution: a mechanism for renaming packages and types outside the
>>>>>> Java language. This would require updating Java's model of binary
>>>>>> compatibility, and would make source code incomprehensible.
>>>>>>
>>>>>> Alex
>>>>>> _______________________________________________
>>>>>> jsr294-modularity-eg mailing list
>>>>>> jsr294-modularity-eg at cs.oswego.edu 
>>>>>> <mailto:jsr294-modularity-eg at cs.oswego.edu>
>>>>>> http://cs.oswego.edu/mailman/listinfo/jsr294-modularity-eg
>>>>> ------------------------------------------------------------------------ 
>>>>>
>>>>> _______________________________________________
>>>>> jsr294-modularity-eg mailing list
>>>>> jsr294-modularity-eg at cs.oswego.edu
>>>>> http://cs.oswego.edu/mailman/listinfo/jsr294-modularity-eg
>>>> _______________________________________________
>>>> jsr294-modularity-eg mailing list
>>>> jsr294-modularity-eg at cs.oswego.edu
>>>> http://cs.oswego.edu/mailman/listinfo/jsr294-modularity-eg
>>> _______________________________________________
>>> jsr294-modularity-eg mailing list
>>> jsr294-modularity-eg at cs.oswego.edu
>>> http://cs.oswego.edu/mailman/listinfo/jsr294-modularity-eg
>> _______________________________________________
>> jsr294-modularity-eg mailing list
>> jsr294-modularity-eg at cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/jsr294-modularity-eg
> 
> _______________________________________________
> jsr294-modularity-eg mailing list
> jsr294-modularity-eg at cs.oswego.edu
> http://cs.oswego.edu/mailman/listinfo/jsr294-modularity-eg