OpenJDK Penrose Project

Tim Ellison Senior Technical Staff Member IBM United Kingdom Ltd. OpenJDK Penrose Project CON-1325

About the Speaker • Technical staff member in the Java Technology Centre, Hursley UK • Working on various runtime technologies for >20 years • Experience in open source communities • Currently focused on class library design and delivery • Overall technical lead for IBM Java 8 SE tim_ellison@uk.ibm.com

Agenda • Why modularity is important to the Java runtime • Introduction to modularity • Modules and the JRE • Java runtime modularity systems • The Jigsaw Project • OSGi • Project Penrose • Implications of runtime modularity • What's changing? • Likely impact to your application • Advice and call to arms

What is a Module? Why do we need a module system? • Modules facilitate software engineering & enhanced runtime capabilities • Decomposition of complex systems • Software assemblies of pre-built modules • Provisioning / deployment, including module repositories • Versioning and configuration management • The module system is used to manage the modules, including installing, activating, resolving module requirements and conflicts. • Interacting with the module system provides a higher level abstraction than the language provides directly.

Desirable characteristics of a module • A module should be highly coherent • A module should be loosely coupled to other modules. • A module should be explicit about what it providesto other modules. • A module should be explicit about what it depends on from other modules.

Modules in Java SE • The Java language already has two kinds of module: classes and packages. • classes encapsulate fields and methods • packages encapsulate classes and resources • Problem: neither of these is a deployableunit (too granular, too tightly coupled). • Java's unit of deployment is the JAR file. • No encapsulation, therefore not a module! • Problem: need an entity that is a deployable unit of encapsulation.

Why modules for the Java runtime (JRE)? • Presently the JRE is a monolithic entity • in general the runtime must assume your application will use anything and everything • class loaders provide runtime type definition and isolation • download time and start-up time are directly impacted by the number of types available in the runtime • Application start-up time includes a linear search through the class path to find system and application code • e.g. Oracle 1.7 Windows bootclass path contains nearly 20k classes • resources.jar rt.jar jsse.jar jce.jar charsets.jar • rt.jar index alone is ~1Mb • To the JRE, your application's jars’ indexes are disjoint & unsorted • there are JRE implementation 'tricks' like look aside tables and shared classes that can help • class loading is a BIG time hog to amortize over the length of the run

Why modules for the Java runtime (JRE)? - continued • Dependency management • avoid “JAR hell” → trying satisfy competing requirements by simple path ordering • e.g. my app depends upon foo-v2.jar and bar-v2.jar but foo-v2.jar depends upon bar-v1.jar - classpath foo-v2.jar; bar-v2.jar; bar-v1.jar → my app “wins” - classpath foo-v2.jar; bar-v1.jar; bar-v2.jar → foo “wins” • a type name space through (unversioned) package name is often insufficient • Module level visibility • public, protected, private and package-private level visibility means some implementation types (com.sun.) need to be marked public to be called by java. APIs. • convention asks people not to depend upon them... • Version control • ability to define the compatibility of a module based on numbering scheme • different module versions for different target devices / resource goals

Introducing Project Jigsaw • Project Jigsaw is an effort currently underway at OpenJDK • defining a simple module system for Java • modularizing the JRE itself • originally intended as an implementation detail for the JRE, but being adopted as a Java standard (JSR) for applications too • originally targeted to Java 8 SE, now moved to Java 9 SE release • Working within constraints of backwards compatibility • existing Java APIs “cannot” be changed • some packages contain wildly different functionalitye.g. java.util contains collection hierarchy and Locale infrastructure • Introduces language changes to define modules src/com/example/myclass.java src/com/example/myclassimpl.java src/module-info.java module com.example @ 1.0 { requires jdk.base; requires foo @ 2.0; requires bar @ 2.0;}

Introducing OSGi • Existing, mature, standards-based modularity solution forJava applications • Rich modularity model based on code-level (package) dependencies, module-level dependencies, and services. • First practical solution for effective component reuse • well supported by multiple implementations, tooling, etc. • Millions use applications built with OSGi • most application servers are based on OSGi • No language changes. • defines info in META-INF/MANIFEST.MF Bundle-ManifestVersion: 2 Bundle-SymbolicName: com.ex.mybundle Bundle-Version: 1.0.0 Import-Package: com.example.bar Export-Package: com.example.foo

Multiple dependency resolution in Jigsaw and OSGi • The module system must resolve bundle dependencies, possibly in the face of multiple valid alternatives • e.g. my app depends upon (foo-v2.jar and (bar-v1.jar or bar-v2.jar)) andfoo-v2.jar depends upon bar-v1.jar • if my app touches bar first, the module system may chose bar-v2.jar • if my app touches foo first, the module system will chose bar-v1.jar • OSGi's dynamic dependency resolution attempts to find the best fit for currently invoked bundles at runtime • modules are apparent at runtime • runtime activation lifecycle: installed → resolved → starting → active → stopping → uninstalled • Jigsaw resolves module dependencies during build, installation, or runtime • gives opportunity for load time optimizations such as pre-verify, indexing, etc. • Equinox OSGi implementation has similar optimizations.

Project Penrose – the road to reality • Both Jigsaw and OSGi are here to stay! • Try to get the best of both worlds • Bring together experience in modularity to Java SE • It need not be a zero-sum game • OSGi is established and will continue to be well used • Jigsaw is underway and a key component to the Java SE plans • Project Penrose's goal is to demonstrate interoperability between Jigsaw and OSGi OSGi support — It must be demonstrated by prototype to be feasible to modify an OSGi micro-kernel such that OSGi bundles running in that kernel can depend upon Java modules. The kernel must be able to load Java modules directly and resolve them using its own resolver, except for core system modules. Core system modules can only be loaded using the module system’s reification API. Java Module-System Requirements

Project Penrose - structure • Penrose is structured to allow collaboration between OSGi and Jigsaw developers • OSGi (Equinox) and Jigsaw are governed by different foundations, companies and licenses • Need to allow separation of developers • Two Penrose code repositories • Main code repository is downstream from Jigsaw • Provides opportunity to be experimental without disrupting Jigsaw progress • Kept in sync with Jigsaw changes, easy to pass code upstream • “show me” working code, rather than arguments on Jigsaw lists • Second code repository specifically for Penrose tools, demos, etc. • These are licensed under GPLv2 + classpath exception • Code developed in Penrose will be delivered through Jigsaw

Project Penrose – technical roadmap • Goal #0 : toleration • ensure that OSGi frameworks continue to run unmodified on a Jigsaw enabled runtime • creating modules / bundles that have both Jigsaw & OSGi metadata on the same JAR • Goal #1 : interoperability of module info metadata • ensure Jigsaw metadata can be enhanced with OSGi concepts • teach OSGi to read Jigsaw module info • mapping Jigsaw metadata into OSGi format for the framework to understand, e.g. requires ⇒ Require-Bundle: • resolve Jigsaw modules using the OSGi resolver • Goal #2 : OSGi implementation exploit of Jigsaw modularity • enhance OSGi to use Jigsaw publication repositories, API, etc • Goal #3+ : Full interop • a blend of OSGi and Jigsaw cross delegation on module phases

Achievements in Level 0: Toleration • Achievement #1: pass the OSGi tests on a Jigsaw enabled runtime • Equinox 3.7.2 – the OSGi reference implementation • OSGi 4.3 Compliance Tests – with minor patch for test case error • Running on Windows XP SP3 & Ubuntu 11.04 • Achievement #2: run a Java application as either OSGi or Jigsaw modules • Took Java 2D demo • Broke it into multiple functional units, and run the demo as OSGi bundles OR Jigsaw modules

Observations : Jigsaw Module Metadata • Jigsaw's module-info.java files compile into module-info.class files • resulting .jar module contains binary metadata • Number of issues with these files • cannot (or very hard) to extend • other module systems or tools may need to store additional module-related data • binary files are not easy to read • you need to use a tool • or create your own .class file reader • may make troubleshooting harder too • Goal to create an readable, easily parsed metadata format • extensible • expressive enough for Jigsaw and OSGi

Achievements in Level 1: Module metadata • Penrose proposes using module-info.json • Plain Text module definitions • In JSON • JSON offers • Better extensibility, additions can be made inline • Easier to read from modular JAR filesfor humansfor tools • For performance, implementations can still cache their own binary representations

JSON module metadata module-info.java (Jigsaw) module org.astro @ 1.2 { exports org.astro; } In the Jar file: binary module-info.class module-info.json (Penrose) "module" : { "name" : "org.astro", "version" : "1.2", "exports" : [{ "name" : "org.astro", "org.osgi.version" : "1.2.3" }]} Note the extension on the exports In the Jar file: plain text module-info.json

To be done: Level 2/3 integration • This pilot needs to start, some areas to look at: • Loading JavaSE modules in OSGi as bundles • OSGi will most likely be extended to understand JavaSE Module Metadata. • Define extensions to JavaSE module metadata to cover OSGi information • Package Versions, Package Imports, etc... • Wiring to existing JavaSE modules in OSGi package imports

Impact to your applications • The serial classpath is being replaced by a graph of module dependencies • Problems (and tricks) you have with classpath ordering will be gone • e.g. detecting the null classloader, occluding types, etc. • java -cp zoo.jar,bar.jar,foo.jar MyApp Class path Bootstrap class loader foo.jar Extension class loader bar.jar delegation System class loader zoo.jar orderedsearch Application class loader

Impact to your applications java -L <location of repository> -m mymodule • Each module has a distinct class loader → no null classloader • Modules hide implementation types that are not explicitly part of the API • Some APIs are being removed or redefined in Java 8 to enable modularity Classlib native APIs Bootstrapclass loader Base module loader Module A class loader Module B class loader Module C class loader Module D class loader

Running Java 9 in legacy mode • Java 9 will be a “revolutionary” release • Support for running in “legacy” mode • Ideally : No Impact • Existing code can continue to be run in class path mode • No changes to the command line required • Don't get any benefits of modularity start-up, customization, etc • Any JRE modularity change which breaks applications in this mode will be treated as bug • Some changes are inevitable as the JRE is restructured to allow for modularity • APIs being deprecated, internal methods being (re)moved • Risk areas: class loader and service loader related code

Taking advantage of modularity • In an ideal world... • Divide your application into multiple modules, or even treat it as a single module. 1) Define module info metadata for each functional area • Requires appropriate modules (jdk.base for core classes, jdk.desktop for awt/swing, jdk.jaxp for xml, etc) • Exports public classes • Permits other modules for isolation 2) Modify legacy code to account for API changes • Explicit class loader type casting may need to be modified • Class.getResource()/ClassLoader.getResource() invocation may need to be modified • ServiceLoader invocation may need to be modified • Many classes only for JDK internal use are not accessible anymore 3) Deploy modules • Include resources and native libs • Install modules into module library

Final words of advice • Stop using internal APIs and implementation classes • You will be broken in a modular world • Be modest in your API usage and we can likely help with a tailored JRE • If you need a modularity story that works before Java 9 (expected summer 2015) then the answer is OSGi • Penrose will ensure OSGi continues to work beyond Java 9 too • OSGi has greater expressiveness • Experience of modularity has resulted in a rich set of behaviors • While you may not need it now, do you want to change system when you do need it? • We would like to see the JSR created right now so that a Java Community Expert Group can get to work on designing the Java Module System. • Come and participate in OpenJDK Jigsaw or Penrose projects!

Links Jigsaw website http://openjdk.java.net/projects/jigsaw/ OSGi Alliance http://www.osgi.org Penrose project http://openjdk.java.net/projects/penrose/

OpenJDK Penrose Project