the isthmus in the VM

the isthmus in the VM

John Rose

Tuesday, March 18, 2014

https://blogs.oracle.com/jrose/the-isthmus-in-the-vm

This is a good time to consider new options for a “native interconnect” between code managed by the JVM and APIs for libraries not managed by the JVM.

Notably, Charles Nutter has followed up on his JVM Language Summit talk (video on this page) by proposing JEP 191, to provide a new foreign function interface for Java. To access native data formats (and/or native-like ones inside the JVM), there are several projects under way including David Chase’s data layout package, Marcel Mitran’s packed object proposal, and Gil Tene’s object layout project.

This article describes some of the many questions related to native interconnect, along with some approaches for solving them. We will start Project Panama in OpenJDK to air out these questions thoroughly, and do some serious engineering to address them, for the JDK.

Let us use the term native interconnect for connections between the JVM and “native” libraries and their APIs. By “native” libraries I simply mean those routinely used by programmers of statically compiled languages outside the JVM.

the big goal

I think the general, basic, idealistic goal is something like this:

If non-Java programmers find some library useful and easy to access, it should be similarly accessible to Java programmers.

That ideal is easy to state but hard to carry out. The fundamental reason is simple—the languages are different. C++ programmers use the #include statement for pulling in APIs, but it would be deeply misguided to try to add #includes to the Java language. For more details on how language differences affect interconnection, see the discussion below. Happily, this is not completely new ground, since managed languages (including Lisp, Smalltalk, Haskell, Python, Lua, and more) have a rich history of support for native interconnect.

Most subtly, even if all the superficial differences could be adjusted, the rules for safe and secure usage of Java differ from those of the native, statically-compiled languages. There is a range of choices for ensuring that a native library gets safely used. The main two requirements are to make VM-damaging errors very rare, and (as a corollary) to make intentional attacks very difficult. We will get into more details below.

Besides safety, Java has a distinctive constellation of “cultural” values and practices, notably the features which provide safety and error management. So, the access to C APIs must be be adapted to the client language (Java) by means of numerous delicate compromises and engineering choices to preserve not only the “look and feel” of Java expressions but also their deeper cultural norms. By using the metaphor of culture, I don’t imagine a “Java way of life”, but I observe that there are “Java ways” of coding, which differ interestingly from other ways of coding. Cultural awareness becomes salient when cultures meet and mix.

Anyway, to get this done, we need to build a number of different artifacts, including Java libraries, JVM support mechanisms, tools, and format specifications. A number of possibilities are enumerated below.

why this is difficult

First, let’s survey some of the main challenges to full native interconnect.

  1. syntax: Since the languages differ, Java user code for a native API will differ in syntax from the corresponding native user code, sometimes surprisingly. For example, Java 8 lambdas are very different in detail from C function pointers, although they sometimes have corresponding uses. Java has no general notions corresponding to C macros or C++ templates.

  2. naming: Different languages have different rules for identifier formation, API scoping (packages vs. namespaces), and API element naming. Languages even have differing kinds of names: Java has distinct name spaces for fields and methods, while C++ has just members.

  3. data types: Basic data types differ. Booleans, characters, strings, arrays differ between the languages C++ uses pointers, sometimes for information hiding, sometimes for structurally transparent data. Java uses managed references, which always have some hidden structure (the object header). And so on. A user-friendly Java interconnect to a native API needs to adjust the types of API arguments and return values to reduce surprises.

  4. storage management: Many native libraries operate through pointers to memory, and they provide rules for managing that memory’s lifetime. Java and native languages have very distinct tactics for this. Java uses garbage collection and C++ libraries usually require manual storage management. Even if C++ were to add garbage collection, the details would probably be difficult to reconcile. A safe Java interconnect to a native API needs to manage native storage in a way that cannot crash the JVM.

  5. exceptions: As with storage management, languages differ in how they handle error conditions. C++ and Java both have exceptions, but they are used (and behave) in very different ways. For example, C++ does not mandate null pointer exceptions. C APIs sometimes require ad hoc polling for errors. A user-friendly Java interconnect to a native API needs a clear story for producing exceptions, which is somehow derived from the native library’s notion of error reporting.

  6. other semantics: Java’s strings are persistent (used to be called “immutable”) while C’s strings are directly addressable character arrays which can sometimes change. (And C++ strings are yet another thing.)

  7. performance: Code which uses Java primitives performs on a par corresponding C code, but if an API exchanges information using other types, including strings, boxing or copying can cause performance “potholes”. I expect that value types will narrow the gap eventually for other C types, but they are not here yet.

  8. safety: I’m putting this last, but it is the most difficult and important thing to get right. It deserves its own list of issues, but the gist of it is the JVM as a whole must continue to operate correctly even in the face of errors or abuse of any single API. The next section examines this requirement in detail.

safety first

The JVM as a whole must continue to operate correctly when native APIs are in use by various kinds of users.

  1. no attacks from untrusted code: Untrusted code must not be allowed to subvert the correct operation of the JVM, even if it makes very unusual requests of native APIs available to it. This implies that many native APIs must be made inaccessible to untrusted code.

  2. no privilege escalation from untrusted code: Untrusted users should not be able to access files, resources, or Java APIs via native APIs, if they would not already have access to them via Java code.

  3. no crashes: It must be difficult for ordinary user code, and impossible for untrusted code, to crash the JVM using using a native API. Native API calls which might lead to unpredictable behavior must be detected and prevented in Java code, preferably by throwing exceptions. Pointers to native memory must be checked for null before all use, and discarded (e.g., set to null) when freed.

  4. no leaks: It must be difficult or impossible for ordinary user code to use a native API to use memory or other system resources in a way that they cannot be recovered when the user code exits. Native resources must be used in a manner that is scoped

  5. no hangs: It must be difficult or impossible for ordinary user to cause deadlocks or long pauses in system execution. Pauses for JVM housekeeping, like garbage collection, must not be noticeably lengthened because of waits for threads running native code.

  6. rare outages: Even if code is partially or fully trusted, errors that might lead to crashes, leaks, or hangs must be detected before they cause the outage, almost always.

  7. no unguarded casts: If privileged Java code must use cast-like operators to adjust its view of native data or functions, the casting must be done only after some kind of check has proven that the cast will be valid. This implies that native data and functions must be accessed through Java APIs that fully describe the native APIs and can mechanically check their use.

From these observations, it is evident that there are at least three trust levels that are relevant to native interconnect: untrusted, normal, and privileged.

Java enforces configurable security policies on untrusted code, using APIs like the security manager. This ensures that untrusted code cannot break the system (or elevate privileges) even if APIs are abused.

Normal code is the sort of code which can run in a JVM without a security manager set. Such code might be able to damage the JVM, using APIs like sun.misc.Unsafe, but will not do so by accident. As a practical way to reduce risk, we can search normal code for risky operations, which should be isolated, and review their use for safety.

I think many of the tricky details of native interconnect are related to this concept of privileged code. Any system like the JVM that enforces safety invariants or access restrictions has trusted, privileged code that performs unsafe or all-access operations, such as file system access, on behalf of other kinds of code.

Put another way, privileged code is expected to be in the risky business. It is engineered with great care to conform to safety and security policies. It supports requests from non-privileged code—even untrusted code—after access checks on behalf of the requester. Privileged code needs maximum access to native APIs of the underlying system, and must use them in a way that does not propagate that access to other requesters.

engineering privileged wrapper code

In the present discussion, we can identify at least two levels of binding from Java code to native APIs: a privileged “raw access” to most or all API features, and a wrapped access that provides safety guarantees that match the cultural expectation of Java programmers.

So let’s examine the process of engineering the wrapper code that stands between normal Java users and native APIs.

In current implementations of the JDK, native APIs are wrapped in hand-written JNI wrapper code, written in C. In particular, all C function calls are initiated from JNI wrappers.

(There is plenty of other privileged code written both in Java and C++. Much Java code in packages under java.lang and sun is privileged in some way. Most of it is not relevant to the present subject.)

Ideally, wrapper code should be constructed or checked mechanically when possible. In the present system, the javah tool assists, slightly, in bridging between Java APIs and JNI code. JNI wrapper code is checked by the native C compiler. And that is about all. Surely Java-centered tools could do more.

On the other hand, as we saw above, bringing the languages together is hard. No tool can erase the cultural differences between Java and native languages. There will always be ad hoc adjustment to reduce or remove hazards from native APIs. Such adjustments will usually be engineered by hand in privileged code, as they are today in JNI wrapper code.

We must ask ourselves, why bother to build new mechanisms for native interconnect when JNI wrappers already do the job? If manual coding will always be required, perhaps it is better to do the coding in the native language, where (obviously) the native APIs are most handy. In that case, there would be no need for Java code ever to perform unsafe operations. Isn’t this desirable?

I think the general answer is that we can improve on the trade-offs provided by the present set of tools and procedures. Specifically, by using more Java-centered tools and procedures, we can improve performance. Independently of performance, we can also decrease the engineering costs of safety.

better performance without compromising safety

Safety will always trade against performance, but—as Java has proven over its lifetime—it is possible with care to formulate and optimize safety checks that do not interfere unacceptably with performance.

Classic JNI performance is relatively poor, and some of the reasons are inherent in its design. JNI wrappers are created and maintained by hand, which means that the JVM cannot “see into” them for optimizing them.

If the JNI wrappers were recoded in Java (or some other transparent representation) then the JVM could much better optimize the enforcement of safety checks. For example, a program containing many JNI calls could be reorganized as one which grouped the required safety checks (and other housekeeping) into a smaller number of common blocks of code. These blocks could then be optimized, amortizing the cost of safety checks across many JNI calls.

Analogous optimizations of lock coarsening or boxing elimination are possible because all the operations are fully transparent to the JVM. By comparison, there is much unnecessary overhead around native calls today.

This sort of optimization is routine when the thing being called can be broken down into analyzable parts by the JIT compiler. But C-coded JNI wrappers are totally opaque to it. The same is currently true of the wrappers created by JNR, but they are regular enough in structure that the JIT can begin to optimize them.

In my opinion, a good goal is to continue opening up the representation of native API calls until the optimized JIT code for a native API call is, well, optimal. That is, it can and should consist of a direct call to the native API, surrounded by a modest amount of housekeeping, and all inlined and optimized with the client Java code.

Making this happen in the compiler will require certain design adjustments. Specifically, the metadata for the native API must be provided in a form suitable for both the JVM interpreter and compiler. More precisely, it must support both execution by the JVM interpreter and/or first-level JIT, and also optimizing compilation by the full JIT. This implies that the native API metadata must contain some of the same kind of information about function and data shape that a C compiler uses to compile calls within C code.

lower engineering costs for safety

I also think that coding more wrapper logic in Java instead of C will provide more correctness at a lower engineering cost. Although wrapper code in C has the advantage of direct access to native APIs, the code itself is difficult to write and to review for correctness. C programmers can create errors such as unsafe casts in a few benign-looking keystrokes. C-oriented tools can flag potential errors, but they are not designed to enforce Java safety norms.

If direct access to C APIs were available to Java code, all other aspects of wrapper engineering would be simpler and easier to verify as correct. Java code is safer and more verifiable than C code. If written by hand, it is often more compact and simple than corresponding C code. Routine aspects of wrapper engineering could be specified declaratively, using specialized tools to generate Java code or bytecode automatically. Whether Java wrapper code is created manually or automatically, it is subject to layers of safety checking (verifying and dynamic linking) that C code does not enjoy. And Java code (both source files and class files) can be easily inspected by tools such FindBugs.

The strength of such an automated approach can be seen in the work noted by JEP 191, the excellent JNR project. For a quick look at a “hello world” type example from JNR, see Getpid.java. Although the emphasis on JNR is on function calling, integrated native interconnect to functions, data, and types is also possible.

Side note: My personal favorite example of automated language integration is an old project that integrated C++ and Scheme on Solaris. The native interconnect was strong enough in that system to allow full interactive exploration of C++ APIs using the Scheme interpreter. That was fun.

One way we can improve on the safe use of these prior technologies is to provide more mechanical infrastructure for reasoning about the safety of Java application components. It should be possible to create wrapper libraries that internally use unsafe native APIs but reliably block their users from accessing those APIs. To me this feels like a module system design problem. In any case, it must be possible to correctly label, track, review, and control both unsafe code and the wrapper code that secures it.

wrapper tactics

A likely advantage of Java-based wrappers is easier access to good engineering tactics for wrapping native APIs. Here are a few examples of such tactics:

Most of these tactics can be made automatic or semi-automatic within a code generation tool, and apply routinely unless manually disabled. This will further reduce the need for tricky hand-maintained code.

Interfaces are particularly useful for expressing groups of methods, since they express (mostly) pure behavior rather than Java object implementation. Also, interfaces are easy to compose and adapt, allowing flexible application of many of the above tactics.

As used to represent an extracted native API, an interface would be unique to that API. Uses of such interfaces would tend to be in one-to-one correspondence with their implementations. In that case JVMs are routinely able to remove the overhead of method selection and invocation by inlining the only relevant implementation.

questions to answer, artifacts to build

A native interconnect story will supply answers to a number of related questions:

Answering these questions affirmatively will require us to build some interesting technology, including discrete and separable projects to enable these functions:

Project Panama in OpenJDK will provide a venue for exploring these projects. Some of them will be closely aligned with OpenJDK JEPs, notably JEP 191, allowing the Project to incubate early work on them.

Other inspiration and/or implementation starting points include:

A native header file import tool scans C or C++ header files and provides raw native bindings for privileged Java code. Such tools exist already for other languages, and can get colorful names like SWIG or Groveller.

For the present purposes, I suggest a simpler name like jextract. A high-quality implementation for Java could start with an off-the-shelf front end like libclang. It would apply Java-oriented rules (with hand-tunable defaults) and produce some form of metadata, such as loadable class files.

A toolchain that embodies many of these ideas could look something like this:

 /-----------|    /-----------|
|  stdio.h   |   | stdio.java |
|------------|   |------------|
      |               |
      v               |
|------------|        |
|  jextract  |  <-----/
|------------|
      |
      v
 /-----------|
| stdio.jar  |     /------------|
|------------|     | userapp.jar|
      |            |------------|
      v                  |
|------------|           |
|    jvm     |  <--------/      /---------|
|            |  <--------------| libc.dll |
|------------|                 |----------|

The stdio.java file would contain hand-written adjustments to the raw API from the header file. The stdio.jar file would contain automatically gathered metadata from the header file, plus the results of compiling stdio.java. The contents of stdio.java could be straight Java code for the user-level API, but could also be annotations to be expanded by a code generation step in the extraction process.

The code in userapp.jar would access the features it needs from stdio.jar. The implementations of these interfaces would avoid C code as much as possible, so that the JVM’s JIT can optimize them suitably.

Side note: The familiar header file I am picking on is actually unlikely to need this full treatment. In a more typical case, a whole suite of header files would be extracted and wrapped.

For bootstrapping or pure interpretation, a minimum set of trusted primitives are required in the JVM to perform data access and function call. these would be coded in C and also known to the JIT as intrinsics. They can be made general enough to implement once in the JVM, rather than loaded (as JNI wrappers are loaded today) separately for each native API. For example, JNR uses a set of less than 100 specially designed JNI methods to perform all native calls; these methods are collectively called jffi.

Building such toolchains will allow cheaper, faster commerce between Java applications and native APIs, much as the famous Panama Canal cuts through the rocky isthmus that separates the Atlantic and Pacific Oceans.

Let’s keep digging.

Appendix: preserving Java culture

Let’s go back to the metaphor of culture as it applies to the world of Java programming.

Here is a list of benefits about Java that programmers rely on, which any design for native interconnect must preserve. As a group, these features support a set of basic programming practices and styles which allow programmers great freedom to create good code. They can be viewed as the basis of a programming “culture”, peculiar to Java, which fosters safe, useful, performant, maintainable code.

Side note: This list contains many truisms and will be unsurprising to Java users. Remember that culture is often overlooked until two cultures meet. I am writing this list in hopes it will prove useful as a checklist to help analyze design problems with native interconnect, and to evaluate solutions. Also, I am claiming that the sum total of these items underlies a unique programming culture or ecosystem to Java, but not that they are individually unique to Java.

All of these benefits are familiar to Java programmers, perhaps even taken for granted. The corresponding benefits for a native language like C++ are often more complex, and require more work and care from the native programmer to achieve.

A good native interconnect story will provide ways to reliably dispose of this work and care before it gets to the end user coding Java to a native API.

This requires native APIs to be acculturated to Java by the artful creation of wrapper code, as noted above.