Interoperability between native code and Java is a complex problem, riddled with many decision points and compromises: starting from the most obvious goal of achieving interoperability with the C programming language, which Java types should be used to encode native C types? Which API does a best job at modeling the set of constructs available in the C programming language? And, beyond the C programming language, how do we make sure that such an API will be reusable when defining interoperability layers targeting other programming languages (C++, COBOL, Pascal, Fortran, ...) - and what about non-language interoperability use cases? Can we e.g. use the same API to model a C native struct and, say, a Protobuf message protocol schema? Speaking of performances, how do we make sure that the new interoperability layer is as fast (or better) than the current status quo (JNI)? And how do we design an API that can be evolved at a later stage to take advantage of significant upcoming JVM optimizations such as value types and generic type specialization? And what about ease of use? If we optimize for automatically generated bundles (e.g. those obtained with tools like jextract), how do we keep the door open for more casual and ad-hoc interoperability use cases? As it should be clear by now, designing an API that satisfies all such constraints is no small feat (if at all possible); in this document we will try to tease apart various aspects of native interoperability, and put forward some concrete proposals as to how such an API might look like.
Note: throughout this document we shall use the term API loosely, to denote not only a set of classes/methods available to programmers, but also the metadata that can be attached to declarations and/or classfiles to achieve native interoperability. While such metadata might be hidden behind adequate tooling support, it is nevertheless there, and it is crucial that we define it precisely, so that 3rd party tools can be written against it.
Throughout this document we are going to assume that native interoperability is achieved according to the reference model depicted in the following diagram.
In this reference model we have native applications - that is, Java applications which need to access native functionalities (e.g. allocate chunks of native memory, call native functions, etc.). Such functionalities are achieved by interacting with interfaces defined in a native bundle; such bundles are generated statically (more on this below). The interfaces in a native bundle (we shall refer to them as native interfaces from now on) are typically associated with additional metadata which can be used to describe e.g. the set of native libraries a class might depend on, or the native layout associated with a native data structure. This extra information is what, crucially, allows the Panama runtime - more specifically, the binder - to mechanically synthesize a native bundle implementation on the fly, that is a set of concrete Java classes that acts as an implementation for the native interfaces defined in the native bundle - as well as to ask the underlying JVM to load any native libraries that the native bundle might depend on.
Therefore, native bundles are the magic glue that ties the Java world and the native world together. Such bundles, as shown in the diagram, can be autogenerated with the aid of offline tools (such as jextract) or they can be written manually as a regular Java source (e.g. an interface with some annotations). There are two main kinds of native interfaces that can be contained in a native bundle, which will be discussed in further details in the following sections; each kind of native interface has, as we shall see, different requirements in terms of the metadata it needs to expose to the binder.
Note: it is important to note that, since the native bundle implementation are generated on the fly, the interfaces in the native bundles are guaranteed to have a single implementation class; this allows the JVM to carry out optimizations, and to effectively de-virtualize calls to such interface methods. This implicit assumption is at the core of the reference model shown above.
One kind of interface that can be found in a native bundle is a so called native type interface. A native type interface is, at its core, an object-oriented view over a native type aggregate. For instance, consider the following native type modeling time/date (this type is defined in the C header file time.h):
struct tm {
int tm_sec; // Seconds after the minute [0, 59]
int tm_min; // Minutes after the hour [0, 59]
int tm_hour; // Hours since midnight [0, 23]
int tm_mday; // Day of the month [1, 31]
int tm_mon; // Months since January [0, 11]
int tm_year; // Years since 1900
int tm_wday; // Days since Sunday [0, 6]
int tm_yday; // Days since January 1 [0, 365]
int tm_isdst; // Daylight Saving Time flag
long tm_gmtoff; // Seconds east of UTC
char * tm_zone; // Timezone abbreviation
};This native type can be modeled using a Java interface containing an accessor method for each struct field:
interface Tm {
int tm_sec();
int tm_min();
int tm_hour();
int tm_mday();
int tm_mon();
int tm_year();
int tm_wday();
int tm_yday();
boolean tm_isdst();
long tm_gmtoff();
??? tm_zone();
}Of course the above code is meant to be a simplification (we shall return to this example later in this document). First, the native type interface doesn't take into account the fact that a client might also want to set the value of the fields of the underlying tm struct. Secondly, this interface does not carry enough information so that the binder can mechanically derive an implementation from it. The binder would need information about the offset at which the underlying struct fields can be found in memory, and also some information on how big these fields are - e.g. should the generated implementation read 16 or 32 bits? These are questions that the metadata associated with a native type interface must answer unambiguously. Note also that we leave out the Java type associated with the last native type accessor; that's because the type of the underlying struct field associated with that accessor is a pointer type - we will discuss how to model pointer in the binder API in great details later in this document. In other words, when going from a native type (e.g. a C struct) to a Java interface modeling that native type, some type adaptation is required, so that suitable Java carriers type is chosen to accurately model the underlying native types. Note also that, while in this document we will use native type interfaces to model C structs, such interfaces can be used to model any kind of foreign data (e.g. network packets, message protocols, etc.).
The second kind of interface that can be found in a native bundle is a so called native library interface. A native library interface defines one or more methods modeling native functions. For instance, consider the following native functions to carry out trigonometric computations (these functions are defined in the C header file math.h):
double sin (double x);
double cos (double x);
double tan (double x);
double asin (double x);
double acos (double x);
double atan (double x);These native functions can be modeled using abstract methods in a Java interface, as follows:
interface Trigo {
double sin(double x);
double cos(double x);
double tan(double x);
double asin(double x);
double acos(double x);
double atan(double x);
}Again, the interface alone does not carry enough information to allow the binder to mechanically generate its implementation. To do so, the binder would need at least some metadata describing where the library defining such functions can be found; additionally, the binder would need some information related to the calling convention to be used when generating the adapter for the native function: for instance, should arguments be passed on the stack, or on machine registers? And, if the arguments are passed on the stack, how much space should be allocated for them? Again, these are crucial questions that the metadata associated with a native library interface must answer unambiguously. Also, as in the case of native type interfaces, some type adaptation has also occurred here - and the native types occurring in the native function declaration have all been mapped to suitable carrier types in the Java programming language. Incidentally, the Java carrier types used in the above native library interface declaration are spelled in the same way as the native types in the native function declarations.
As we have seen, modeling native types and libraries through Java interfaces is a powerful trick which allows programmers to easily access to native features - as such features are simply modeled as Java interface methods. But we have seen how native interfaces alone do not contain enough information for the binding process described above to take place. Instead, some additional metadata is required to help the binder to mechanically derive an implementation for such interfaces. The most obvious kind of missing metadata is some information that would allow the binder to answer to layout-related questions, such as: how big is a given struct field? At which offset from the start of the struct can such field be found? If the struct is passed as argument to a native function, how much space in the callee stack should be allocated? In the next section we shall explore how such layout information can be embedded in native interfaces.
We have also seen how the process of deriving an interface from a native type or library always involves some kind of type adaptation: for each native type occurring in the declaration to be modeled (be it a type or a function), a suitable Java carrier type must be chosen, so that values of that native type can be adequately modeled. As we shall see, this is not an easy task - especially when it comes to representing pointers and arrays, and we will shall discuss this topic in much greater detail in a subsequent section.
As we have seen, artifacts in a native bundle require some form of metadata to indicate properties associates with the layout of the elements being modeled. Consider an interface method modeling a struct field: what is the offset of that field in the struct? What is its size? And its alignment? Does loading/storing the field value require sign extension? Should bits corresponding to the field value be stored in a big- or little-endian order? These some of the properties that a layout metadata needs to capture. In the following section we define a way to precisely describe bit layouts. Before we jump to the core of the issue - a quick disclaimer: this is not a new topic, and many of the conclusions reached in this document can be thought of as refinements of previous explorations - the work on the LDL proposal - a joint effort with IBM has proven particularly fruitful.
When designing a layout description there is a subtle tension: on the one hand, the more mathematically pure the definition is, the wider is the applicability of that description. On the other hand, a description that is too pure might also be problematic, as it might be too abstract, and miss distinctions that ends up being important in terms of how the layout is going to be used in practice. Let's consider an example: floating pointed-ness. While it might be tempting to consider floating point-ness as an implementation detail that should not affect a layout description - in reality many system ABIs (SystemV, Windows) define special rules for handling integral vs. floating point data by calling conventions - typically this is done by assigning values of each family to different machine registers (e.g. floating point values are stored in XMM registers, whereas integral values are stored in RCX registers). This seems to suggest that floating point-ness is, after all, an important property to capture in a layout description, even though, from a theoretical perspective, purists might be inclined to think that e.g. 32 bits are just 32 bits, regardless of whether they model a floating point value or not.
So, while in the following sections we will try to come up with a layout description that relatively abstract, few pragmatic concessions will be made in order to capture aspects that are likely to result in crucial differences when synthesizing native bundle implementations.
In the following sections we will explore the various elements that form the backbone of the proposed layout description. Such elements are values, addresses, groups, sequences and holes. To describe layout elements we will use an hypothetical Java-like API, while we will defer to a later section how such an API can be reified into a DSL that can be easily embedded into Java annotations attached to native interfaces members. For instance, we can model a generic layout element with the following pseudo interface:
interface Layout {
long bitsSize();
Map<String, String> annotations();
}Every layout element should support at least two common operations: (i) obtain the size in bits of such element; and (ii) obtain the set of annotations attached to the layout element. Annotations are simple (name, value) pairs, which can be used to convey additional information which typically is outside the domain of the layout description. For example, an annotation might associate a layout with a symbolic name that can be used to refer to that layout. This trick will come in handy when discussing layout holes.
If we think of layouts as trees, then values are the leaves in such trees. Value layouts can be used to describe the most primitive forms of native data: numerics. Numerics come in different forms and shape, but in our description we think it is essential to capture at least the following properties: (i) the size of the value, (ii) its signed-ness, (iii) whether it models a floating point value (as we shall see these last two properties are not completely orthogonal) and (iv) its endianness.
interface Value extends Layout {
enum Kind {
INTEGRAL_SIGNED,
INTEGRAL_UNSIGNED,
FLOATING_POINT;
}
enum Endianness {
LITTLE_ENDIAN,
BIG_ENDIAN
}
@Override
long bitsSize();
Kind kind();
Endianness endianness();
}These distinctions are important, as they affects how a value is treated (e.g. as we have seen, a calling convention might chose different machine registers to pass different kinds of values to a native function). The size of a value layout should always be a multiple of 8, except when a value layout occurs in the context of a container definition (see below).
An address layout denotes a sequence of bits which does not hold a real domain value - instead it holds a pointer to some memory region with a given layout. As such, an address layout, in addition to the properties in common with all value layouts, it should also feature an (optional) associated pointee layout:
interface Address extends Value {
Optional<Layout> layout();
}Note that it might be tempting to assume that addresses always have an implicit size, sign and endianness as required by the host platform; in reality, it is quite common to model different kind of pointers in different ways, depending on whether they are near or far - which typically results in layouts with different sizes (e.g. an example of this are Hotspot compressed oops and near/far pointers in the Cap'n'Proto message protocol schema).
It is also important to note that an associated layout might or might not be there; for instance, if an address layout modeling a C void pointer will have no associated pointee layout information.
A group layout combines together one or more layout elements; there are two kinds of group: structs and unions. As the names imply, the former can be used to model aggregates, such as C structs, while the latter can be used to model alternatives, such as C unions:
interface Group extends Layout {
public enum Kind {
STRUCT,
UNION;
}
Kind kind();
List<Layout> elements();
@Override
default long bitsSize() {
var sizes = elements().stream().mapToLong(Layout::bitsSize);
return kind() == Kind.STRUCT ?
sizes.sum() :
sizes.max().getAsLong();
}
}As it can be seen, the size of a group layout depends on the group kind: in the case of a struct group, the layout size is the sum of the sizes of the constituting layout elements; in the case of an union group, the layout size is the maximum size of any of its constituting elements.
A sequence layout denotes a repetition of a layout element; as such it can be modeled as a special kind of group (struct) layout, consisting of at least two properties: (i) the layout element to be repeated and (ii) the repetition count:
interface Sequence extends Group {
@Override
default Kind kind() {
return Kind.STRUCT;
}
long elementsSize();
Layout element();
@Override
default List<Layout> elements() {
return Collections.nCopies(element(), elementsSize());
}
}It follows that the size of a sequence layout can be derived from the size of its element, multiplied by the repetition count. In the edge case where the repetition count is set to 0, the resulting size will also be 0.
A value layout can sometimes be associated with a group overlay, which define a substructure that is to be associated with said value layout. This comes in handy when modelling e.g. bitfields, where a value layout can be broken down to one or more bit fields; also, it is common for message protocol schemas, such as Cap'n'Proto, to specify some meaningful substructure of pointers (e.g. reserve some bit to distinguish between near and far pointers). To model such features we can simply expose an additional method in the Value layout interface which (optionally) returns an associated group layout, as follows:
interface Value extends Layout {
...
Optional<Group> contents();
}Note that, within the substructure of the overlay group, value fields can have sub-byte sizes (the only place where this can happen).
Sometimes there is a need to reference a layout from another layout using some kind of symbolic description; consider the following mutually recursive struct definition in C:
struct A {
struct B *b;
};
struct B {
struct A *a;
};How should we model the layout of the struct A ? It is a product group, which contains a single address layout element (the layout for A::b); the pointee layout associated with that address layout is the layout for the B struct. Unfortunately, if we try to use the layout description for B as a pointee layout, we run into a problem, as the layout for B refers back to A (via the layout for B::a). In other words, there's no (finite) way to construct a layout information for the struct fields A::b and B::a given the layout elements we have seen so far! One option would be to cut the recursion: pretend that the address layout for e.g. A::b did not have any associated pointee layout. But such a description would be lossy, and could potentially lead to dynamic unsafety.
Layout holes give us a way to address this problem; instead of emitting a precise layout description for the pointee layout associated to A::b, we could instead refer to such layout using an annotated hole. The annotations attached to the hole might contain relevant information as to how the hole should be resolved (e.g. the name of the C struct whose layout should be used to replace the hole):
interface Unresolved extends Layout {
@Override
default long bitsSize() {
return resolve().bitsSize();
}
Layout resolve() throws UnsupportedOperationException;
}As we can see, an hole layout has (i) a size, which is the size of the hole after resolution has occurred, and (ii) a way to dynamically resolve the hole to some concrete layout. This resolution process is not specified (but could be in the future), and takes as input the annotations attached to the hole layout.
In this section we will discuss a possible way to reify the layout description proposal outlined in the previous sections into a little DSL language. The goal is to be able to associate layout descriptions to native interface members via Java annotations - although we must note that this is just one of the many possible ways in which additional metadata can be injected into Java classfiles (e.g. one possible implementation scheme could use ldc and dynamic constants to load API elements directly).
It is also important to note that the syntax described below does not make any claim to be user-friendly; its primary function is the ease of generation (e.g. from tools, like jextract) and consumption (from the binder). As such, it is possible to envision other strategies to achieve user friendliness - e.g. by means of annotation processors and/or compiler plugins that would consume higher-level annotations and generate the low-level annotations described in this document.
As we have seen, at the higher level, a layout is either a value, a group or an hole; so let's start from this production:
layout = (value / group / unresolved)
annotations = +( annotation )
annotation = '(' string [ '=' string ] ')'Annotations could be (name=value) pairs, or, more compactly, just (name), in case the annotation is used to denote the layout name.
Note: to allow layout descriptions to be embedded inside annotations, the layout parser would need to count the level of nested parenthesis found inside an annotation.
Values are defined by the following production:
value = valueTag number [annotations] [ '=' group ] [ addressRest ]
valueTag = 'u' / 'U' / 'i' / 'I' / 'f' / 'F'That is, a value is denoted by a tag, followed by a non-negative number (the scalar size), optionally followed by one or more annotations. The value can then be associated with a group sub-layout (through =) and/or with a pointee layout (for address layouts, via :). There are three main scalar tags: u (for unsigned), i (for signed) and f for floating point; a lower case tag denotes little-endiannes, while an upper case tag (as in U, I and F) denotes big-endianness. For example, the following are valid scalar layouts:
u32 //unsigned, 32bit
f64 //double-precision floating point, 64bit
u32(name=anUnisgnedInt)
u32(anUnisgnedInt)If the description of a value layout is followed by :, then it denotes an address layout, as per the following production:
addressRest = ':' addresseeInfo
addresseeInfo = ( 'v' / layout )As it can be seen, this grammar is not sufficiently expressive to represent function pointers - we shall return to this point in the next section. Examples of address layouts are:
u64(aVoidPointer):v
u32(compressedIntPointer):i32A group layout is always surrounded by square brackets and optionally followed by one or more annotations:
group = '[' (structOrUnionRest / sequenceRest) ']' [annotations]
structOrUnionRest = layout *(layout / unionMember)
sumMember = '|' layoutIf the group layout denotes a struct, the element layouts just occur one after the other; if the group layout denotes an union, the elements are separated by the special token |. Example of struct/union groups are:
[i32i32](intPair)
[u64|u64:f32](aUnion)A sequence layout is denoted by a repetition count followed by an element layout:
sequenceRest = [number] layoutExamples of sequence layouts are:
[ 5u32 ] //unsigned, 32bit, 5 elements
[ 42f32 ] (name=floatArray)A group layout can be associated to a value layout (container) via the = suffix:
u16(aBitField)=[u4(flag)u8(value)u4(padding)]Finally, hole layouts are introduced with the special token $, followed by one or more annotations:
hole = '$' [annotations]Examples of hole layouts are:
[u64:$(B)](A)
[u64:$(A)](B)Functions are not, strictly speaking, layouts - but as we've seen earlier there are at least two cases in which layouts and functions can meet: first, a native function definition should have a layout description for all the arguments/return type in case they are passed on the stack; secondly, a function can act as a pointee for an address layout - this is needed to model C's function pointers.
We could define a function as an aggregate of one or more layouts, as follows:
interface Function {
Optional<Layout> returnLayout();
List<Layout> argumentLayouts();
boolean isVariadic();
}That is, a function has an optional return layout - the optionality here models the fact that functions can be void - and zero or more argument layouts (or, even variable-arity argument layout - more details will be provided in a subsequent section). In terms of our layout description DSL, a function can be represented as follows:
function = '(' *( layout ) [ '*' ] ')' returnLayout
returnLayout = ( 'v' / layout )Where the special character * denotes variable-arity layout. Then, we just need to adjust the production for address layouts, to take into account functions - as follows:
addresseeInfo = ( 'v' / function / layout )Now that we have all the pieces of a layout description in place, let's go back to the examples we have seen before and see how we could leverage Java annotations to embed layout descriptions in native interfaces. Earlier we have shown a native interface that modeled the tm struct from C standard library; to add back layout information, we can decorate that declaration, as shown below:
@NativeStruct("[" +
" i32(get=tm_sec)" +
" i32(get=tm_min)" +
" i32(get=tm_hour)" +
" i32(get=tm_mday)" +
" i32(get=tm_mon)" +
" i32(get=tm_year)" +
" i32(get=tm_wday)" +
" i32(get=tm_yday)" +
" i32(get=isdst)" +
" i64(get=tm_gmtoff)" +
" u64:u8(get=tm_zone)" +
"](Tm)")
interface Tm (
int tm_sec();
int tm_min();
int tm_hour();
int tm_mday();
int tm_mon();
int tm_year();
int tm_wday();
int tm_yday();
boolean tm_isdst();
long tm_gmtoff();
??? tm_zone();
)The native type interface is now annotated with the @NativeStruct annotation, which can be used to attach a layout description to the interface. The layout description uses several annotations of the kind get=sec to bind layout elements with interface members (similar annotations can be used for e.g. setters, or to retrieve the address of a given struct field). That is, there's an unambiguous mapping between the interface accessors and the layout element in the description - meaning that the binder will be able, for each method implementation, to determine field offsets, alignments and sizes.
Let's now turn at another example we gave earlier, which modeled trigonometric native functions using a native library interface; again we can use Java annotations to associate function descriptions to the interface members, as follows:
@NativeHeader(
"sin=(f64)f64" +
"cos=(f64)f64" +
"tan=(f64)f64" +
"asin=(f64)f64" +
"acos=(f64)f64" +
"atan=(f64)f64")
interface Trigo {
double sin(double x);
double cos(double x);
double tan(double x);
double asin(double x);
double acos(double x);
double atan(double x);
}Our native interface library is annotated with the @NativeHeader annotation, which contains several function layout descriptions, according to the following grammar:
definitions = *( definitions )
definition = name '=' descriptor
descriptor = (function / layout)Again, there's an unambiguous mapping between the interface members and the layout of the native function arguments/return type in the description; this information will allow the binder to e.g. allocate a big enough buffer on the callee stack to contain the arguments to be passed (in case the arguments are passed through the native stack) or which register should be used to pass the argument to the callee (e.g. floating point values, as in this example, should be passed in different registers than, say, integral values). Note that, in order to allow the annotation to correctly model global variables, the descriptor associated with a give library symbol can also be a layout.
We have seen in previous sections how layout descriptions can help the binder when generating implementation for native bundles; on the one hand, layout descriptions provide a clear, unambiguous way to e.g. locate struct fields, given that they can be used to derive offsets, alignment and size properties; on the other hand layout descriptions can, when associated with arguments and return value of a native function, be used to generate a generic adapter which calls the underlying native function according to a given calling convention.
Layouts are, however, only half of the story when it comes to providing interoperability with native code. Let's consider a native accessor which loads some value from native memory; as we have seen, the layout description associated with said accessor will help the binder locating the region of memory holding the value to be loaded. But, in order to perform the load operation, the binder has to know another piece of information: what Java type should the native value be converted to? We call that type the (Java) carrier for the underlying native value.
Associating Java carriers to native types is not an exact science and, as we shall see in this section, it is an area where many compromises have to be made; there are however some principles we will try to adhere to in the following discussions:
lossless; that is, the Java type should be big enough to carry the set of values in the native type domain
flexibile; that is, it should be possible for developers to access a value with given layout using different carriers, depending on the circumstances
expressive; that is, the Java carrier should preserve some basic expressiveness property of the underlying native type being modeled.
efficient; that is, mapping a native value into a carrier value should (ideally) be an O(1) operation.
Another distinction that is relevant to the problem of mapping native types to carriers is that between small vs. big types. This distinction is especially important when considering efficiency - and is coincidentally reflected by the Value vs. Group split in the layout API. A small type typically denotes values that can fit into a single machine word - meaning that there should be ways to map values of that type to values of its corresponding Java carrier with relatively little effort. Conversely, a big type is typically an aggregate (think of a C struct, or an array); in such cases, it might be much harder to find a corresponding Java carrier so that it satisfy the efficiency property. This might mean some laziness could be involved, to avoid expensive bulk conversions.
In the following sections we will show how native types are mapped to Java carries in the context of the C language; that is, we will show how C types can be modeled in terms of Java types in order to satisfy the principles outlined above. But first, let's start with a quick digression for the mathematically inclined.
From a mathematical perspective, we can think of mapping native types to Java carriers as an embedding/projection pairs; that is, to go from a native type N to a Java carrier C we use an embedding function:
e -> N -> CWhereas to go from a Java carrier back to a native type we use a projection function:
p -> C -> NThe relationship between the embedding/projection function is such that the following properties should be satisfied (where n and c are values of types N and C, respectively):
p(e(n)) = n
e(p(c)) <= cThat is, starting from a native value, and applying an embedding, then a projection leads you back from where you started: the underlying native value. This makes sense. But, when starting from a carrier value, and then applying a projection and then an embedding, we might get back a carrier value that has slightly different properties than the one where we started from. Typically, for small types it is easy to find (quasi) identity embedding/projection pairs; on the other hand, for big types, more complex projection and embedding functions are required, typically exploiting some kind of laziness features to achieve efficiency.
Mapping C primitives into Java types is relatively straightforward; for the most part we can pick a suitable Java primitive type that is big enough (in order to preserve lossless-ness) to carry the underlying native primitive value. A list of possible mappings is defined in the table below (for completeness we also report the layout information, with reference to the x64 platform):
| C type | C type (stdint.h) | Layout (x64) | Java type |
|---|---|---|---|
char |
int8_t |
i8 |
byte |
unsigned char`short |
uint8_tint16_t |
u8i16 |
short |
unsigned shortint |
int16_tint32_t |
u16i32 |
int |
unsigned intlongunsigned long |
int32_tint64_tuint64_t |
u32i64u64 |
long |
long longunsigned long long |
N/A | i128u128 |
BigInteger |
float |
N/A | f32 |
float |
double |
N/A | f64 |
double |
long double |
N/A | f128 |
BigDecimal |
There are some subtleties to note in the table above: first, in order to preserve sign information, sometimes a C primitive type is mapped onto a bigger Java primitive type. For instance, we model an C unsigned int as a Java long. If we simply used a Java int carrier, this would have led to issues, as the range of the two types is slightly different, because of signed-ness: a Java int is always signed, meaning that the maximum value is defined by 2^31 - 1; On the other hand, since a C unsigned int is, well, unsigned, its maximum value can be 2^32 - 1.
Unfortunately, it is not always possible to pick the next primitive up the chain: in the case of C long, the corresponding Java carrier is long. If we wanted to preserve the full value range here, the only option would have been to use BigInteger, which would have led to considerable usability issues.
Another anomaly in our table, which we alluded to earlier, is that there are primitive types which are too big to fit into Java primitives, as long long and long double. Currently, the only viable option to model those is to use reference types such as BigInteger/BigDecimal, but that comes with a considerable expressiveness and usability burden. The hope here is that with the addition of value types we might be able to model such types in a much more straightforward way.
Finally, the mapping above should not be taken too literally; as we have seen - mapping C types to Java carrier should be flexible - it should be possible for clients to view a native char as a Java char if they decide to do so, regardless of the size mismatch. When considering such moves, however, we should pay attention: while loading a native values as a Java value with a bigger range is typically non problematic (after all, the native value can fit into the Java value - with room to spare), storing a Java value back into native memory is more problematic - as that would require some truncation of the underlying Java representation in order to fit the target layout. For instance, if we were to store a Java char in a struct field of type char, we should drop the 8 most significant bits from the Java char value and store the remaining portion into the native memory region associated with the struct field. Here we see the true nature of the project/embedding pair at play: starting from a Java char, then going to a native char and then back to a Java char might not give us the same value we started from.
We envision that layout annotations could be used in order to specify what the binder behavior should be in case of truncation; one option would be to silently ignore truncation, and let the client deal with its consequences. Another option would be to require an exception to be thrown whenever a truncating projection occurs.
Both Java and C have enums; however C enums are a very different from Java enums. In the C programming language, an enum is simply a collection of numeric values (the C compiler picks the size of the underlying numeric value depending on the number and value of the constants being defined). Numerics and enums in C are completely interchangeable - so programmers can pass e.g. an int where an enum is expected. On the other hand, in Java an enum constant is, essentially, a singleton object; while all constants in a Java enum do have some underlying numeric value associated with them (the enum constant's ordinal), Java enums are not numeric values. The range associated with a Java enums is also closed: an enum declaration must list the full set of constants which belong to that enum. This makes a Java enum unsuitable to encode the open-ended ranges implied by C enums.
As such, while it is tempting to model C enums as Java enums, C enums are best modeled as a set of loosely related constants defined somewhere in a native interface. There are two kinds of enums in C: anonymous and named enums, depending on whether the corresponding declaration has an associated name. For named enums, annotations can be used in order to retain some information about the fact that the constant was in fact defined inside an enum declaration; so the following enum declaration:
enum day {sunday, monday, tuesday, wednesday,
thursday, friday, saturday};Could be translated as follows:
@interface day { }
@day int sunday();
@day int monday();
@day int tuesday();
@day int wednesday();
@day int thursday();
@day int friday();
@day int saturday();Where the accessors sunday(), monday(), etc. can be part of the native library interface modeling the header file where the enum declaration was found.
Note: the attentive reader might have noted that we have decided to model enum constants as methods, rather than Java constants; the rationale behind this decision is that to avoid the semantics associated with Java static constants - such as compile-time folding leading to separate compilation mismatches.
typedefThe C language allows programmers to define type aliases for commonly referred types; for instance, one could define a byte type to model 8-bit unsigned numbers as follows:
typedef unsigned char ubyte;Strictly speaking, typedefs do not require additional carriers; we can imagine a process in which a tool generating a native bundle normalizes all type references by getting rid of type aliasing. As such, a function declaration such as the one below:
ubyte read(ubyte nbytes)Can be modeled by the following Java method:
byte read(byte nbytes);In the above declaration there's no mention of the fact that the function parameter and return types were in fact aliased. Using a trick similar to the one shown for enums, we could preserve the type aliasing information:
@interface ubyte { }
@ubyte byte read(@ubyte byte nbytes);This is a generally useful move which improves readability of the native interfaces without negatively impacting on the complexity of the binder implementation.
Pointers are an integral part of the C language, but, as we shall see, are much, much more difficult to model in Java. At its core, a C pointer is just a memory address, so, at least in principle, we should be able to model a pointer as a Java long. While this is a very direct - and efficient - approach, it completely violates the expressiveness principle shown earlier: a Java long is a primitive type - as such it exposes no meaningful operations, and it fails to model accurately the way in which C programmers typically operate on pointers.
So, taking a step back, what are the operations that a C programmer might want to do on a pointer? The following list is a good start:
pointer arithmetic: that is, given a pointer, programmers should be able to obtain another pointer by adding/subtracting to it; note that adding 1 to a pointer doesn't simply mean adding 1 to its underlying address - instead it means moving the pointer by an offset which is equal to the size of the pointer type (e.g. as per sizeof operator).
dereference: that is, given a pointer, programmers should be able to get/set the contents of the memory the pointer points to.
cast: that is, it should be possible to convert a pointer of a given type, to a pointer of a different type (provided the two types are compatible).
From the above properties it should now be clear that a pointer should at least be represented as a (long, layout) pair: that is, a pointer has some memory address (the long value), but also some kind of layout information which allows the pointer to perform arithmetic and dereference operations correctly. A Java interface is emerging here:
interface Pointer {
long addr();
Pointer offset(long elements);
Object get();
void set(Object value);
Pointer cast(Layout that);
}Without taking the above proposal too seriously - it clearly has some expressiveness problems - an interface like the one above would be a good candidate to model a C pointer in Java, as it correctly expresses the operations that can be done on a pointer. It can also be seen how, for the binder implementation, it is straightforward to go back and forth from native pointers to Java instances of the Pointer interface: the embedding consists in creating a new Java instance wrapping a long value (along with some layout info which we assume to be available in some metadata), whereas the projection amounts at calling the addr() method on said interface - both O(1) operations, thus respecting the efficiency of the mapping.
Where do pointers come from? That is, following our object-oriented analogy, if pointers are instances, where do these instances come from? There are two main ways to obtain a pointer: the first is to obtain the address of some native object (this is discussed in greater details in the subsequent section). But there's also another way to obtain a pointer - and that's through dynamic memory allocation (in C often known as malloc). For the purpose of the following discussion, and for the sake of simplicity, we can assume that the Pointer interface has an extra static factory, as follows:
interface Pointer {
...
static Pointer allocate(Layout layout) { ... }
}Note: in reality, allocation routines will be defined in a separate abstraction, called
Scopeto allow the runtime to manage pointer life-cycle, the details of which are beyond the scope of these document.
It is important to keep allocation in mind when thinking about how to model pointers; as we shall see, any attempt to enhance the static type information associated with the pointer API will affect the signature of the allocation factory.
As defined, the Pointer interface is far from being ideal; while it correctly models all main pointer operations, it has few disadvantages: first, its API boxes values though the dereference operations get and set. Since it is likely that the implementations generated by the binder (esp. those corresponding to native interface definitions) will heavily rely on pointer manipulation and dereference, we need to find a better, box-free solution. Secondly, from an expressiveness point of view, this API is not ideal, as it maps different native types such as int* and char* down to the same Pointer carrier; this could result e.g. in readability issues when looking at signatures of interface members modeling native functions.
In a subsequent section, we are going to refine the above basic Pointer interface, showing how to get rid of boxing and also how to enhance the interface declaration to take into account additional static type information.
A Java variable can be read or written to; a C variable, on the other hand, allows an extra operation, which in C is sometimes referred to as &, or the addressof operator - that is, given a variable, obtain the address of the native object denoted by that variable. The way in which this is achieved in C is by resorting to a classification of all expressions into two kinds: lvalues and rvalues - thus, an identifier expression denoting a variable name is an lvalue, whereas e.g. 1 + 2 is an rvalue. The addressof operator is only defined on lvalues. One way to capture this distinction in our API would be to have an abstraction to model lvalues directly:
interface LValue {
Object get();
void set(Object o);
Pointer addressof();
}If we had such an abstraction, we could delegate the dereference operations in the Pointer interface shown above to an lvalue, as follows:
interface Pointer {
long addr();
Pointer offset(long elements);
LValue deref();
Pointer cast(Layout that);
}In addition, we could also avoid generating getters/setter pairs for fields in native type interfaces and simply generate lvalue accessors instead - the lvalue can then be used to get or set the contents of the underlying struct field, as well as to obtain the struct field address.
While this is a totally sensible choice from an API perspective, in the remainder of this document we will assume that C variables will be modeled through accessor tuples of the kind (getter, setter, pointer) - for instance, in the case of native structs, a given struct field would be modeled with three interface methods: a getter, a setter and a pointer accessor. The same approach can be leveraged for modeling other kind of variables - e.g. global variables.
Let's go back to the basic Pointer interface shown before, and try to add back some static information about the pointee type; we could do something like this:
interface Pointer<X> {
long addr();
Pointer<X> offset(long elements);
X get();
void set(X value);
Pointer<?> cast(Layout that);
static Pointer<?> allocate(Layout layout) { ... }
}This looks like a step forward - we can now represent types such as Pointer<Integer>, Pointer<Character> and the likes. But if we look at the generic Pointer interface above, it seems like there something missing: the layout information is not enough for the static type system to drive the instantiation of the Pointer type-variable - as a result, type-dependent operations such as cast() and allocate() are still lossy, as they just return an untyped Pointer<?>.
To overcome this limitation we need some way to model the description of a native type in the static type system; let's then introduce a new abstraction, called LayoutType, which can be thought of the result of coupling a layout description with some Java carrier information:
interface LayoutType<X> {
MethodHandle getter();
MethodHandle setter();
Layout layout();
LayoutType<Pointer<X>> pointer();
//factories
LayoutType<Integer> ofInt = ...
LayoutType<Long> ofLong = ...
}As it can be seen, a LayoutType is composed of a layout description, and a pair of getter/setter method handles - the getter method handle takes an argument of type Pointer and returns a value of type T; while the setter method handle takes two arguments, a Pointer and a value (of type T) and returns nothing. Crucially, LayoutType is itself a generic class, and it exposes combinators that, given a LayoutType for a type T obtain the LayoutType for Pointer<T> (additional combinators will be needed for obtaining array layouts, which will be discussed in a subsequent section). So, to create the layout for a Pointer<Integer> one can do:
LayoutType<Pointer<Integer>> pIntLayout = LayoutType.ofInt.pointer();The LayoutType abstraction significantly improves the Pointer interface, as shown below:
interface Pointer<X> {
long addr();
Pointer<X> offset(long elements);
X get();
void set(X value);
LayoutType<X> type();
<Z> Pointer<Z> cast(LayoutType<Z> that);
static <Z> Pointer<Z> allocate(LayoutType<Z> type) { ... }
}Now all operations, including cast and allocate are sharply typed - this means that the following code would result in a compile-time error:
Pointer<Pointer<Character>> ppc = Pointer.allocate(pIntLayout);This is very useful to keep pointer types straight and to avoid to accidentally treat an int pointer as a char pointer. The attentive reader might also have noticed that, given the getter/setter method handles, it is relatively straightforward to add extra primitive accessors to the Pointer interface, as follows:
interface Pointer<X> {
...
int getAsInt() {
return (int)type().getter().invokeExact(this);
}
void setAsInt(int i) {
type().setter().invokeExact(this, i);
}
...
}In terms of the binder implementation, we can leverage the sharp getter/setter method handles (through the LayoutType instance associated with the pointer - see the type() method); so the getter implementation for a struct field can be written as follows:
class Tm$impl implements Tm {
public int tm_sec() {
Pointer<Integer> tm_sec$ptr = tm_sec$ptr();
MethodHandle getter = tm_sec$ptr.type().getter();
return (int)getter.invokeExact(tm_sec$ptr);
}
Pointer<Integer> tm_sec$ptr() { ... }
...
}Note that, since the underlying getter/setter method handles are both sharp (box-free) and static, the above code should offer good inlining guarantees.
There's however an important fact hiding behind the apparent simplicity of this code snippet: for this to work, the pointer to the struct field (obtained via tm_sec$ptr()) will have to be cast to the correct LayoutType, otherwise the associated getter method handle will have the wrong dynamic type! This means that the binder has to somehow be able to convert the carrier information available in the accessor signature into the correct LayoutType modeling the carrier for that accessor. Something like this:
<Z> LayoutType<Z> ofClass(Class<Z> carrier) { ... }While this approach works well for primitive types (and also for structs, which are likely to be non-generic types), the above routine would not be sufficient to handle fully parameterized pointer types - because of erasure, Class cannot be used to dynamically retrieve information about generic type arguments. That means that, in the general case, our routine would look like:
LayoutType<?> ofClass(java.lang.reflect.Type type) { ... }That is, rather than accepting a Class input, our routine should accept a Type instance, which can then be used to obtain information about generic type arguments. This also means that the bytecode of the generated implementation will need some way to encode Type instances into the constant pool (this can be done via a dynamic ldc, or through constant pool patching, if the implementation class is generated using Unsafe.defineAnonymousClass).
The lesson here is that expressiveness comes at a cost: if the Pointer interface is made generic, the binder should face additional complexity when trying to reify the static type information associated with the accessor (possibly generic) signatures.
There is also a more subtle cost: because of limitations in the generic type-system, the only way to model a pointer to int is through a Pointer<Integer>, which mentions a boxed type. While the results of this conversion are still expressive and readable enough, there could be challenges when attempting to specialize, or anyfy the type variable associated with the generic Pointer interface. Using Valhalla specialized generics, we will be able to generalize the Pointer generic interface as follows:
interface Pointer<any X> {
...
<any Z> Pointer<Z> cast(LayoutType<Z> that);
static <any Z> Pointer<Z> allocate(LayoutType<Z> type);
}Thanks to specialized generics, we can now accurately model an int pointer as a Pointer<int>; but what would happen to code that has been already written against the Pointer<Integer> type? As we have learned when attempting specialization of the Stream API, doing this kind of migration is very hard. Note that this is not just about code generated by tools like jextract (one could imagine there being a flag that switch specialized generics on or off, for compatibility reasons) - but also extends to issues with the LayoutType interface - which would have to be specialized too. More specifically, what does LayoutType.ofInt returns in the new specialized world? If it returns LayoutType<int>, then we risk breaking existing clients; so perhaps a better alternative would be to add a new set of factory methods returning specialized LayoutType instances.
Arrays are also part of the C programming model and, as we shall see, modeling them in Java present similar issues as those described for pointers. One important point we need to make about native array types is that there are two places where they can occur in C API: (i) in function signatures and (ii) in struct field declarations. Of these two cases, the only important case the binder needs to support is the latter. That's because array function parameters are simply treated by the C compiler as pointers - e.g. consider the following C snippet:
void f1(int a[], int b[]) {
a = b; //ok
}
void f2() {
int a[2];
int b[2];
a = b; //error
}Here we can see that the array-ness of the a and b variable is only enforced in the second example, where a and b are local variables. Conversely, if a and b are function parameters, the compiler treats them as pointers, meaning that assigning b to a is allowed! A similar argument holds for array types in return position:
int[] f4(int[] arr) {
return arr;
}The above declaration is not even legal in the C programming language - array types cannot be used as function return types. This suggests that the binder should always model array function parameters as pointers, avoid arrays in return type position (as they are illegal) and only use arrays to model array struct fields (and global variables).
While it would be tempting to model C arrays using Java arrays, doing so would violate the efficiency principle laid out before - going from a native array to a Java array requires an expensive O(n) bulk operation. Therefore, to achieve an efficient mapping, we should resort to a more lazy approach, where the native array is wrapped into an instance of some Array interface which exposes operations that a C programmer might expect to be able to do on arrays. Let's try to enumerate such operations below:
array to pointer conversion: that is, given an array, it should always be possible to convert the array into a corresponding base pointer - e.g. a pointer to the first element of the array.
size: that is, given an array, it should be possible to obtain the length (in number of elements) associated with said array.
dereference: that is, given an array, programmers should be able to get/set the contents of the i-th element of the array.
cast: that is, it should be possible to convert an array of a given type, to an array of a different type (provided the two types are compatible).
From the above properties it should now be clear that an array should at least be represented as a (pointer, long) pair: that is, an array has some base address (the pointer value, corresponding to the address of the first element of the array), and a length; the layout information associated with the pointer allows the array to perform (indexed) dereference operations correctly. As in the case of pointers, we can model all this with a suitable Java interface:
interface Array {
Pointer basePointer();
long length();
Object get(int index);
void set(int index, Object value);
Array cast(Layout that);
}The above proposal clearly has some expressiveness problems (which we will attempt to rectify in the subsequent section), but it's nevertheless a good start, as it correctly expresses the operations that can be done on an array. It can also be seen how, for the binder implementation, it is straightforward to go back and forth from native arrays to Java instances of the Array interface: the embedding consists in creating a new Java instance wrapping some base pointer (along with some size info which we assume to be available in some metadata), whereas the projection amounts at either calling the basePointer method on said interface (in case an array is passed as a function parameter) or extracting the contents of the memory region associated with the base pointer - both O(1) operations, thus respecting the efficiency principle.
There are two main ways to create native arrays; the first would be by converting a Java array into a native array, through appropriate factory/conversion methods. The second would be, again, through dynamic memory allocation (in C often known as calloc). For the purpose of the following discussion, and for the sake of simplicity, we can assume that the Array interface has extra static factories, as follows:
interface Array {
...
Object toArray();
static Array fromArray(Object arr) { ... }
static Array allocate(Layout layout, int size) { ... }
}As for pointers, any attempt to enhance the static type information associated with the array API will affect the signature of the allocation factories.
Finally, in C it is always possible to go from an array to its base pointer without the need of an explicit conversion. This seems to suggest that arrays are pointers. While we could definitively design an API around this principle (e.g. make Array a sub-interface of Pointer), we believe this choice would be too detrimental in terms of usability: now Array would also inherit all the methods available in Pointer (e.g. non-indexed getter/setters, pointer arithmetic methods, etc.). For this reason, we have decided to model Array as a separate abstraction, and to insert an explicit conversion operation to go from array to pointers - namely Array::basePointer.
As done for the Pointer interface, we will now show how to enhance the Array interface, adding back some static type information.
The main problem of the interface presented in the previous section is the lack of expressiveness - the accessor methods get and set are defined in terms of Object. Similarly to what we have done with pointers, we could try to add back some static information about the element type by doing something like this:
interface Array<X> {
Pointer<X> basePointer();
long length();
LayoutType<X> type();
//accessors
default X get(int index) {
return basePointer().offset(index).get();
}
default void set(int index, X value) {
basePointer().offset(index).set(value);
}
<Z> Array<Z> cast(LayoutType<Z> that);
Object toArray();
static <Z> Array<Z> fromArray(LayoutType<Z> type, Object arr) { ... }
static <Z> Array<Z> allocate(LayoutType<Z> type, long size) { ... }
}This looks like an improvement - we can now represent types such as Array<Integer>, Array<Character> and the likes. As for pointers, the carrier information associated with the array is modeled as an instance of the LayoutType class, which will need some additional combinators to handle arrays:
interface LayoutType<X> {
MethodHandle getter();
MethodHandle setter();
Layout layout();
LayoutType<Array<X>> pointer();
LayoutType<Array<X>> array();
LayoutType<Array<X>> array(long size);
...
}This means that we can programmatically construct the layout type associated with an int array of 5 elements as follows:
LayoutType<Array<Integer>> pIntLayout = LayoutType.ofInt.array(5);This approach, while expressive, also inherits all the disadvantages of the generic pointer approach it is based upon: the complexity of the binder implementation is increased, since instances of Class (and MethodType) are not sufficient to model array carrier types; additionally, since this approach uses boxed argument types such as Integer to model primitive element types, there are unavoidable issues when it comes to migrate the generic Array interface to be a specialized generic interface; such issues are likely to be exacerbated by the existence of the toArray factory method, which just returns Object here.
Native structs are modeled as instances of native type interfaces (see above). As we have seen, an interface modeling a struct must be annotated with a @NativeStruct annotation defining the layout of the underlying native struct. This allows the binder implementation to work out details such as the offsets and sizes of each struct fields.
In terms of efficiency, modeling structs as interfaces is quite effective - again, we have efficient embedding/projection pairs with O(1) complexity: creating a Java view of a native struct only requires to wrap a struct pointer into a suitable struct interface implementation. Conversely, turning a Java interface modeling a struct into a native struct can be achieved by extracting the contents of the memory region associated with the struct pointer embedded in the Java interface.
In addition to the @NativeStruct annotation, it is also useful to define a root type for all native struct interfaces:
interface Struct<X extends Struct<X>> {
Pointer<X> pointer();
}For instance, a native type interface modeling a C struct can be defined as follows:
@NativeStruct("...")
interface Tm extends Struct<Tm> {
int tm_sec();
...
}If all native struct interfaces extend the Struct interface, such interface can now be used it to define operations that works on structs; for instance, the LayoutType interface could define a factory like the one below:
interface LayoutType<X> {
...
static <Z extends Struct<Z>>
LayoutType<Z> ofStruct(Class<Z> clazz) { ... }
}In the remainder of this document we will discuss how advanced interoperability features such as variadic functions, macros and callbacks can be achieved. Given the complexity of some of the features discussed in the sections below, which will virtually be impossible to render in a succint way, in some cases we will limit ourselves to provide some high level detail on how interoperability with respect to these features can be achieved.
The C language allows for variadic functions, which can take a variable number of arguments. Perhaps the most known case of variadic function is printf in the standard library:
int printf(const char* fmt, ...); //from <stdio.h>
printf("%d + %d = %d", 1, 2, 3)When we have a variadic function, there must be some regular argument of that function which gives away some information about the arity of the call. In the case of the printf function shown above, the arity can be inferred by parsing the format string argument; since in this case it features three format specifiers of the kind %d, it follows that the function call needs three extra arguments. The C language tries to enforce this idiom, by requiring that at least one named parameter appears before the ... used in variadic function declarations.
Java also has (since Java SE 5) the ability of declaring variable-arity methods (this feature is commonly referred to as varargs):
interface List<E> {
...
<Z> List<Z> of(Z... args);
}
List<Integer> li = List.of(1, 2, 3);It would be tempting to use the Java varargs feature to model C variadic functions; for instance, we could model printf as follows:
@NativeHeader("printf=(u64:u8*)i32")
interface Stdio {
int printf(Pointer<Byte> fmt, Object... args);
...
}While something like this could be made to work, it is important to note that this translation would be lossy, as the additional variadic arguments will have no layout information. While this might be ok when working with small values (esp. given that, in the case of variadic call, the set of default conversions associated with prototype-less calls take place), the approach does not scale when taking into account bigger values, such as structs. In fact, in order to pass a struct argument, some system ABIs (see SystemV) will have to perform some kind of recursive classification of the struct fields, which is only possible if the layout information is available. While it could be possible, in principle, to dynamically fetch the layout information from the struct argument being passed to the variadic function (e.g. through reflection, by parsing the contents of the @NativeStruct annotation), doing so seems inefficient and irregular.
In order to correctly preserve layout information, each additional variadic argument must be accompanied with some kind of layout information; this can be achieved by modeling the trailing arguments as pointers, as follows:
@NativeHeader("printf=(u64:u8*)i32")
interface Stdio {
int printf(Pointer<Byte> fmt, Pointer<?>... args);
...
}Since a pointer always has layout information (via its LayoutType field), the binder can dynamically recover the layout information associated with variadic arguments, and use such information to perform argument classification, if needed, as required by the native ABI.
While this representation is complete, it is worth noting that it leads some issues. First, from an usability perspective, the client will now have to buffer variadic arguments and extract pointers - although this could be mitigated by providing an helper class for generating variadic argument lists. Secondly, from a performance perspective, such a representation is not very flat, which might make standard optimization strategies less effective - one possible solution here would be to define hand-written non-variadic versions of the variadic method, following the approach described here.
The C language allows to define function-like macros - that is, macros whose use looks very similar to a function call; an example is given below:
#define SQUARE(x) x * x
...
int sq = SQUARE(4); //16
sq = SQUARE(sq); //256Macros are introduced using the #define pre-processor directive, and macro arguments feature a syntax which is reminiscent of that used in function declarations. That said, while a function-like macro might look similar to a function, the similarity is purely syntactic. First, macros are eagerly expanded by the preprocessor, ahead of type-checking; this means that the macro definition is removed and the use-site code above is turned into the one shown below:
int sq = 4 * 4; //16
sq = sq * sq; //256This reveals an important difference between macros and functions; when a function is called, some kind of transfer of control always takes place - typically governed by some system ABI. Macro-like functions, on the other hand, are just expanded in place, meaning no real call takes place. This can be a powerful optimization trick for when the cost of computation is negligible compared to the cost of transferring control flow using a classic function; for this reason, function-like macros are given even more of a central role in C++, with the ability of defining inline functions - although the C++ compiler will at least validate the actual arguments passed to the inline functions against the declared function prototype.
Implementing proper function-like macro support is hard - after all, the native C compiler does all the lifting when it comes to a macro call, by expanding the macro definition at the use-site. No trace of the macro exists in the dynamic library, meaning that the usual linking tools available to the binder will not help here. To allow function-like macro support, it is therefore necessary to wrap the macro into a function, as follows:
int square(int x) {
return SQUARE(x);
}By doing this, we give the macro an entry-point the binder can reason about. Note that here we have made an arbitrary choice - the prototype of the square function explicitly states that the function works on int arguments, therefore limiting the applicability of the original macro. This suggests that the process of wrapping a function-like macro in a real function always involves some kind of human intervention and cannot be fully automated - although approaches such as the one described here could help in reducing the cost associated with the generation of such custom wrappers.
The C language supports callbacks via function pointers. Consider the following popular example, the qsort function, defined in the C standard library:
void qsort(void *base, int nitems, int size, int (*compar)(void*, void*))This function can be used to sort the contents of any array, regardless of the element type. This is achieved by delegating the element comparison logic to some user-provided function - the compar parameter in the definition above.
To model these kind of idioms in Java code, we need to associate some kind of carrier to function pointer parameters such as the one shown above. A good carrier type for function pointer is a Java functional interface, that is, an interface that contains a single abstract method (and can therefore be used as the target-type for a lambda expression):
@NativeHeader("qsort=(u64:[0i32]i32i32u64:(u64:i32u64:i32)i32)v")
public interface StdLib {
void qsort(Pointer<Integer> base, int nitems,
int size, QsortComparator comparator);
@NativeCallback("compare=(u64:i32u64:i32)i32")
interface QsortComparator {
int compare(Pointer<Integer> p1, Pointer<Integer> p2);
}
}Note: the above code adopts the generic pointer approach, to increase readability - but similar conclusions apply for the other approaches described in this document.
Here we have a native library interface which defines the qsort method; crucially, the third parameter of this method is another interface type, namely QsortComparator. QsortComparator is a functional interface - its only abstract method is compare - and this interface features another annotation, namely @NativeCallback, which is used to associate a native layout to the arguments of the function modeled by the interface (this can be used by the binder to cross-validate the @NativeCallback layout with the layout information contained in the @NativeHeader annotation). In the snippet below, we show how clients interacts with the qsort function:
Array<Integer> qsort(Array<Integer> array) {
//call the function
return qsort(array, array.length, 4,
(p1, p2) -> {
int i1 = p1.get();
int i2 = p2.get();
return i1 - i2;
});
}The net effect is quite pleasing: since the function pointer parameter is modeled as a functional interface, clients can just pass lambdas which define the desired element comparison logic.
In terms of binder implementation, achieving this is relatively complex: the functional interface instance must be converted into a native function pointer - which means that the functional interface declaration has to be reflectively inspected, the associated layout extracted, and some kind of native stub allocated on the native stack. Then, the functional interface argument is replaced by a pointer to such stub, so that the native call can take place. While some of these operations (such as reflective lookups) can be performed only once (when we build the recipe of a given native call), the allocation of the required stubs would still need to take place at every call (since the Java method called by the stub changes with the specific functional interface instantiation passed in as parameter). Some strategy would need to be put in place in order to mitigate some of the performance penalties associated with this solution; for instance, stubs could be pre-allocated and shared on a per-form (e.g. erased parameter types, erased return types) and per-thread basis.
Callbacks are not the only place where function pointers show up; in fact, the C programming language allows structs to declare fields whose type is a function pointer type. Here things get even more convoluted, because we need to support multiple use cases: first, a function pointer field can be read or written; secondly, the source of a function pointer (e.g. where its code comes from) could be either Java, native code or the binder itself! This gives us a two by three matrix of use cases, as shown below:
| Source | Setter | Getter |
|---|---|---|
| Java | allocate stub | allocate proxy (scoped) |
| C (native) C (stub) |
reuse address |
allocate proxy (non-scoped) allocate proxy (scoped) |
When a client pass some Java functional interface instance to a struct setter, the outcome is similar to what has been described in the callback case: a native stub is allocated, its address retrieved and stored in the memory address corresponding to the struct field being set. Conversely, if we retrieve a function pointer from a struct field, as a functional interface instance, the binder will generate a proxy on the fly - that is, an implementation of the desired functional interface whose body will perform a native call to the function whose address is stored in the struct field. Note that strategy works regardless the underlying function pointer is targeting a binder-generated stub, or a true native function.
It is important to note that, depending on whether the proxy allocated by the getter is associated to some binder-generated native stub, the proxy might need to keep track of the original scope in which the native stub has been allocated. This is crucial: a scoped proxy can only be called if its underlying scope is still alive - in fact, when the stub goes out of scope, the binder might want to reclaim the native memory associated with the stub. This problem is absent in case a functional interface proxy points to a true native function, in which case we can safely assume that the native function life-cycle will be managed outside the Java code.
Finally, if the functional interface instance to be set on the struct field is a binder-generated proxy, the binder can skip the stub allocation, obtain the underlying address associated with the proxy and write that address into native memory. This is a far more efficient approach, as it does not involve allocation of any native resources.
Note: the attentive reader might have noted that keeping track of scope information, once a function pointer has been written into native memory is hard; in fact, the written value will be a machine word pointing to some native entry point. That said, we can assume the existence of ways in which we can quickly check as to whether an address is indeed the one of a binder-generated stub. If that is indeed the case, since native stubs are generated by the binder itself, we can also assume that there is some way to get at the scope information directly from the stub - e.g. such info could be stored at a fixed offset inside the stub (and have the stub code jump over it).
I would like to thank John Rose, Brian Goetz, Athijegannathan Sundararajan and Henry Jen for the countless (and fruitful!) discussions which helped shape this document. A big thank also goes to Paul Sandoz, whose experiments with the OpenCL library proved very helpful in evaluating the differences between the various design tactics.