On treatment of null in pattern matching

July 2017: (v. 0.1)

Maurizio Cimadamore

This document contains some of the explorations we have carried out regarding the treatment of null in patterns. As per Brian's document, there are many different kinds of patterns:

For all the above, we have to answer the question: what happens if the value against which the pattern is tested is null ? There are of course many possible answers to this seemingly simple question - this email is an attempt at describing some of those.

Now, for constant patterns the answer is pretty straightforward - assuming that a constant pattern is just testing a value for equality against the constant mentioned in the pattern. This means that a constat pattern of the kind case 42 would always reject a target expression whose value is null. On the other hand, a constant pattern of the kind case null would always reject a target expression whose value is not null.

Note that the addition of the null constant pattern allows us to generalize over the status quo (where switch on null always throws a NullPointerException) - as we can reinterpret existing reference switches as if they were written as follows:

switch (s) {
  case null: throw new NullPointerException(); //synthetic code
  case "Hello": ...

That is, having an explicit case null let us define precisely the semantics of existing switch statements and, possibly, generalize over it, by allowing custom handling of nulls. This seems a powerful mechanism, especially considering our desire of merging pattern matching switch with legacy switch. That said, this can also be viewed as an orthogonal improvement, whose consequences do not largely affect the discussion in this email. Therefore, from now on we will refrain from discussing null patterns (or any other constant patterns) in more details.

In order to further simplify the discussion, let's also ignore var patterns; in fact, a pattern such as var x can always be thought of a type-test pattern where the type of the test is implicitly inferred by the compiler using some target-type information. in other words, given a target expression s whose static type is String, the following two snippets should behave in the same fashion:

if (s matches String x) { ... }
if (s matches var x) { ... }

That is, a var pattern can be thought of a shorthand for a more explicit type-test pattern 1, without losing generality.

As a final simplification, let us also ignore the special _ pattern; since this pattern is meant to match anything, this implies that it should match nulls too.

Option 1: Instance test-based interpretation

So, our goal is to define the behavior of type-test patterns and deconstructing patterns with respect to nulls. A possible interpretation for both patterns is that they are morally equivalent to an instance test (instanceof). That is, in code like this:

Object o = ...
if (o matches String s) { ... }

or code like this:

Object o = ...
if (o matches Point(int x, int y)) { ... }

the role of both patterns is to check as to whether the target value of the match operation has a type that is compatible with the type mentioned in the pattern. More specifically, in the first example, the translated code would perform an instance test of the kind o instanceof String, while in the second example an instance test of the kind s instanceof Point would be carried out.

If we pull on this string some more, we see that the instance test metaphore is already suggesting a possible interpretation with respect to handling of null in these patterns; in fact, since an instanceof test would typically fail on a null value, this seems to suggest that the above patterns should equally fail to match expressions whose value is null.

While this is a totally sensible interpretation, there are some signs suggesting that the instance test interpretation might not be totally adequate. More specifically, this interpretation suffers from two major issues:

Let's look at the first problem:

ArrayList<String> ls = ...
if (x matches List<String> ls) { ... }

Obviously, there's nothing wrong with the code above - the type mentioned in the pattern is clearly compatible with the type of the match operand - in fact this is a statically provable property of the program. And yet, if we apply the instance test metaphore strictly, what the code is really asking here is ls instanceof List<String> which is not an allowed operation in the Java language. One could argue that in such cases, there's really no instance test going on (more on that later), given the compiler can prove that the type in the predicate is always a supertype of the type of the matched expression; but what about this:

Object o = ...
if (o matches List<String> ls) { ... }

In this case a dynamic check seems unavoidable; but even so, should the program above issue an unchecked warning, rather than just failing to compile - as it would happen if we used a semantics based on instanceof?. Something is clearly off here.

The second problem is that failing to match case Box(var x) on new Box(null) is totally counterintuitive, as demonstrated in the following snippet:

Object o = ...
if (o matches LinkedList(Object head, LinkedList tail)) { ... }

If we applied strict instance test semantics, the above pattern would fail to match if the head (or tail) of the list is null. This leaves us in a bad place - how do we match a linked list whose head is null?

The attentive reader could point out that, by adopting some kind of opt-in for nullity, we might be able to solve this problem; after all, all we need to do would be to re-purpose the null pattern?

if (o matches LinkedList(null, LinkedList tail)) { ... }

While this looks fine, note that when deconstructing a class with N fields, this would generally lead to an exponential explosion of pattern forms - as 2N patterns might be required to match against all possible forms. Again, something looks odd here.

Option 2: Type-system driven interpretation

As we noted above, not all type-test patterns are created equals. Some type tests feature a type that can be statically proven to be compatible with the type of the matched expression. That is, if T is the static type of the type test T t, and E is the static type of the matched expression, we have two cases:

We call a type-test that falls in the first category a type-restate patterns, because it basically re-enforces knowledge already available in the static type system.

Now that we have a more fine-grained distinction between regular and restating type-tests, we could exploit this classification for two purposes:

Let's call this a type-system driven interpretation. To see how such an interpretation works, let's look at some concrete examples:

String s = null;
if (s matches String x) {
   System.out.println(x.length()); //NPE?

Using a basic instance test interpretation (see section above) the code above would not throw. However, if we adopt the type-system driven interpretation discussed in this section, this code starts to throw exceptions - this is caused by the fact that we have a type-restate pattern (String <: String) - and, as a result, nulls are accepted.

On the other hand, the following program will not throw, regardless of the interpretation:

Object o = null;
if (o matches String x) {
   System.out.println(x.length()); //NPE - but we can't get here!

The above is a regular type-test (Object is not a subtype of String), so nulls would be rejected by the pattern test, and the code inside the if would not be executed. In other words, treatment of nulls here is the same as with the more basic test interpretation described earlier.

What about nested patterns? The same rules would apply:

LinkedList ll = new LinkedList(null, null);
if (o matches LinkedList(Object head, LinkedList tail) {
   System.out.println(head.getClass()); //NPE?

Again, this code did not throw under the instanceof-driven semantics described earlier; however, things are different under a type-system driven interpretation, since we now have a nested type-restate pattern which is allowed to accept nulls. On the other hand, if the code is rewritten as follows:

LinkedList ll = new LinkedList(null, null);
if (o matches LinkedList(String head, LinkedList tail) {
   System.out.println(head.getClass()); //NPE - but we can't get here!

No exception occurs, since the nested pattern is now a regular type-test. Again, for non-restating patterns, semantics of the new type-system driven behavior is the same as with the more basic instance test-based interpretation.

But what if ll is itself null? On the one hand, the static type of ll is compatible with that of the deconstructing pattern, suggesting that a type-restate semantics (which accepts nulls) should be adopted. On the other hand, it should be noted that in the vast majority of cases, after matching a deconstructing pattern the code is likely to dereference it to extract one or more components. This means that accepting nulls in this case is a bit moot, given that the pattern test would NPE immediately afterwards.

To fix this issue, we can view destructuring patterns as a composition of two ingredients: a type test pattern (which could be regular, or restating) plus a sequence of extraction operations. This mental model offers us a possible way out to the problem mentioned above: while a type-restating is always null friendly, an extraction never is. This means that a destructuring pattern should always fail if the matched value is null (either because its type-test component is non-restating, or, if it is restating, because its underlying extraction 2 operation would fail).

Too clever?

Having static typing drive dynamic properties such as null friendliness is clever, but maybe too subtle. As we have already seen, changing a static type assertion (e.g. change the type of the matched expression from String to Object) affects the runtime behavior of the program. That seems too subtle, and has also the potential to lead to effect-at-distance issues: if the type of a pattern has its supertype hierarchy rewired (which is possible if the type is a private type in the same compilation unit), the hierarchy change can cause ripple effects on handling of nulls. For instance:

Foo foo = null;
if (foo matches Serializable s) {

class Foo implements Serializable { ... }

This would compile, run and print Hello. In fact, since Foo <: Serializable, this is a type-restate pattern, so null would be accepted here.

But what if Foo is later changed to this:

class Foo { ... }

that is, what if we drop the Serializable interface from Foo?

Well, the program would still compile. Because Foo and Serializable are still castable one to the other thanks to the side-cast rules - that is, it's possible for a subtype of Foo to implement Serializable, so, unless Foo is made final, it would be too harsh for the compiler to reject this possibility.

But this is no longer a type-restate pattern - because Foo is not a subtype of Serializable. Which means null would not be tolerated, and, at runtime the Hello string is not printed.

Option 3: Cast-based interpretation

Ultimately, the type-restate vs. type-test distinction is a good one to make when reasoning about static properties of type test patterns, but one that ultimately is leading us to simply saying: the static rules for typing type-test patterns have to be the same rules used in cast conversion (which already makes the distinction about conversions that are statically provable, with respect to those that aren't). Once cast conversion is embraced, then all type-test patterns look the same - and are treated the same; because of the cast rules, some of them would be unchecked (or outright banned - if we wanted a stricter semantics), while others will not.

In other words, all this suggest a possible, alternate intepretation to the instance test-based one: a type test pattern succeeds if a cast conversion between the type of the matched expression and the type of the type-test pattern also succeeds. In terms of code, we could define the cast-based semantics as follows:

boolean matched;
try {
    T t = (T) expr;
    matched = true;
} catch (ClassCastException _) {
    matched = false;

This cast-based semantics leads us to a slightly different place, which can be summarized as follows:

This seems to lead to an acceptable - and more straightforward - user model - after all, case Point() is used to denote deconstruction (e.g. the dual operation to a default constructor call), while case Point denotes a simple type-test pattern. Not only the semantics of the two pattern is very different, but there is a not too subtle syntactic difference between the two patterns (a distinction we're already using in other places, to distinguish between field accesses and method calls, for one). Since this distinction is more more manifest than the subtyping distinction which occurs under the type-system driven interpretation (e.g. type restate vs. type test), this leads to more explicit and readable code. The fact that we have an extra null pattern allows the user to be more explicit about null handling - if he chooses to do so. Also, as shown earlier, null patterns can also be used recursively inside components of a deconstructuring pattern, which allows for nice expressiveness:

if (ll matches LinkedList(var head, null)) { ... }

Unfortunately, the cast-based interpretation has also a major downside; let's consider the following code:

if (o instanceof String) {

How would we translate it using patterns? One might think that the following code suffices:

if (o matches String s) {

But that's unfortunately not true, given that, under the cast-based interpretation, the above pattern would also match nulls. So, to avoid bugs (dereferencing null), the translation has to be more defensive, as follows:

if (o != null && o matches String s) {

This is an unavoidable consequence of the fact that the language applies different rules for cast conversions and instanceof tests - so, e.g. choosing an interpretation of nulls based on cast conversion rules would result in anomalies for code written using instanceof-based interpretation.

Option 4: Bimodal interpretation

We have perhaps come at the core of the issue here: there is a fundamental difference between variable and deconstructing patterns. For variable patterns, it might or might not acceptable for a pattern to match null, but in the case of a destructuring patterns, it is always an issue to match nulls, given that the very nature of a destructuring pattern will potentially lead to derefrencing operations (which will lead to runtime exceptions). This seems to suggest that destructuring patterns should always reject null, regardless of what we do for variable patterns.

What about variable patterns then? It is helpful here to distinguish between toplevel variable patterns and nested variable patterns. Toplevel variable patterns are more likely to model instance tests, so it would be desirable for them not to match target expressions whose value is null. On the contrary, nested variable patterns obey quite a different rule; their semantics is arguably closer in spirit to the cast-based interpretation. That is, a nested variable pattern should match nulls - this avoids the exponential explosion problem discussed above.

The table below summarizes the results described above:

pattern toplevel nested
variable rejects null accepts null
destructuring rejects null rejects null

This model seems promising; one the one hand it allows straightforward refactoring of instance-test based code (shown in the previous section); it also prevents exceptions when deconstruction occurs; finally, its flexible treatment of nulls in nested variable patterns avoids the exponential pattern explosion pitfalls outlined in earlier sections.

What this model does give up is uniformity: by design, treatment of variable patterns is context-dependent: toplevel patterns are more strict when it comes to handling nulls, nested patterns are more relaxed. While this could be problematic when refactoring a pattern from a toplevel context to a nested one (or the other way around), considering all the issues discussed in this document, this one seems the least harmless.

  1. if the target-type is non-denotable it might be a case that such rewriting might not be possible - this is also a problem tackled by the local variable type inference JEP, and currently under discussion. For the time being, let's just assume that all types are denotable.

  2. the attentive reader would have noticed that it is possible for a deconstructing pattern not to have any components:

    if (p matches Point()) //shorthand for Point(0,0)

    Since there's no extraction going on here, is accepting null acceptable here?