Updating URI support for RFC 3986 and RFC 3987 in the JDK

Motivation

java.net.URI claims conformance to RFC 2396 (Uniform Resource Identifiers (URI): Generic Syntax) and RFC 2732 (Format for Literal IPv6 Addresses in URLs), with deviations. These two RFCs have been obsoleted by RFC 3986, which is now an Internet Standard. In addition, java.net.URI supports non US-ASCII characters in URIs, which has been later formalized by RFC 3987. Because java.net.URI predates both RFC 3986 and RFC 3987 the documentation makes no mention on how java.net.URI deviates from these two RFCs.

Many bugs and confusion could be avoided if java.net.URI could be upgraded to support the newer RFCs.

Historical consideration

An attempt was made in the past to upgrade java.net.URI to support RFC 3986 and RFC 3987. This caused many JCK tests failures and compatibility issues and the change had to be backed out (see JDK-6394131). To reduce the risk of incompatibilities arising again several roads have been explored.

Significant differences between RCF 2396/RFC 2732 versus RFC 3986/RFC 3987 which impact compatibilty in java.net.URI

RFC 3986, Appendix D has an exhaustive list of differences between RFC 2396 and RFC 3986. Because java.net.URI supports RFC 2386 and RFC 2732 with deviations this list is not an exact match for java.net.URI, but is still very useful as a starting point to understand the impact on java.net.URI.

The most significant differences however are:

• changes in the parsing of the authority component
• clarification on the path normalization and resolution operations
• clarification on the opaque versus hierarchical terminology
• obsolescence of the scheme-specific-part (opaque_part) rules
• introduction of the gen-delims/sub-delims character categories

The table below shows the differences in legal characters accepted for each component, for java.net.URI versus RFC 3986:

    +-----------+------------------------------------------------------+-------------------------------------------------------+-------------------+
    | component | java.net.URI (current implementation)                | RFC 3986                                              | Diff              |
    +-----------+------------------------------------------------------+-------------------------------------------------------+-------------------+
    | scheme    | [A-Za-z0-9] + "+-."                                  | [A-Za-z0-9] + "+-."                                   | same              |
    | user      | [A-Za-z0-9] + "-_.!~*'()" + ";:&=+$," + %-enc        | [A-Za-z0-9] + "-._~" + "!$&'()*+,;=" + ":" + %-enc    | same              |
    | reg_name  | <user> + "@"                                         | <user> without ':'                                    | old = new + "@:"  |
    | path      | [A-Za-z0-9] + "-_.!~*'()" + ":@&=+$," + ";/" + %-enc | [A-Za-z0-9] + "-._~" + "!$&'()*+,;=" + ":@/" + %-enc  | same              |
    | opaque    | [A-Za-z0-9] + "-_.!~*'()" + ";/?:@&=+$,[]" + %-enc   | [A-Za-z0-9] + "-._~" + "!$&'()*+,;=" + ":@/" + %-enc  | old = new + "[]?" |
    | query     | [A-Za-z0-9] + "-_.!~*'()" + ";/?:@&=+$,[]" + %-enc   | [A-Za-z0-9] + "-._~" + "!$&'()*+,;=" + ":@/?" + %-enc | old = new + "[]"  |
    | fragment  | [A-Za-z0-9] + "-_.!~*'()" + ";/?:@&=+$,[]" + %-enc   | [A-Za-z0-9] + "-._~" + "!$&'()*+,;=" + ":@/?" + %-enc | old = new + "[]"  |
    +-----------+------------------------------------------------------+------------------------ ------------------------------+-------------------+

basically this means that, if we compare the current implementation of java.net.URI with what RFC 3986 mandates:

• java.net.URI does not throw if the authority component contains more than one @ or : while it should according the new RFC.
• java.net.URI opaque path "contains" the query, while it should not according to the new RFC
• java.net.URI opaque path, query, and fragment, can contain unencoded [], while they should not according to the new RFC.

At this point a table showing some concrete examples might be the best way to visualize what could be the impact on java.net.URI if it were to be updated to conform to the newer RFCs.
If java.net.URI was updated to support RFC 3986 / RFC 3987 then we could chose not to support some of these differences and list them as new deviations. Others would be more awkward to justify.

The table above shows the challenge of providing support for the newer RFC in a backward compatible way. There are also more differences in the way that java.net.URI and RFC 3987 restrict non ASCII characters. The following section lists the alternatives that have been envisaged.

Possible Solutions

Several different solutions have been explored. Two of them have been prototyped.

1. Do nothing. Just stay with the old RFCs (what we have now)
2. Update URI in a major version, force old code to adapt (rejected: basically what has been attempted before, and which lead to a backout)
3. Same as above - but add a big switch, e.g: a system property, to select conformance to the new or old RFC (rejected).
4. Same as above, but on a per-URI instance basis, depending on how the URI is constructed (rejected: too many combinatorial issues, see below).
5. Add a new subclass of java.net.URI (rejected: URI is final but has public constructors, we can't remove the final keyword. Even if we could, this would still present major compatibility risks when passing the subclass to old code expecting the super class behavior, and we would stumble on the same issues than with the previous bullet).
6. Leave java.net.URI alone, and add a new class e.g. java.net.IRI, which would implement the new RFC. To ease migration and possible future evolution, also introduce an abstract common ancestor to both classes, e.g. java.net.ResourceIdentifier (prototyped).
7. Do not change the behavior of java.net.URI, but document its deviations towards RFC 3986 / RFC 3987 instead of documenting its deviation towards RFC 2396 / RFC 2732. Where possible, add new APIs to bring java.net.URI closer to RFC 3986 / RFC 3987. Only change the behavior of existing methods marginally, when regressions are unlikely (prototyped).

Prototypes

Prototype 1: Introducing a new public java.net.IRI class

This solution has been prototyped. The prototype is in a reasonable shape - but may still require more work. Here is a high level description of the prototype:

• For a complete list of the changes see the prototype webrev
• A new java.net.IRI class that supports RFC 3986 and RFC 3987 and supercede java.net.URI is introduced. java.net.URI is left unchanged (except for some API clarifications).
• In order to ease migration, a new abstract java.net.ResourceIdentifier class that is a common super class to java.net.URI and java.net.IRI is introduced. The class is abstract - and cannot be subclassed outside of the java.net package. Code that accepts a ResourceIdentifier as parameter should always convert it to one of its concrete subtype, e.g. by calling java.net.IRI.of(ResourceIdentifier).
• The new java.net.IRI class is final, offers factory methods and lightweight builders, but has no public constructor.
• Conversion between URI and IRI can be performed using java.net.URI.of(ResourceIdentifier) and java.net.IRI.of(ResourceIdentifier).
• The new java.net.IRI class offers static convenience methods to help deal with encoding and decoding printable US-ASCII charcters which are illegal in URIs, such as the pipe characters ('|'). For instance the IRI::parseLenient method can thus be used to easily build an IRI from a string representation that contains '|'.
• The new java.net.IRI implements the restrictions on non US-ASCII characters as specified by RFC 3987, with only one minor deviation (non US-ASCII space chars are rejected, like the US-ASCII space character, as java.net.URI used to do).
• New APIs are added to File, Path, and HttpClient.RequestBuilder to accept ResourceIdentifier (the common ancestor).

Performance and footprint considerations:

• The java.net.URI class is left unchanged. There should be no performance impact on existing APIs.
• The new java.net.IRI class tries to optimize parsing as much as it can, but the new parsing rules and need to deal with internationalized characters as specified by RFC 3987 may have an impact. However this shouldn't affect existing code that deals with java.net.URI.

Testing:

• All existing unit tests are passing.
• All existing JCK tests are passing.
• All java.net.URI tests have been cloned, adapted, and extended to verify that java.net.IRI doesn't re-introduce bugs previously fixed on java.net.IRI, and that the new behavior is covered.
• A new BuilderTest has been added to test the new lightweight builder.

Compatibility:

• To reduce the risk of surprises, the IRI::getHost() method will also return null if the string it returns is not a syntactically valid IP address or DNS name. Alternate accessors are provided to get at the raw and decode host string, regardless of its form.
• IRI::getPath() however, will not return null when the path is opaque: it will return the opaque path. Given that this is a new API, the risk should be acceptable, even with the addition of the new ResourceIdentifier abstraction.

List of issues logged against java.net.URI that have been fixed in (or are no longer applicable to) java.net.IRI

• JDK-6523089 java.net.URI.relativize(java.net.URI) mishandles non-slash-terminated base URI
• JDK-4666701 URI.resolve wrong when base path empty and given URI relative
• JDK-6920138 URI not handling empty reference resolution
• JDK-4708535 same document URI resolution not consistent (same as above)
• JDK-8075116 java.net.URI does not parse empty authority correctly.
• JDK-8051628 java.net.URI 5-arg ctor is not verifying server authority = > This is fixed by the virtue of being not applicable
• JDK-8180809 URI and URL conflicts with dealing with hostname contains underscore => The IRI prototype provides additional method to query the nature of a host component.
• JDK-7037120 Identity violation in java.net.URI (this is also about normalization/relativize that do not conform to their spec)
• JDK-6579275 JAR URL/URI parsing is not correct for empty relative paths (I believe the IRI prototype fixes that too - for `file://` IRIs, not for URI/URL and not for `jar:` URIs).
• JDK-8190362 java.net.URI API documentation: empty authority not a deviation from RFC 2396
• JDK-8054024 URI constructor violates guaranteed identity for URI with [brackets] in path (Identities have been fixed in the IRI prototype) The issue with this is that using schemeSpecificPart is confusing. This has been fixed by RFC 3986.
• JDK-8067436 URI (multiparam) constructor throws URISyntaxException when path contains [] => same as above
• JDK-8184958 URI getSchemeSpecificPart() does not decode properly => same as above.
• JDK-8140634 URI getSchemeSpecificPart returns incorrect value => same as above.
• JDK-8051627 Invariants about java.net.URI resolve and relativize are wrong => resolve/relativize and the documentation of the related invariant has been fixed in the prototype.
• JDK-8022748 new URI(u.toString()).equals(u) does not hold with paths containing colons => (fixed in the prototype same as above)
• JDK-8149517 URI problems with paths that include the colon character
• JDK-7037120 Identity violation in java.net.URI => this also relates to ':' in path
• JDK-6427170 URI needs more static factory methods => the IRI.Builder fixes that RFE
• JDK-5064980 URI compareTo inconsistent with equals for mixed-case escape sequences
• JDK-8214219 URISyntaxException: Illegal character in query at index is thrown for the specified URL address => this is because those URLs contains `unwise` characters. I have verified that they can be parsed by the new IRI.createLenient() factory method we have in the prototype.
• JDK-8214448 URI::resolve broken if the base URI has an empty path => this is fixed in the IRI prototype.

Note: the following issues below should probably be closed as Won't Fix:

• JDK-7012041 java.net.URI can not resolve relative path => this looks like a user error to me and is a limitation of the generic URI syntax which is scheme agnostic.
• JDK-6587780 java.net.URI#normalize method filters out multiple slashes => this is not explicitly defined by the RFCs. I believed it should be listed as a deviation, rather than fixed.
• JDK-8204530 Operation of URLEncoder not congruent with RFC 3986 - URI Generic Syntax => one of the task associated with the prototype is to provide a toolbox for encoding. We should check that what we provide can supersede URLEncoder.
• JDK-6226081 URI should be able to relativize paths with partial roots. => fixing this could return misleading results. It is not sure whether adding a new convenience for this would carry its own weight
.

The following issues still need investigation/fixing

• JDK-8071458 URI handling of IPv6 scopes needs to be re-examined => some characters such as '@' are valid in scope ids. This is not well supported.
• JDK-8051632 java.net.URI.relativize doesn't handle escapes properly => relativise fails to relativize properly if encoded triplet differ only in case (e.g. %4a vs %4A) - and it's unconsistent with URI::equals
• JDK-8188769 NullPointerException in java.net.URI.toASCIIString()
=> toASCCIIString() doesn't handle unmmapable/illegal characters properly.
• JDK-8199396 Parser in java.net.URI incorrectly rejects scope_id with "-"
• JDK-8214423 URI.getQuery(): decoding all percent-encoded octets can prevent the returned string from being correctly interpreted => this should be better explained in the API documentation. Whether some convenience could be provided needs to be examined.
• JDK-8188305 java.net.URI does not parse hosts in an URI according to the latest RFC 1123 => RFC 2396 claims to follows RFC 1123 for DNS names, but the rules defined in RFC 2396 are more strict. Whether URI and IRI should be changed in the manner described in RFC 1123 and accept top-level labels that begin with a digit should be examined.

Prototype 2: Re-wording java.net.URI API documentation, adding new methods for new behaviors

This second solution has been prototyped too. The prototype is in a reasonable shape - but may still require more work. Here is a high level description of the prototype.

Claiming conformance to RFC 3986 / RFC 3987, even with deviations, still requires the addition of some new method in the API in order to make the claim acceptable. This prototype thus comprises the following:

• For a complete list of the changes see the prototype webrev
• The java.net.URI class level API documentation claims conformance to RFC 3986 and RFC 3987, with deviations.
• Existing APIs and their implementation behaviors (constructor, most getters, methods) are left unchanged, but now list deviations to RFC 3986 and RFC 3987.
• URI.getSchemeSpecificPart(), URI.getRawSchemeSpecificPart(), as well as the constructors that accept a schemeSpecificPart parameter are @Deprecated. Scheme specific part no longer exist in the new RFC. This can be seen as a blessing because decoding it when the URI is hierarchical is actually a can of worms, and is the cause of many bugs logged against java.net.URI (see list of issues above)
• New accessors are added (URI.getHostString(), URI.getRawHostString(), URI.getPathString(), URI.getRawPathString()), which allows getHost() and getRawPath() and getPath() to return the same values than before [necessary for backward compatibility]
• A couple of getters - getUserInfo(), getQuery(), and getPort() may now return non null (non -1) results when getHost()/getPath() return null - this is the only incompatible change, and hopefully should not cause regressions. It is due to the new RFC 3986 parsing rules for authority, and the new definition of opaque URIs. It does make 2 JCK tests fail though (1 scenario out of 15 in 1 test, and 1 scenario out of 12 in another one). No existing JDK unit test is failing though.
• All existing constructor and factory methods which are not deprecated document that they construct URIs conforming to the obsoleted RFC 2396, and are tagged obsoleted in their one-liner synopsis.
• New factory methods are added that will throw URISyntaxException for those cases that where accepted before, but which have become illegal with RFC 3986. These URI can still be constructed using the old APIs (old constructors) though.
• Because the old obsoleted constructors & factories allow to construct URIs that are no longer legal, getHostString()/getRawHostString() may sometime throw an IllegalArgumentException wrapping a URISyntaxException. This was chosen over returning null because the authority component is illegal in that case and we don't want that to be confused with no authority.
• The new factories also allow to construct scheme only URLs (like "about:") but still won't allow to construct URIs of the form "//" or "s://". Allowing to build these URIs would be problematic - as for compatibility reasons (especially for compatibility with file: URIs) we want URI.getAuthority() to return null in case of empty authority. Note that older versions of the JDK will not be able to deserialize scheme-only URIs: trying to deserialize such URIs in an older version of the JDK will cause an InvalidObjectException to be thrown. Whether we want to keep this behavior, or prevent construction of scheme-only URIs and list that as an additional deviation should be evaluated.
• For compatibility reasons normalize/resolve/relativize claim conformance to RFC 2396 and list their deviations towards RFC 3986. This means that none of the bugs that have been logged against these methods have been fixed - as trying to fix them in turn would cause incompatible changes that might cause regressions.
• Equals/HashCode remain unchanged - and as a result list new deviations to RFC 3986
• A new HostType enum {IPv4Address, IPv6Address, IPvFuture, DNSRegName, RegName, None} has been added, as well as a static method that will return the corresponding enum given a host string. The host string can be any returned by getHost() getRawHostString() or getHostString(), but only None or DNSRegName can be returned for getHost(), as getHost() behavior remains unchanged.
• New lightweight builders have been added. By default, the encoding performed by the new lightweight builders and new factory methods is idempotent, but a new builder that re-encodes any percent encoded triplets (as the obsoleted constructors used to do) can also be created.
• An @apiNote warning has been added to all decoding getters (getAuthority, getUserInfo, getHostString, getPath, getPathString, getQuery, getFragment) to warn the user that the returned string may contain any characters, including \n or \r.
• The way that java.net.URI deals with non US-ASCII characters has been left unchanged. In particular it doesn't implement the restrictions on non US-ASCII characters as specified by RFC 3987.

Performance and footprint considerations:

• The java.net.URI class now has an additional handful of transient fields: an int parsingstate, a hoststr, a pathstr, a decodedHostString and decodedPathString (one int, 4 strings).
• The parsing according to RFC 3986 as implemented by the new factory methods and builders is maintained in a separate (package protected) RFC3986Parser class (derived from the previous prototype). That separate class should not be needed unless the new factory or builders are called, or host is null, authority is not null, and getUserInfo or getPort are called, or one of the new getPathString, getRawPathString, getHostString, getRawHostString accessors is called, or getHostType is called. This should limit the performance impact of the new implementation on startup and existing code, at least until these get more widely used.

Testing:

• All existing unit tests are passing.
• A new BuilderTest (derived from the previous prototype implementation) has been added to test the new lightweight builder.
• A new RFC3986ParserTest (derived from test/jdk/java/net/IRI/Test.java in the previous prototype implementation) has been added to test the new factories and APIs.

Compatibility:

• Two JCK tests are failing - one because http://:21/ .getPort() returns 21 instead of -1 (one of the incompatibilities mentioned above), one because urn:isbn:096139210?x now has a non null query string ("x").
• Another side effect which has not been picked up by any tests is that "file:?hmm" which was previously opaque is now hierarchical.
• The deprecation of the 3 args constructor and getSchemeSpecificPart()/getRawSchemeSpecificPart() is most probably what would cause the most developer annoyance - as the webrev attests. We could choose to simply obsolete them in the API documentation, but @Deprecated is sending a stronger message: indeed, these notions are 1. confusing WRT to the new RFCs and 2. desperately unfixable WRT to the many confusing issues they are causing (see list of issues above).
• Also note that none of issues above are fixed in that webrev, and that fixing them might bring new incompatibilities.

Notes

Note on per-URI instances and combinatorial complexity in Solution 4

The idea there was to change java.net.URI to delegate everything to a wrapped instance of a package private URIImpl class - and have two implementations - one that is basically a clone of the current java.net.URI, and one other that implements the newer RFCs behaviors. Then we would have the old constructors and factories instantiate an instance of the old implementation, and add new factory methods to create instances of URI wrapping instances of the new implementation. A new public enum/boolean accessor could be added to figure out which flavor of the RFCs an instance of URI conforms to.
This idea was rejected for the following reasons:

• The compatibility risk would not be eliminated: old code that is expecting an instance of URI behaving according to the old behavior might break if it is passed an instance that has the new behavior. For instance, calling getHost() might return a string that is not a DNS host and might even contain \r\n. Old code can't be expected to deal with that properly. Although new code might be written to test which implementation an instance of URI actually encapsulates, and act in consequence, old code would not know how to do that.
• There's a combinatory nightmare whenever a method is specified to work on two instances of URI (e.g., receiver, argument) - then what should the method do if the receiver is of one type and the argument is of another type? For instance - what should equals() do in these cases? and resolve/relativize?
• Specifying a dual behavior in the API documentation of java.net.URI might become quite be arduous, and would increase the difficultly level for developers
• Serialization/Deserialization would be an issue, as additional data would be needed to figure out which implementation class to use at deserialization time.