Sunday, October 05, 2025

HTTP/3 support now available in Java HttpClient for upcoming Java 26 release

Many of you might have heard about the recent Java 25 release. If not, then here's a very good and extensive article on what Java 25 contains https://blogs.oracle.com/java/post/the-arrival-of-java-25.

Today though I will be writing about what's coming in Java 26, which is proposed to be released in March 2026 https://mail.openjdk.org/pipermail/jdk-dev/2025-October/010505.html. Of course, just like Java 25, Java 26 too will have numerous new features, enhancements and bug fixes. As usual early access builds are available and for Java 26 you can find them at https://jdk.java.net/26/

One new feature in Java 26 is that HttpClient, which is part of Java SE since Java 11, will now support HTTP/3 protocol version. Before getting into the details about what the API looks like, let's see what HTTP/3 is about. From a HTTP protocol point of view, it is not much different than HTTP/2. What's significantly different is that unlike HTTP/2 which works using TCP (https://en.wikipedia.org/wiki/Transmission_Control_Protocol), HTTP/3 instead works using UDP (https://en.wikipedia.org/wiki/User_Datagram_Protocol). HTTP/3 protocol is built on top of the (relatively new standard) QUIC protocol. I won't go into the details of the protocol itself (QUIC is vast), but for those who might be interested in these details, JEP 517, through which the HTTP/3 support in HttpClient was integrated has all the details including links to the relevant RFCs https://openjdk.org/jeps/517.

Let's now see how to utilize the HTTP/3 feature of the java.net.http.HttpClient API. If you haven't previously used the Java SE HttpClient then the API doc here https://docs.oracle.com/en/java/javase/25/docs/api/java.net.http/java/net/http/package-summary.html is a good start. Briefly, the application code builds a java.net.http.HttpClient instance and maintains that instance as long as it wishes to (typically throughout the application's lifetime). When issuing HTTP requests, the application code then builds a java.net.http.HttpRequest instance and uses the HttpClient.send(...) method to send the request and obtain a java.net.http.HttpResponse. Some advanced usages, where the application doesn't want to wait until the response arrives, may use the HttpClient.sendAsync(...) method to send the request and obtain a java.util.concurrent.Future instance, which the application can use at a later point to obtain the HttpResponse. The HttpResponse itself has methods which allow the application to obtain the HTTP response code, the HTTP protocol version that was used for that request/response exchange and of course the body of the response. Typical code would look something like:


HttpClient client = HttpClient.newBuilder().build(); // create a HttpClient instance

...

URI reqURI = new URI("https://www.google.com/");

HttpRequest req = HttpRequest.newBuilder().uri(reqURI).build(); // create a request instance

HttpResponse<String> resp = client.send(req, BodyHandlers.ofString(StandardCharsets.UTF_8)); // send the request and obtain the response as a String content

System.out.println("status code: " + resp.statusCode() + " HTTP protocol version: " + resp.version()); // print the response status code and the HTTP protocol version used


None of this is new and this API has been around since Java 11. So let's see what's new in Java 26 and how to enable HTTP/3 usage by the HttpClient.

By default, the HttpClient (even in Java 26) is enabled with HTTP/2 as the preferred HTTP version to use when issuing requests. The default can be overridden per HttpClient instance basis by setting a preferred version of choice. For example:


HttpClient client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build();


will construct a client which will use HTTP/1.1 as the preferred version for all the requests that it issues. Unless the HttpRequest instance is coded to use a different version. For example:


HttpRequest req = HttpRequest.newBuilder().uri(reqURI).version(HttpClient.Version.HTTP_2).build();


In this case the HttpRequest's preferred version (HTTP/2) will be the one that the HttpClient will prefer to use when issuing the request. If the server doesn't support HTTP/2 then the internal implementation of the HttpClient will do the necessary work to downgrade to HTTP/1.1 protocol version for that request and establish a HTTP/1.1 request/response exchange with the server and provide the application with that HTTP/1.1 response.

Again, this too is not new and has been this way in the implementation of HttpClient. What's new in Java 26 is that a new version value has been introduced - HttpClient.Version.HTTP_3 representing the HTTP/3 protocol version. Applications can opt in to use HTTP/3 protocol version by either setting it as a preferred version at the HttpClient instance level:


HttpClient client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_3).build();


or on specific HttpRequest instances:


HttpRequest req = HttpRequest.newBuilder().uri(reqURI).version(HttpClient.Version.HTTP_3).build();


In either case, when HTTP_3 version is set as the preferred version, the HttpClient implementation will attempt to establish a UDP based communication (remember HTTP/3 is a UDP based protocol) with the server to which the request is targeted. If that UDP based QUIC connection attempt fails (because the server may not support HTTP/3) or the connection attempt over UDP doesn't complete in a timely fashion (datagram delivery is unreliable), then just like for HTTP/2, the internal implementation of the HttpClient will downgrade the protocol version to use HTTP/2 (remember, over TCP) and try and complete that request using HTTP/2 version (and if the server doesn't support HTTP/2 then like before the request will be internally downgraded to HTTP/1.1 version).

So, the application code would look similar to what we saw previously with just one change, where the preferred version is set as HTTP/3 either when building the HttpClient instance or when building the HttpRequest instance:


HttpClient client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_3).build(); // create a HttpClient instance with HTTP/3 as the preferred version

...

URI reqURI = new URI("https://www.google.com/");

HttpRequest req = HttpRequest.newBuilder().uri(reqURI).build(); // create a request instance

HttpResponse<String> resp = client.send(req, BodyHandlers.ofString(StandardCharsets.UTF_8)); // send the request and obtain the response as a String content

System.out.println("status code: " + resp.statusCode() + " HTTP protocol version: " + resp.version()); // print the response status code and the HTTP protocol version used


Do note that setting HTTP/3 as the preferred version doesn't necessary mean that the request will mandatorily use HTTP/3 as the protocol version. That's why it's called the preferred version. The HttpClient will not have the necessary knowledge to be sure that HTTP/3 is indeed supported by the server against which the request is issued. So, especially for the first request against that server, the HttpClient instance will use an internal implementation specific algorithm which involves an attempt to establish a TCP based communication (HTTP/2) or a UDP based communication (HTTP/3) against that server. Whichever succeeds first will be used as the communication mode and thus decides which HTTP protocol version is used for that request.

There are a few other details to the HTTP/3 version discovery. I won't go into those for now, but the API documentation of java.net.http.HttpOption.Http3DiscoveryMode has some of those details https://download.java.net/java/early_access/jdk26/docs/api/java.net.http/java/net/http/HttpOption.Http3DiscoveryMode.html.

With that background, let's now consider a few specific cases and some code examples demonstrating the usage. Let's consider the case where the application wants to force the use of HTTP/3 - i.e. try and communicate with the server only through QUIC (i.e. over UDP) and then issue the HTTP/3 request. If that fails, then don't attempt to use HTTP/2 for that request. Applications would typically do this only when they are absolutely certain that the target server (represented by the host and port used in the request URI) supports HTTP/3 over that host/port combination. Let's for this example consider "google.com" as the server against which we will issue that request. (Based on prior experiments) We know that "google.com" supports HTTP/3 at the same host/port where it support HTTP/2 (or HTTP/1.1). Here's what the code would look like in this case:


import java.net.http.HttpClient;

import java.net.http.HttpOption;

import java.net.http.HttpOption.Http3DiscoveryMode;

import java.net.http.HttpClient.Version;

import java.net.http.HttpRequest;

import java.net.http.HttpResponse.BodyHandlers;

import java.net.http.HttpResponse;

import java.net.URI;

import java.nio.charset.StandardCharsets;

...

HttpClient client = HttpClient.newBuilder()

                     .version(Version.HTTP_3) // configure HTTP/3 as the preferred version of the client

                     .build();

URI reqURI = new URI("https://www.google.com/");

HttpRequest req = HttpRequest.newBuilder()

                   .uri(reqURI)

                   .setOption(HttpOption.H3_DISCOVERY, Http3DiscoveryMode.HTTP_3_URI_ONLY) // enforce that only HTTP/3 be used

                   .build();

HttpResponse<String> resp = client.send(req, BodyHandlers.ofString(StandardCharsets.UTF_8)); // send the request and obtain the response as a String content

System.out.println("status code: " + resp.statusCode() + " HTTP protocol version: " + resp.version()); // print the response status code and the HTTP protocol version used


Apart from configuring the HttpClient instance with version(Version.HTTP_3) as the preferred version, the other important detail in this code is the line where we configure the HttpRequest with:


setOption(HttpOption.H3_DISCOVERY, Http3DiscoveryMode.HTTP_3_URI_ONLY) // enforce that only HTTP/3 be used


The H3_DISCOVERY option with a value of Http3DiscoveryMode.HTTP_3_URI_ONLY instructs the HttpClient instance that this request must only use HTTP/3 as the protocol version and if it's not possible to use that version, then the request should fail (with an exception). Like previously noted, we know that google at "https://www.google.com/" supports HTTP/3. So we know for a certainty that this request should succeed with HTTP/3. The complete code for this demo is here:


import java.net.http.HttpClient;

import java.net.http.HttpOption;

import java.net.http.HttpOption.Http3DiscoveryMode;

import java.net.http.HttpClient.Version;

import java.net.http.HttpRequest;

import java.net.http.HttpResponse.BodyHandlers;

import java.net.http.HttpResponse;

import java.net.URI;

import java.nio.charset.StandardCharsets;


public class Http3Usage {

    public static void main(final String[] args) throws Exception {

        final boolean printRespHeaders = args.length == 1 && args[0].equals("--print-response-headers");

        try (final HttpClient client = HttpClient.newBuilder()

                                         .version(Version.HTTP_3)

                                         .build()) {


            final URI reqURI = new URI("https://www.google.com/");

            final HttpRequest req = HttpRequest.newBuilder()

                                       .uri(reqURI)

                                       .setOption(HttpOption.H3_DISCOVERY, Http3DiscoveryMode.HTTP_3_URI_ONLY)

                                       .build();


            System.out.println("issuing first request: " + req);

            final HttpResponse<String> firstResp = client.send(req, BodyHandlers.ofString(StandardCharsets.UTF_8));

            System.out.println("received response, status code: " + firstResp.statusCode() + " HTTP protocol version used: " + firstResp.version());

            if (printRespHeaders) {

                System.out.println("response headers: ");

                firstResp.headers().map().entrySet().forEach((e) -> System.out.println(e));

            }

        }

    }

}


When you use Java 26 early access build and run this as:


java Http3Usage.java


you should see the following output:


issuing first request: https://www.google.com/ GET

received response, status code: 200 HTTP protocol version used: HTTP_3


Notice that the response's protocol version is HTTP_3, so it did indeed use HTTP/3 as the protocol version for that request.

At this point we know for certain (and even demonstrated) that the request to "https://www.google.com/" support HTTP/3 and so the HttpRequest can enforce the use of HTTP/3. Let's see what would have happened if we had not instructed the HttpClient to enforce the HTTP/3 usage. So let's remove (or comment out) this line from the above code and keep the rest of the code as-is:


.setOption(HttpOption.H3_DISCOVERY, Http3DiscoveryMode.HTTP_3_URI_ONLY)


When you do that and run that program again using "java Http3Usage.java" you should see the following output:


issuing first request: https://www.google.com/ GET

received response, status code: 200 HTTP protocol version used: HTTP_2


Notice the difference in the response's protocol version. The request/response exchange used HTTP_2 when HTTP/3 was the preferred version but not enforced for this request. As noted previously, this is understandable because the HttpClient instance that we used to issue this request can't say for sure if the server at that port/host does indeed support the "preferred" HTTP/3 version. So the HttpClient implementation used an internal algorithm which ended up establishing a TCP based connection first and thus using it to issue the HTTP/2 request.

Moving one step further with this example - some of you might ask whether over time, the HttpClient is capable of gaining knowledge that a server residing at host/port combination supports HTTP/3. The answer is yes. There are several ways this can happen. In fact, there's a standard called "HTTP Alternative Services" (RFC-7838 https://datatracker.ietf.org/doc/html/rfc7838). HTTP Alternative Services (we will refer to it as Alt-Services to keep it short) is a standard way for servers to advertize alternative services that are supported by that server. The RFC states the different ways it can be done. One way is for the server to send back a HTTP response header named "alt-svc" in response to some HTTP request. This header's value is then expected to contain details about the Alt-Services that the server supports and the host/port combination where it is supported. For example, an "alt-svc" header of the following form:


alt-svc=h3=":443"


implies that the server at the host, which was used for the HTTP request, at port 443 supports HTTP/3 protocol (h3 represents an ALPN for HTTP/3 support). When such a response header is advertized by the server, the HttpClient recognizes this as a standard header and makes note of this detail. So the next time a request is issued against that same server host/port combination then it will look into its internal registry to see if an "h3" Alt-Service was advertized for that server and if so, then will try to establish the HTTP/3 request against that alternate host/port combination.

Let's see if we can see this in action, in our demo code. Like previously, we will configure the HttpClient instance with HTTP/3 as the preferred version. We won't enforce HTTP/3 on the HttpRequest because that's not what we want now. Then, we will send 2 requests using the same HttpClient instance against the same google URI and see how it behaves. Here's the code:


import java.net.http.HttpClient;

import java.net.http.HttpOption;

import java.net.http.HttpOption.Http3DiscoveryMode;

import java.net.http.HttpClient.Version;

import java.net.http.HttpRequest;

import java.net.http.HttpResponse.BodyHandlers;

import java.net.http.HttpResponse;

import java.net.URI;

import java.nio.charset.StandardCharsets;


public class Http3Usage {

    public static void main(final String[] args) throws Exception {

        final boolean printRespHeaders = args.length == 1 && args[0].equals("--print-response-headers");

        try (final HttpClient client = HttpClient.newBuilder()

                                         .version(Version.HTTP_3)

                                         .build()) {


            final URI reqURI = new URI("https://www.google.com/");

            final HttpRequest req = HttpRequest.newBuilder()

                                       .uri(reqURI)

                                       .build();


            System.out.println("issuing first request: " + req);

            final HttpResponse<String> firstResp = client.send(req, BodyHandlers.ofString(StandardCharsets.UTF_8));

            System.out.println("received response, status code: " + firstResp.statusCode() + " HTTP protocol version used: " + firstResp.version());

            if (printRespHeaders) {

                System.out.println("response headers: ");

                firstResp.headers().map().entrySet().forEach((e) -> System.out.println(e));

            }


            System.out.println("issuing second request: " + req);

            final HttpResponse<String> secondResp = client.send(req, BodyHandlers.ofString(StandardCharsets.UTF_8));

            System.out.println("received response, status code: " + secondResp.statusCode() + " HTTP protocol version used: " + secondResp.version());

            if (printRespHeaders) {

                System.out.println("response headers: ");

                secondResp.headers().map().entrySet().forEach((e) -> System.out.println(e));

            }

        }

    }

}


With Java 26 early access release, let's run it again as:


java Http3Usage.java


This should now print the following output:


issuing first request: https://www.google.com/ GET

received response, status code: 200 HTTP protocol version used: HTTP_2

issuing second request: https://www.google.com/ GET

received response, status code: 200 HTTP protocol version used: HTTP_3


Notice how the first request used HTTP/2 and the second request against the same request URI using the same HttpClient instance used the preferred HTTP/3 version. That demonstrates that the HttpClient instance is capable of gaining knowledge that a server at a particular host/port supports HTTP/3, and then is capable of using that knowledge to issue HTTP/3 requests against that server, if the application prefers that protocol version.

We talked about Alt-Services being advertized by servers, as response headers. Since we already have the code here which has access to the HttpResponse, let's see if google did indeed advertize the "h3" Alt-Service as a response header. The above code is already equipped to print the response headers if the application is launched with the "--print-response-headers" program argument. So let's do that:


java Http3Usage.java --print-response-headers


Like previously, you will notice that the first request/response exchange ended up using HTTP/2 and the second request/response exchange used HTTP/3. Additionally you will see a lot more lines in the output, these are all the response headers that we printed from the HttpResponse. In those lines, if you search for "alt-svc" you should find:


alt-svc=[h3=":443"; ma=2592000,h3-29=":443"; ma=2592000]


So google responded to the requests and included this additional response header advertizing the Alt-Service representing HTTP/3 support.

There are a few other ways the HttpClient instance gains knowledge of HTTP/3 support but that's not too important for now.

The support for HTTP/3 in Java's HttpClient has been integrated into the mainline JDK repo and is available in the early access builds of Java 26. The early access builds are available for download at https://jdk.java.net/26/. Although the API enhancement from the usage side looks very trivial (and that's intentional), the work to support QUIC and then HTTP/3 on top of QUIC is the result of several years of work in the JDK. New tests have been added and extensive manual testing has also been carried out to address any issues. However, the implementation is new and hasn't seen much usage outside of the people involved in developing this implementation. If the support for HTTP/3 in JDK's HttpClient is of interest to you, then it will be very valuable if you run your applications/experiments using an early access build of Java 26 (https://jdk.java.net/26/) and provide any feedback/bug reports. JEP 517 has all the necessary details about this feature https://openjdk.org/jeps/517 and if you have any feedback about the usage of this feature, then please subscribe to net-dev OpenJDK mailing list https://mail.openjdk.org/mailman/listinfo/net-dev and start a discussion with the relevant details.


Boot a custom (MacOS) kernel on MacOS Apple Silicon

The past several months I've been attempting to debug some (Java related) issues on macos. Most of these issues relate to the syscalls or native function calls that the JDK does on macos in order to provide implementation for some of the Java SE APIs. Naturally, debugging those issues means looking at macos specific system/kernel logs and sometimes even the macos kernel code. In some cases even that isn't enough.

Several parts (but not all) of macos kernel source is available as opensource code at https://opensource.apple.com/releases/. For each macos release, Apple updates that page with the relevant kernel versions which allows you to download the source or check out relevant tag from the linked GitHub repo. For example, here's the XNU repo https://github.com/apple-oss-distributions/xnu which represents parts of the kernel code that runs macos.

Given this, for debugging some of the issues, I've been looking to build the XNU kernel code and then have macos boot with that custom built kernel. That's anyway the end goal. It isn't easy though, because although there are build instructions on how to do it, those aren't always up to date. Plus, I'm told that it requires the exact set of tools and exact versions of those tools to be able to locally build a kernel and then have macos succesfully boot using that built kernel. I haven't yet managed to accomplish that.

For now though, I'll focus on a slightly different but related topic. For each release of macos, Apple provides a Kernel Debug Kit (KDK). Among other things, each KDK consists of pre-built kernels for different models of the macos hardware. Furthermore, for each of these macos models, the KDK has different variants of the kernel - "release" variant, "development" variant and "debug" variant. I haven't found an official document which states what exactly is different between each variant (or how exactly they are built), but from what I have read so far, the "release" variant is the one that matches the kernel that runs "production" macos. "development" and "debug" on the other hand, I'm guessing, are built using different flags to allow kernel developers and kernel extension developers to experiment with the kernel or kernel extensions.

Given that the KDK ships these pre-built kernels, it then means that I should be able to skip the part where I was considering building the XNU project and just jump to the step where I somehow point macos to use the kernel binary that's present in the KDK instead of the one it uses by default to boot macos. Unfortunately, there's no official documentation which explains how to do it. I have been looking for some hints and articles to find a way to boot a custom kernel binary and some of the articles I read had instructions that no longer worked. So I finally asked in the macos developer forum https://developer.apple.com/forums/thread/801605. A big thank you to Kevin for not just responding but also providing additional guidance on how to get this working.

In the rest of this article, I'll summarize the steps that I followed to have macos boot a kernel binary that's shipped in the Apple provided KDK. Do note that these steps are only applicable for Apple Silicon based macos systems (i.e. macos ARM64 architecture) and aren't applicable for Intel based macos systems.

Also note that these steps have been influenced heavily by the inputs provided in that developer forum thread as well as the following 2 articles:

https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html

and

https://khronokernel.com/macos/2023/08/08/AS-VM.html

The steps I note here are essentially a working and trimmed down version of what's explained in those articles.

It's very important to first know which KDK is relevant for use. Each release of XNU kernel has a corresponding KDK version. First, we will see what macos version we are currently running on. Note that macos OS version and XNU kernel version are not the same. Each has a different value.

In order to find out the macos version, run the "sw_vers" command:

sw_vers

ProductName:        macOS
ProductVersion:        15.6
BuildVersion:        24G84

So we are on 15.6 version of macos. Next we go to https://developer.apple.com/download/all and sign in with an Apple developer account. This page will list various products including the Kernel Debug Kit (KDK) for various versions of macos. We will locate the KDK for 15.6 of macos here. You will find it listed as "Kernel Debug Kit 15.6 build 24G84" (the build matches with the output we saw for sw_vers). Download the linked ".dmg" file from there and once downloaded, install it (like you would install any other dmg file). Upon installation, this version of the KDK will end up under the "/Library/Developer/KDKs/" directory on the filesystem. This specific KDK installation will be at "/Library/Developer/KDKs/KDK_15.6_24G84.kdk".

Let's set an environment variable named "KDK" to point to this location and for the rest of this article we will refer to that environment variable.

export KDK=/Library/Developer/KDKs/KDK_15.6_24G84.kdk

Next, let's list the kernel binaries that are shipped in this KDK. They will be present under the "$KDK/System/Library/Kernels/" directory. So:

ls $KDK/System/Library/Kernels/

This will print out several files, like:

kernel                kernel.kasan.t8112.dSYM
kernel.development        kernel.kasan.t8122
kernel.development.t6000    kernel.kasan.t8122.dSYM
kernel.development.t6020    kernel.kasan.t8132
kernel.development.t6030    kernel.kasan.t8132.dSYM
kernel.development.t6031    kernel.kasan.vmapple
kernel.development.t6041    kernel.kasan.vmapple.dSYM
kernel.development.t8103    kernel.release.t6000
kernel.development.t8112    kernel.release.t6000.dSYM
kernel.development.t8122    kernel.release.t6020
kernel.development.t8132    kernel.release.t6020.dSYM
kernel.development.vmapple    kernel.release.t6030
...


Each of these are either kernel binaries or debug symbols for the kernel binaries. And each of these binaries are for a different model of macos - the text "t8122", "t8132" and such is the macos model name.

So at this point, we have identified the macos operating system version we are on, then downloaded the corresponding KDK version for that macos version and installed it. The next step is to identify the macos model of our system. To do that we use the "uname -v" command:

uname -v

This prints:

Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:30 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6020

In the above output you will notice "RELEASE_ARM64_T6020". The T6020 is the macos model of this system and that's the one we are interested in. Note that the output will be different depending on which macos model you are running on and it is very important to make note of the exact model from the output.

Now that we know the model of our macos, let's again list the contents of our KDK installation and this time "grep" only for this T6020 model:

ls $KDK/System/Library/Kernels/ | grep -i T6020

This prints:

kernel.development.t6020
kernel.kasan.t6020
kernel.kasan.t6020.dSYM
kernel.release.t6020
kernel.release.t6020.dSYM

You will notice that there are 3 variants of the kernel available for this macos model in this KDK installation - "development", "kasan" and "release". Furthermore, the "release" and "kasan" variants each have a separate debug symbols file. For the "development" variant, there isn't a debug symbols file and I'm guessing it's because the "development" variant is compiled with debug symbols enabled within the binary. But that's merely a guess, like I noted previously, I haven't yet found the exact details of what each of these variants represent or how they are compiled. Curiously, there isn't a "debug" variant. There is a "kasan" variant, but I hadn't heard about that one before and I don't know what it is. But that's not important for now since our experiment is merely to boot a custom kernel binary instead of the default one. For this experiment, we will choose the "development" variant as the kernel binary we want to boot, so the "$KDK/System/Library/Kernels/kernel.development.t6020" file.

At this point, we have everything available locally to start the steps that are necessary to initiate the boot. First thing we need to do is create a "kext collection" using this kernel binary. To do that we use the "kmutil create" command ("kmutil" is a tool that's by default shipped in macos). Before doing that, I will create a directory on my filesystem where I would like to have this "kext collection" generated. This will be the file which will eventually be used by macos to boot the system. You can specify a location of your choice, and I chose "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/" as the directory:

mkdir -p /Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/

Next we will use the "kmutil create" command to create the kext collection in this directory. The following command is borrowed from the articles I linked previously:

kmutil create \
        --arch arm64e \
        --no-authorization \
        --variant-suffix development \
        --new boot \
        --boot-path /Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc \
        --kernel $KDK/System/Library/Kernels/kernel.development.t6020 \
        --repository $KDK/System/Library/Extensions \
        --repository /System/Library/Extensions \
        --repository /System/Library/DriverExtensions \
        --explicit-only $(kmutil inspect -V release --no-header | grep -v "SEPHiber" | awk '{print " -b "$1; }')



Except for the "--variant-suffix", "--boot-path" and the "--kernel" options, the rest of them will usually remain the same if you had to run it on your Apple Silicon based macos system.

We are choosing the "development" variant of the kernel, so I use "development" as the "--variant-suffix" option's value.

For the "--boot-path", I provide "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc" as the value. Notice that "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/" is the directory we just created a few lines above to use it as a destination directory for this generated "kext collection". Also notice that for this "--boot-path" option value, I use "macos-15.6.kc" as the file name. It can be anything, I chose this value to be clear what version of macos it corresponds to. When "kmutil create" successfully completes and creates the "kext collection", you will find a file named "macos-15.6.kc.development" where "development", in that file name, is the value you provided for the "--variant-suffix" option.

For the "--kernel" option, we point to the "$KDK/System/Library/Kernels/kernel.development.t6020" kernel file that we previously listed and identified for the current macos model, under the KDK installation directory.

The rest of the options I left as-is.

When you run this command and after it completes successfully, you should find the generated "kext collection" file at "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development" (or whatever option values you provided for "--boot-path" and "--variant-suffix").

At this point we now have a bootable file for macos boot process to use. Our next step is to configure macos to use this file. To do this we will first have to boot macos in recovery mode. So we first shutdown macos and then start macos in recovery mode by pressing down and holding the "power" button. When you do that, you will see a message which says macos is loading options. Continue to hold down the "power" button. This is a standard way to boot into recovery mode in macos, so I won't go into this detail. Once it boots in recovery mode, you will see a screen which has a "Options" icon. Click on it and then it will show a "Continue" button. Click on the "Continue" button. That will then show a screen with various menu options (at the top). Choose the "Utilities" -> "Terminal" menu option. It will open the terminal window.

On the terminal window, we will first disable System Integrity Protection (SIP) using the "csrutil" command:

csrutil disable

Confirm the action when prompted and provide the user/password if prompted.

Then we will run a command which will relax the restriction which prevents macos from booting arbitrary kernel binaries. We use the "bputil" command as follows:

bputil --disable-boot-args-restriction

Confirm the action when prompted and provide the user/password if prompted.

And now the last step where we configure macos to boot the kext collection file that we generated previously at "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development". We use the "kmutil configure-boot" command for that:

kmutil configure-boot --volume "/Volumes/Macintosh HD" --custom-boot-object "/Volumes/Macintosh HD/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development"

The "--volume" options points to the target volume which is being configured. Typically, there's only one volume on the system and usually it's named "Macintosh HD", so the path is almost always "/Volumes/Macintosh HD". But if you have something different or if you have more than one volume on your system, then choose the appropriate value for this option.

The "--custom-boot-object" is the full path to the kext collection file that we created using the kernel binary shipped in the KDK. The full path includes the volume on which that kext collection file resides. So "/Volumes/Macintosh HD/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development".

It's important to note that the volume name and the path name may have space characters in their name. So remember to enclose these values in double quotes (like in the above example) or use appropriate escaping using the backslash (\) character.

Also note that the terminal window on which you type these commands has a very small font (at least on my instance) so it isn't easy to type in this lengthy command/text there. This became a bit of hassle when I was experimenting with these commands. So before booting into recovery mode, I created a ".sh" file beforehand with the complete "kmutil configure-boot" command and saved it at the "/Users/me/configure-boot.sh". Then instead of typing that "kmutil configure-boot" command manually, I ran:

bash "/Volumes/Macintosh HD/Users/me/configure-boot.sh"

from the terminal in the recovery mode. To be clear, creating this script file and using it is completely optionally. If you prefer typing the "kmutil configure-boot" command, that's fine too.

At this point, after the "kmutil configure-boot" completes successfully, all the steps needed to boot this custom built kernel are now done. Next, we shutdown and restart the system as usual (there's the Shutdown/restart option in the top menu).

When the system starts next, it will use the "macos-15.6.kc.development" binary that we built. To verify that it indeed is using this custom kernel binary, we can run the following command from the terminal once we have logged in to the system:

sysctl kern.osbuildconfig

This should output:

kern.osbuildconfig: development

which is the variant we built and configured to boot. This implies that we have successfully booted a custom variant of the kernel that is shipped in the KDK.

Do note that, sometimes, booting of this custom kernel binary may not be successful. When that happens, due to issues with the binary or some other issue that causes the boot process to fail, then during the boot you will see the Apple icon and it will stay up for a while before the system reboots and this will keep happening in a (never ending) loop. That's a sign that something has gone wrong and you won't be able to use this custom kernel binary. So to restore back to your normal kernel binary, you will have to reboot into recovery mode, like you did the previous time, by holding down the power button and waiting for the screen to show the "Options" screen and then clicking "Continue" button. Once you are into recovery mode, this time you will choose the "Utilities" -> "Startup Security Utility" menu option. The screen that then shows up will have 3 options and would have selected the "Permissive" mode selected (because we ran the csrutil and the bputil commands previously). Since the custom kernel binary that we have configured for the boot is not functioning, we will have to select the "Restricted" mode in that screen options. Choosing "Restricted" mode will prevent the use of the custom kernel binary and instead will use the one shipped by macos by default, when we boot next (and that's what we want). Save/Submit/Continue with the "Restricted" mode and, if prompted, provide the user/password to apply the change. After that's done, Shutdown/restart from the top menu option and let the system boot in normal mode. This will boot you in the release version of the kernel that macos ships by default and you can verify that by running the following command from the terminal once you have logged in to the system:

sysctl kern.osbuildconfig

That will output:

kern.osbuildconfig: release

That completes the steps needed to boot a custom kernel variant on macos and the step to recover if that process fails. Do note that we did not build the XNU project from source and instead reused a previously built kernel variant shipped in the KDK. Building the XNU from source (with trivial code modifications for experiment/debugging) is still on my TODO list but I am told that it isn't straightforward nor guaranteed to work. So that's for some other time.

Thursday, October 05, 2023

Using JAXB in custom Ant tasks on recent Java versions

Apache Ant 1.10.14 was released a few weeks ago https://lists.apache.org/thread/9vhk51nkw9wjxzm7xk2q9xm6s803prmr. Like noted in that announcement, apart from the regular bug fixes, this release also has an important change which allows it to be used in the recently released Java 21 https://inside.java/2023/09/19/the-arrival-of-java-21/. One major goal of Ant project is to make sure that it can be used to build projects using latest versions of Java. As such, the Ant project team keeps a watch on any changes in the Java releases that could affect Ant and does necessary changes in Ant and releases them in its 1.10.x release series.

The announcement of 1.10.14 and this changelog https://github.com/apache/ant/blob/rel/1.10.14/WHATSNEW contains the details of what’s exactly changed in this release, so I won’t go into those details again. In this post I’ll however go through one interesting issue that was brought to my notice by more than one projects, when using recent versions of Java. The issue specifically relates to custom Ant tasks that use JAXB. However, in general, it applies to some of the APIs that have been removed from recent versions of Java (details in https://openjdk.org/jeps/320)

JAXB as noted in the reference doc https://docs.oracle.com/javase/tutorial/jaxb/intro/arch.html is an API and implementation for XML binding. Up until Java 11, JAXB API and implementation classes were shipped as part of the Java runtime. What it meant was that any application code, like custom developed Ant tasks, could just use the JAXB APIs in their code without having to explicitly specific any external library dependency. In Java 11, JAXB along with few other modules was removed from the JDK. The release notes of JDK 11 lists this change https://www.oracle.com/java/technologies/javase/11-relnote-issues.html#JDK-8190378. Additionally, JEP-320 https://openjdk.org/jeps/320 has all the details related to this removal. When using JAXB APIs,  the usage of these modules from the JDK was transparent to the application. So although the application may not explicitly have referred to these module names, it was still reliant on them because they were providing public APIs which the application had references to. Effectively, if a project was using some Ant task which used JAXB, then those projects when they switch to any Java version >= 11 will now start seeing issues due to missing compile and runtime dependency on JAXB. Let’s now consider a simple custom Ant task and see what kind of errors it might encounter. But if you are just interested in the shorter answer and some sample code to fix these classloading issues with JAXB, then please check the end of this article, starting here. For the complete details, please read on.

We will use a trivial custom Ant task implemented by a class called org.myapp.HelloTask, which looks like:


package org.myapp;

import org.apache.tools.ant.Task;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;

public class HelloTask extends Task {

    private String message;

    public void setMessage(final String message) {
        this.message = message;
    }

    @Override
    public void execute() {
        try {
            final JAXBContext context = JAXBContext.newInstance(DataContainer.class);
            System.out.println("Created JAXB context " + context);
        } catch (JAXBException e) {
            throw new RuntimeException(e);
        }

        System.out.println(this.message);
    }

    private static class DataContainer {
        private String data;
    }
}

The HelloTask is just here for demonstration and in practice doesn’t provide any real value. This task allows for a message attribute to be specified. Furthermore, in its execute() method it creates a javax.xml.bind.JAXBContext for the application specific DataContainer class and just prints that context. Additionally, it also prints the message that was set when the task is launched. Let’s assume that this task has been compiled and packaged as a jar file and made available to the build process. Now consider this trivial build.xml file which declares this task and then launches it:


<project default="invoke-task">
    <property name="build.dir" value="build"/>
    <property name="jar.name" value="hellotask.jar"/>

    <taskdef name="hello" classname="org.myapp.HelloTask">
        <classpath>
            <pathelement location="${build.dir}/${jar.name}"/>
        </classpath>    
    </taskdef>  

    <target name="invoke-task" description="invokes the HelloTask">
        <hello message="hello world"/>
    </target>       

</project>

There’s not much in this build.xml. All it does is, use a taskdef to define the HelloTask with the classpath containing (just the) jar file containing the HelloTask. Then in the invoke-task target we launch this hello task by passing it a message.

Let’s now run this build against Java 8:

export JAVA_HOME=<path-to-JDK-8>
ant invoke-task

When you run this (on Java 8) you should see the output similar to:


invoke-task:
    [hello] Created JAXB context jar:file:/<path-to-jdk-8>/jre/lib/rt.jar!/com/sun/xml/internal/bind/v2/runtime/JAXBContextImpl.class Build-Id: ...
    [hello] Classes known to this context:
    [hello]   [B
    [hello]   boolean
    [hello]   byte
    [hello]   char
    [hello]   com.sun.xml.internal.bind.api.CompositeStructure
    [hello]   double
    [hello]   float
    [hello]   int
...
    [hello]   long
    [hello]   org.myapp.HelloTask$DataContainer
    [hello]   short
    [hello]   void
    [hello] 
    [hello] hello world

You’ll see that the JAXB usage in the task was successful and the build completed successfully and we didn’t have to configure any classpath to include any JAXB jar files. As noted previously, this is because the Java 8 runtime ships with the relevant JAXB API and implementation classes.

Now, without changing any code in the task or in the build.xml, let’s just switch to a recent Java version. Let’s say Java 17 and run the build:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

When you do this, you will now see:

BUILD FAILED
build.xml:5: taskdef A class needed by class org.myapp.HelloTask cannot be found: javax/xml/bind/JAXBException
 using the classloader AntClassLoader[build/hellotask.jar]

You’ll notice that the build now fails and the error message states that the class javax/xml/bind/JAXBException cannot be found in the classpath which as defined in the build.xml only included the hellotask.jar (which just has the org.myapp.HelloTask). As noted previously, this is because the JAXB API and implementation is no longer shipped in the JDK. Applications, like this project, are expected to now include the JAXB API and implementation jars in the application classpath. JEP-320 https://openjdk.org/jeps/320 lists the potential Maven co-ordinates to find such jar(s). In case of JAXB it suggests the JAXB reference implementation available in Maven central repo com.sun.xml.bind:jaxb-ri as a potential candidate. Do note that, just like several other Java EE APIs and implementations, JAXB too has several vendors which implement the JAXB API. A “reference implementation” as the name states is meant to demonstrate the implementation of the specified API. There can, and are, several other vendor implementations for such APIs. It’s upto the applications to choose the ones that they desire to use. In this demonstration, we will use the 2.3.8 version of com.sun.xml.bind:jaxb-ri dependency (available at https://repo.maven.apache.org/maven2/com/sun/xml/bind/jaxb-ri/2.3.8/). There’s no specific reason for my choice of this specific version - it’s only for demo.

Now that we have this dependency made available, let’s include it in the classpath of the taskdef of HelloTask and our build.xml (snippet) will now look like:

...
    <taskdef name="hello" classname="org.myapp.HelloTask">
        <classpath>
            <pathelement location="${build.dir}/${jar.name}"/>
            <!-- JAXB dependencies -->
            <pathelement location="${lib.dir}/jakarta.activation.jar"/>
            <pathelement location="${lib.dir}/jakarta.xml.bind-api.jar"/>
            <pathelement location="${lib.dir}/jaxb-impl.jar"/>
        </classpath>    
    </taskdef>  
...

You’ll notice that the classpath of the taskdef now includes the JAXB related dependency jars. Now let’s rerun the build on Java 17, like previously:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

When you run this, you will now see something that starts like this:


BUILD FAILED
build.xml:17: java.lang.RuntimeException: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.bind.v2.ContextFactory]
    at org.myapp.HelloTask.execute(Unknown Source)
    

So the build still fails, but unlike previously where it had failed when trying to define the HelloTask itself, this time it fails in the execute() implementation of the HelloTask.

So why does it fail even with the JAXB jars in the classpath. This has to do with JAXB (and several other Java EE APIs), which rely on thread context classloader. Several of these APIs, including this call to:

final JAXBContext context = JAXBContext.newInstance(DataContainer.class);

relies on thread context classloader to find the JAXB related classes and resources. It expects the thread context classloader, whichever it is, to be able to load these JAXB classes. Thread context classloaders are specific to the thread that is currently executing the code. So let’s quickly see which classloaders are in play in the execute() method of the HelloTask. To see that, let’s add some trivial debug messages in the code, whose snippet will now look like:


    @Override
    public void execute() {
        System.out.println(HelloTask.class + " was loaded by classloader: " + HelloTask.class.getClassLoader());
        final ClassLoader tccl = Thread.currentThread().getContextClassLoader();
        System.out.println("Context classloader of current thread is " + tccl);
        if (tccl instanceof java.net.URLClassLoader) {
            // let's additionally print the classpath of the URLClassLoader
            final java.net.URL[] classpath = ((java.net.URLClassLoader) tccl).getURLs();
            System.out.println("Context classloader's classpath is " + java.util.Arrays.toString(classpath));
        }
        try {
            final JAXBContext context = JAXBContext.newInstance(DataContainer.class);
            System.out.println("Created JAXB context " + context);
        } catch (JAXBException e) {
            throw new RuntimeException(e);
        }

        System.out.println(this.message);
    }

So what we have done here is that added a few System.out.println messages which prints the classloader which loaded the HelloTask and also prints the current thread’s context classloader. Additionally if the context classloader is a java.net.URLClassLoader, we even print the classpath used by the URLClassLoader. The goal of these debug messages is to see what classloaders are in play when using that JAXB API. Let’s rerun the build again on Java 17 - it’s still expected to fail like previously, but this time we should see these debug messages:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

This fails with the same exception stacktrace as previously, but this time you should also see, something like:


[hello] class org.myapp.HelloTask was loaded by classloader: AntClassLoader[build/hellotask.jar:lib/jakarta.activation.jar:lib/jakarta.xml.bind-api.jar:lib/jaxb-impl.jar]
[hello] Context classloader of current thread is java.net.URLClassLoader@6d6f6e28
[hello] Context classloader's classpath is [file:/apache-ant-1.10.14/lib/ant-commons-net.jar, file:/apache-ant-1.10.14/lib/ant-xz.jar, file:/apache-ant-1.10.14/lib/ant-junit4.jar, file:/apache-ant-1.10.14/lib/ant-jai.jar, file:/apache-ant-1.10.14/lib/ant-apache-resolver.jar, file:/apache-ant-1.10.14/lib/ant-jdepend.jar, file:/apache-ant-1.10.14/lib/ant-apache-regexp.jar, file:/apache-ant-1.10.14/lib/ant-apache-log4j.jar, file:/apache-ant-1.10.14/lib/ant-javamail.jar, file:/apache-ant-1.10.14/lib/ant-apache-bcel.jar, file:/apache-ant-1.10.14/lib/ant.jar, file:/apache-ant-1.10.14/lib/ant-netrexx.jar, file:/apache-ant-1.10.14/lib/ant-swing.jar, file:/apache-ant-1.10.14/lib/ant-jsch.jar, file:/apache-ant-1.10.14/lib/ant-junitlauncher.jar, file:/apache-ant-1.10.14/lib/ant-jakartamail.jar, file:/apache-ant-1.10.14/lib/ant-junit.jar, file:/apache-ant-1.10.14/lib/ant-imageio.jar, file:/apache-ant-1.10.14/lib/ant-launcher.jar, file:/apache-ant-1.10.14/lib/ant-antlr.jar, file:/apache-ant-1.10.14/lib/ant-testutil.jar, file:/apache-ant-1.10.14/lib/ant-apache-oro.jar, file:/apache-ant-1.10.14/lib/ant-jmf.jar, file:/apache-ant-1.10.14/lib/ant-apache-xalan2.jar, file:/apache-ant-1.10.14/lib/ant-apache-bsf.jar, file:/apache-ant-1.10.14/lib/ant-commons-logging.jar]

You’ll see that the HelloTask was loaded using an instance of AntClassLoader (which is internal implementation detail of the Ant project) and this classloader has the relevant JAXB jars in its classpath (as seen in the message above). You’ll also notice in the log message that the thread’s context classloader is an instance of URLClassLoader:

[hello] Context classloader of current thread is java.net.URLClassLoader@6d6f6e28

and this instance of URLClassLoader has a classpath which has only Ant specific jars and nothing related to JAXB jars. Now when the specific call to JAXBContext.newInstance(...) gets made it ends up using the URLClassLoader (since it is the thread context classloader) which doesn’t have JAXB jars. Effectively, you end up seeing the classloading failures and the build fails.

So how do we fix this. The important bit here is that the thread context classloader should be the one which has the JAXB classes available, so that it can load them. Java’s java.lang.Thread class allows the context classloader to be changed/switched. In fact, in the Java EE ecosystem, frameworks, servers and other implementations (typically not the application code), switch the thread’s context classloader to a “relevant” classloader at the “right place” and then switch it back to the old context classloader when the operation completes. We will need a similar implementation here in the custom task’s execute() method. Here’s what the snippet will now look like (we no longer need the debug logging, so that’s now been removed):

@Override
    public void execute() {
        // get the current context classloader
        final ClassLoader tccl = Thread.currentThread().getContextClassLoader();
        try {
            // change the thread context classloader to this task's classloader
            // before using the JAXB API
            Thread.currentThread().setContextClassLoader(HelloTask.class.getClassLoader());
            final JAXBContext context = JAXBContext.newInstance(DataContainer.class);
            System.out.println("Created JAXB context " + context);
        } catch (JAXBException e) {
            throw new RuntimeException(e);
        } finally {
            // restore back the old context classloader
            Thread.currentThread().setContextClassLoader(tccl);
        }

        System.out.println(this.message);
    }

Notice that we get hold of the current context classloader and then before calling JAXBContext.newInstance(...) we change the context classloader to the HelloTask’s classloader. The HelloTask’s classloader, as we have seen so far, has the necessary JAXB jars (since we defined it in the classpath of the taskdef in the build.xml) from which it should be able to load the JAXB classes. Finally, and very importantly, in a finally block we restore the context classloader to whatever it was previously - that way rest of the code isn’t impacted by switching the thread context classloader. Let’s now build the project again on Java 17:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

When you now run this, you should see:

invoke-task:
    [hello] Created JAXB context jar:file:/lib/jaxb-impl.jar!/com/sun/xml/bind/v2/runtime/JAXBContextImpl.class Build-Id: 2.3.8
    [hello] Classes known to this context:
    [hello]   [B
    [hello]   boolean
    [hello]   byte
    [hello]   char
    [hello]   com.sun.xml.bind.api.CompositeStructure
    [hello]   double
    [hello]   float
    [hello]   int
    ...
    [hello]   org.myapp.HelloTask$DataContainer
    [hello]   short
    [hello]   void
    [hello] 
    [hello] hello world

So the build now succeeds.

To summarize, recent versions of JDK have removed certain APIs from the JDK. Some of such APIs are available as external libraries which can be configured in the application classpath. In context of Ant, if a task that is shipped by Ant makes use of such APIs, then the Ant release note and the manual of that task will make a note of such change and will also note what needs to be done to get that task functional. In other cases, where custom tasks are involved and depend on APIs that are no longer part of the JDK, on most occasions they will need to update their build files to include those dependencies in their taskdef’s classpath. In some additional cases, like this very specific JAXBContext API usage, they might even have to do changes to the task’s code to use the right classloader. Do note that I decided to use this specific API of JAXB only to demonstrate the classloader change that would be needed in the task. Not all tasks that use JAXB would need this change in the task’s code - the build.xml classpath changes are expected. Also note that switching of classloaders shouldn’t be done blindly as it can cause other classloading issues. It should only be done when the specific API call in question has semantics which specify the use of a context classloader.

Some of you would be wondering if Ant itself should be doing a change where it sets the “right” thread context classloader before invoking the tasks. It wouldn’t be right for Ant to be doing such a change - depending on what the custom task does, there could be different answers to what is the “right” thread context classloader to use and for what duration. Answers to either of those questions are task specific and as such should be handled/implemented in the (custom) tasks.