Boot a custom (MacOS) kernel on MacOS Apple Silicon

The past several months I've been attempting to debug some (Java related) issues on macos. Most of these issues relate to the syscalls or native function calls that the JDK does on macos in order to provide implementation for some of the Java SE APIs. Naturally, debugging those issues means looking at macos specific system/kernel logs and sometimes even the macos kernel code. In some cases even that isn't enough.

Several parts (but not all) of macos kernel source is available as opensource code at https://opensource.apple.com/releases/. For each macos release, Apple updates that page with the relevant kernel versions which allows you to download the source or check out relevant tag from the linked GitHub repo. For example, here's the XNU repo https://github.com/apple-oss-distributions/xnu which represents parts of the kernel code that runs macos.

Given this, for debugging some of the issues, I've been looking to build the XNU kernel code and then have macos boot with that custom built kernel. That's anyway the end goal. It isn't easy though, because although there are build instructions on how to do it, those aren't always up to date. Plus, I'm told that it requires the exact set of tools and exact versions of those tools to be able to locally build a kernel and then have macos succesfully boot using that built kernel. I haven't yet managed to accomplish that.

For now though, I'll focus on a slightly different but related topic. For each release of macos, Apple provides a Kernel Debug Kit (KDK). Among other things, each KDK consists of pre-built kernels for different models of the macos hardware. Furthermore, for each of these macos models, the KDK has different variants of the kernel - "release" variant, "development" variant and "debug" variant. I haven't found an official document which states what exactly is different between each variant (or how exactly they are built), but from what I have read so far, the "release" variant is the one that matches the kernel that runs "production" macos. "development" and "debug" on the other hand, I'm guessing, are built using different flags to allow kernel developers and kernel extension developers to experiment with the kernel or kernel extensions.

Given that the KDK ships these pre-built kernels, it then means that I should be able to skip the part where I was considering building the XNU project and just jump to the step where I somehow point macos to use the kernel binary that's present in the KDK instead of the one it uses by default to boot macos. Unfortunately, there's no official documentation which explains how to do it. I have been looking for some hints and articles to find a way to boot a custom kernel binary and some of the articles I read had instructions that no longer worked. So I finally asked in the macos developer forum https://developer.apple.com/forums/thread/801605. A big thank you to Kevin for not just responding but also providing additional guidance on how to get this working.

In the rest of this article, I'll summarize the steps that I followed to have macos boot a kernel binary that's shipped in the Apple provided KDK. Do note that these steps are only applicable for Apple Silicon based macos systems (i.e. macos ARM64 architecture) and aren't applicable for Intel based macos systems.

Also note that these steps have been influenced heavily by the inputs provided in that developer forum thread as well as the following 2 articles:

https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html

and

https://khronokernel.com/macos/2023/08/08/AS-VM.html

The steps I note here are essentially a working and trimmed down version of what's explained in those articles.

It's very important to first know which KDK is relevant for use. Each release of XNU kernel has a corresponding KDK version. First, we will see what macos version we are currently running on. Note that macos OS version and XNU kernel version are not the same. Each has a different value.

In order to find out the macos version, run the "sw_vers" command:

sw_vers

ProductName:        macOS
ProductVersion:        15.6
BuildVersion:        24G84

So we are on 15.6 version of macos. Next we go to https://developer.apple.com/download/all and sign in with an Apple developer account. This page will list various products including the Kernel Debug Kit (KDK) for various versions of macos. We will locate the KDK for 15.6 of macos here. You will find it listed as "Kernel Debug Kit 15.6 build 24G84" (the build matches with the output we saw for sw_vers). Download the linked ".dmg" file from there and once downloaded, install it (like you would install any other dmg file). Upon installation, this version of the KDK will end up under the "/Library/Developer/KDKs/" directory on the filesystem. This specific KDK installation will be at "/Library/Developer/KDKs/KDK_15.6_24G84.kdk".

Let's set an environment variable named "KDK" to point to this location and for the rest of this article we will refer to that environment variable.

export KDK=/Library/Developer/KDKs/KDK_15.6_24G84.kdk

Next, let's list the kernel binaries that are shipped in this KDK. They will be present under the "$KDK/System/Library/Kernels/" directory. So:

ls $KDK/System/Library/Kernels/

This will print out several files, like:

kernel                kernel.kasan.t8112.dSYM
kernel.development        kernel.kasan.t8122
kernel.development.t6000    kernel.kasan.t8122.dSYM
kernel.development.t6020    kernel.kasan.t8132
kernel.development.t6030    kernel.kasan.t8132.dSYM
kernel.development.t6031    kernel.kasan.vmapple
kernel.development.t6041    kernel.kasan.vmapple.dSYM
kernel.development.t8103    kernel.release.t6000
kernel.development.t8112    kernel.release.t6000.dSYM
kernel.development.t8122    kernel.release.t6020
kernel.development.t8132    kernel.release.t6020.dSYM
kernel.development.vmapple    kernel.release.t6030
...


Each of these are either kernel binaries or debug symbols for the kernel binaries. And each of these binaries are for a different model of macos - the text "t8122", "t8132" and such is the macos model name.

So at this point, we have identified the macos operating system version we are on, then downloaded the corresponding KDK version for that macos version and installed it. The next step is to identify the macos model of our system. To do that we use the "uname -v" command:

uname -v

This prints:

Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:30 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6020

In the above output you will notice "RELEASE_ARM64_T6020". The T6020 is the macos model of this system and that's the one we are interested in. Note that the output will be different depending on which macos model you are running on and it is very important to make note of the exact model from the output.

Now that we know the model of our macos, let's again list the contents of our KDK installation and this time "grep" only for this T6020 model:

ls $KDK/System/Library/Kernels/ | grep -i T6020

This prints:

kernel.development.t6020
kernel.kasan.t6020
kernel.kasan.t6020.dSYM
kernel.release.t6020
kernel.release.t6020.dSYM

You will notice that there are 3 variants of the kernel available for this macos model in this KDK installation - "development", "kasan" and "release". Furthermore, the "release" and "kasan" variants each have a separate debug symbols file. For the "development" variant, there isn't a debug symbols file and I'm guessing it's because the "development" variant is compiled with debug symbols enabled within the binary. But that's merely a guess, like I noted previously, I haven't yet found the exact details of what each of these variants represent or how they are compiled. Curiously, there isn't a "debug" variant. There is a "kasan" variant, but I hadn't heard about that one before and I don't know what it is. But that's not important for now since our experiment is merely to boot a custom kernel binary instead of the default one. For this experiment, we will choose the "development" variant as the kernel binary we want to boot, so the "$KDK/System/Library/Kernels/kernel.development.t6020" file.

At this point, we have everything available locally to start the steps that are necessary to initiate the boot. First thing we need to do is create a "kext collection" using this kernel binary. To do that we use the "kmutil create" command ("kmutil" is a tool that's by default shipped in macos). Before doing that, I will create a directory on my filesystem where I would like to have this "kext collection" generated. This will be the file which will eventually be used by macos to boot the system. You can specify a location of your choice, and I chose "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/" as the directory:

mkdir -p /Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/

Next we will use the "kmutil create" command to create the kext collection in this directory. The following command is borrowed from the articles I linked previously:

kmutil create \
        --arch arm64e \
        --no-authorization \
        --variant-suffix development \
        --new boot \
        --boot-path /Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc \
        --kernel $KDK/System/Library/Kernels/kernel.development.t6020 \
        --repository $KDK/System/Library/Extensions \
        --repository /System/Library/Extensions \
        --repository /System/Library/DriverExtensions \
        --explicit-only $(kmutil inspect -V release --no-header | grep -v "SEPHiber" | awk '{print " -b "$1; }')



Except for the "--variant-suffix", "--boot-path" and the "--kernel" options, the rest of them will usually remain the same if you had to run it on your Apple Silicon based macos system.

We are choosing the "development" variant of the kernel, so I use "development" as the "--variant-suffix" option's value.

For the "--boot-path", I provide "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc" as the value. Notice that "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/" is the directory we just created a few lines above to use it as a destination directory for this generated "kext collection". Also notice that for this "--boot-path" option value, I use "macos-15.6.kc" as the file name. It can be anything, I chose this value to be clear what version of macos it corresponds to. When "kmutil create" successfully completes and creates the "kext collection", you will find a file named "macos-15.6.kc.development" where "development", in that file name, is the value you provided for the "--variant-suffix" option.

For the "--kernel" option, we point to the "$KDK/System/Library/Kernels/kernel.development.t6020" kernel file that we previously listed and identified for the current macos model, under the KDK installation directory.

The rest of the options I left as-is.

When you run this command and after it completes successfully, you should find the generated "kext collection" file at "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development" (or whatever option values you provided for "--boot-path" and "--variant-suffix").

At this point we now have a bootable file for macos boot process to use. Our next step is to configure macos to use this file. To do this we will first have to boot macos in recovery mode. So we first shutdown macos and then start macos in recovery mode by pressing down and holding the "power" button. When you do that, you will see a message which says macos is loading options. Continue to hold down the "power" button. This is a standard way to boot into recovery mode in macos, so I won't go into this detail. Once it boots in recovery mode, you will see a screen which has a "Options" icon. Click on it and then it will show a "Continue" button. Click on the "Continue" button. That will then show a screen with various menu options (at the top). Choose the "Utilities" -> "Terminal" menu option. It will open the terminal window.

On the terminal window, we will first disable System Integrity Protection (SIP) using the "csrutil" command:

csrutil disable

Confirm the action when prompted and provide the user/password if prompted.

Then we will run a command which will relax the restriction which prevents macos from booting arbitrary kernel binaries. We use the "bputil" command as follows:

bputil --disable-boot-args-restriction

Confirm the action when prompted and provide the user/password if prompted.

And now the last step where we configure macos to boot the kext collection file that we generated previously at "/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development". We use the "kmutil configure-boot" command for that:

kmutil configure-boot --volume "/Volumes/Macintosh HD" --custom-boot-object "/Volumes/Macintosh HD/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development"

The "--volume" options points to the target volume which is being configured. Typically, there's only one volume on the system and usually it's named "Macintosh HD", so the path is almost always "/Volumes/Macintosh HD". But if you have something different or if you have more than one volume on your system, then choose the appropriate value for this option.

The "--custom-boot-object" is the full path to the kext collection file that we created using the kernel binary shipped in the KDK. The full path includes the volume on which that kext collection file resides. So "/Volumes/Macintosh HD/Users/me/macos-kernel-experiments/macos-15.6/kext-collection/development/macos-15.6.kc.development".

It's important to note that the volume name and the path name may have space characters in their name. So remember to enclose these values in double quotes (like in the above example) or use appropriate escaping using the backslash (\) character.

Also note that the terminal window on which you type these commands has a very small font (at least on my instance) so it isn't easy to type in this lengthy command/text there. This became a bit of hassle when I was experimenting with these commands. So before booting into recovery mode, I created a ".sh" file beforehand with the complete "kmutil configure-boot" command and saved it at the "/Users/me/configure-boot.sh". Then instead of typing that "kmutil configure-boot" command manually, I ran:

bash "/Volumes/Macintosh HD/Users/me/configure-boot.sh"

from the terminal in the recovery mode. To be clear, creating this script file and using it is completely optionally. If you prefer typing the "kmutil configure-boot" command, that's fine too.

At this point, after the "kmutil configure-boot" completes successfully, all the steps needed to boot this custom built kernel are now done. Next, we shutdown and restart the system as usual (there's the Shutdown/restart option in the top menu).

When the system starts next, it will use the "macos-15.6.kc.development" binary that we built. To verify that it indeed is using this custom kernel binary, we can run the following command from the terminal once we have logged in to the system:

sysctl kern.osbuildconfig

This should output:

kern.osbuildconfig: development

which is the variant we built and configured to boot. This implies that we have successfully booted a custom variant of the kernel that is shipped in the KDK.

Do note that, sometimes, booting of this custom kernel binary may not be successful. When that happens, due to issues with the binary or some other issue that causes the boot process to fail, then during the boot you will see the Apple icon and it will stay up for a while before the system reboots and this will keep happening in a (never ending) loop. That's a sign that something has gone wrong and you won't be able to use this custom kernel binary. So to restore back to your normal kernel binary, you will have to reboot into recovery mode, like you did the previous time, by holding down the power button and waiting for the screen to show the "Options" screen and then clicking "Continue" button. Once you are into recovery mode, this time you will choose the "Utilities" -> "Startup Security Utility" menu option. The screen that then shows up will have 3 options and would have selected the "Permissive" mode selected (because we ran the csrutil and the bputil commands previously). Since the custom kernel binary that we have configured for the boot is not functioning, we will have to select the "Restricted" mode in that screen options. Choosing "Restricted" mode will prevent the use of the custom kernel binary and instead will use the one shipped by macos by default, when we boot next (and that's what we want). Save/Submit/Continue with the "Restricted" mode and, if prompted, provide the user/password to apply the change. After that's done, Shutdown/restart from the top menu option and let the system boot in normal mode. This will boot you in the release version of the kernel that macos ships by default and you can verify that by running the following command from the terminal once you have logged in to the system:

sysctl kern.osbuildconfig

That will output:

kern.osbuildconfig: release

That completes the steps needed to boot a custom kernel variant on macos and the step to recover if that process fails. Do note that we did not build the XNU project from source and instead reused a previously built kernel variant shipped in the KDK. Building the XNU from source (with trivial code modifications for experiment/debugging) is still on my TODO list but I am told that it isn't straightforward nor guaranteed to work. So that's for some other time.

Using JAXB in custom Ant tasks on recent Java versions

Apache Ant 1.10.14 was released a few weeks ago https://lists.apache.org/thread/9vhk51nkw9wjxzm7xk2q9xm6s803prmr. Like noted in that announcement, apart from the regular bug fixes, this release also has an important change which allows it to be used in the recently released Java 21 https://inside.java/2023/09/19/the-arrival-of-java-21/. One major goal of Ant project is to make sure that it can be used to build projects using latest versions of Java. As such, the Ant project team keeps a watch on any changes in the Java releases that could affect Ant and does necessary changes in Ant and releases them in its 1.10.x release series.

The announcement of 1.10.14 and this changelog https://github.com/apache/ant/blob/rel/1.10.14/WHATSNEW contains the details of what’s exactly changed in this release, so I won’t go into those details again. In this post I’ll however go through one interesting issue that was brought to my notice by more than one projects, when using recent versions of Java. The issue specifically relates to custom Ant tasks that use JAXB. However, in general, it applies to some of the APIs that have been removed from recent versions of Java (details in https://openjdk.org/jeps/320)

JAXB as noted in the reference doc https://docs.oracle.com/javase/tutorial/jaxb/intro/arch.html is an API and implementation for XML binding. Up until Java 11, JAXB API and implementation classes were shipped as part of the Java runtime. What it meant was that any application code, like custom developed Ant tasks, could just use the JAXB APIs in their code without having to explicitly specific any external library dependency. In Java 11, JAXB along with few other modules was removed from the JDK. The release notes of JDK 11 lists this change https://www.oracle.com/java/technologies/javase/11-relnote-issues.html#JDK-8190378. Additionally, JEP-320 https://openjdk.org/jeps/320 has all the details related to this removal. When using JAXB APIs,  the usage of these modules from the JDK was transparent to the application. So although the application may not explicitly have referred to these module names, it was still reliant on them because they were providing public APIs which the application had references to. Effectively, if a project was using some Ant task which used JAXB, then those projects when they switch to any Java version >= 11 will now start seeing issues due to missing compile and runtime dependency on JAXB. Let’s now consider a simple custom Ant task and see what kind of errors it might encounter. But if you are just interested in the shorter answer and some sample code to fix these classloading issues with JAXB, then please check the end of this article, starting here. For the complete details, please read on.

We will use a trivial custom Ant task implemented by a class called org.myapp.HelloTask, which looks like:

package org.myapp;

import org.apache.tools.ant.Task;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;

public class HelloTask extends Task {

    private String message;

    public void setMessage(final String message) {
        this.message = message;
    }

    @Override
    public void execute() {
        try {
            final JAXBContext context = JAXBContext.newInstance(DataContainer.class);
            System.out.println("Created JAXB context " + context);
        } catch (JAXBException e) {
            throw new RuntimeException(e);
        }

        System.out.println(this.message);
    }

    private static class DataContainer {
        private String data;
    }
}

The HelloTask is just here for demonstration and in practice doesn’t provide any real value. This task allows for a message attribute to be specified. Furthermore, in its execute() method it creates a javax.xml.bind.JAXBContext for the application specific DataContainer class and just prints that context. Additionally, it also prints the message that was set when the task is launched. Let’s assume that this task has been compiled and packaged as a jar file and made available to the build process. Now consider this trivial build.xml file which declares this task and then launches it:

<project default="invoke-task">
    <property name="build.dir" value="build"/>
    <property name="jar.name" value="hellotask.jar"/>

    <taskdef name="hello" classname="org.myapp.HelloTask">
        <classpath>
            <pathelement location="${build.dir}/${jar.name}"/>
        </classpath>    
    </taskdef>  

    <target name="invoke-task" description="invokes the HelloTask">
        <hello message="hello world"/>
    </target>       

</project>

There’s not much in this build.xml. All it does is, use a taskdef to define the HelloTask with the classpath containing (just the) jar file containing the HelloTask. Then in the invoke-task target we launch this hello task by passing it a message.

Let’s now run this build against Java 8:

export JAVA_HOME=<path-to-JDK-8>
ant invoke-task

When you run this (on Java 8) you should see the output similar to:

invoke-task:
    [hello] Created JAXB context jar:file:/<path-to-jdk-8>/jre/lib/rt.jar!/com/sun/xml/internal/bind/v2/runtime/JAXBContextImpl.class Build-Id: ...
    [hello] Classes known to this context:
    [hello]   [B
    [hello]   boolean
    [hello]   byte
    [hello]   char
    [hello]   com.sun.xml.internal.bind.api.CompositeStructure
    [hello]   double
    [hello]   float
    [hello]   int
...
    [hello]   long
    [hello]   org.myapp.HelloTask$DataContainer
    [hello]   short
    [hello]   void
    [hello] 
    [hello] hello world

You’ll see that the JAXB usage in the task was successful and the build completed successfully and we didn’t have to configure any classpath to include any JAXB jar files. As noted previously, this is because the Java 8 runtime ships with the relevant JAXB API and implementation classes.

Now, without changing any code in the task or in the build.xml, let’s just switch to a recent Java version. Let’s say Java 17 and run the build:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

When you do this, you will now see:

BUILD FAILED
build.xml:5: taskdef A class needed by class org.myapp.HelloTask cannot be found: javax/xml/bind/JAXBException
 using the classloader AntClassLoader[build/hellotask.jar]

You’ll notice that the build now fails and the error message states that the class javax/xml/bind/JAXBException cannot be found in the classpath which as defined in the build.xml only included the hellotask.jar (which just has the org.myapp.HelloTask). As noted previously, this is because the JAXB API and implementation is no longer shipped in the JDK. Applications, like this project, are expected to now include the JAXB API and implementation jars in the application classpath. JEP-320 https://openjdk.org/jeps/320 lists the potential Maven co-ordinates to find such jar(s). In case of JAXB it suggests the JAXB reference implementation available in Maven central repo com.sun.xml.bind:jaxb-ri as a potential candidate. Do note that, just like several other Java EE APIs and implementations, JAXB too has several vendors which implement the JAXB API. A “reference implementation” as the name states is meant to demonstrate the implementation of the specified API. There can, and are, several other vendor implementations for such APIs. It’s upto the applications to choose the ones that they desire to use. In this demonstration, we will use the 2.3.8 version of com.sun.xml.bind:jaxb-ri dependency (available at https://repo.maven.apache.org/maven2/com/sun/xml/bind/jaxb-ri/2.3.8/). There’s no specific reason for my choice of this specific version - it’s only for demo.

Now that we have this dependency made available, let’s include it in the classpath of the taskdef of HelloTask and our build.xml (snippet) will now look like:

...
    <taskdef name="hello" classname="org.myapp.HelloTask">
        <classpath>
            <pathelement location="${build.dir}/${jar.name}"/>
            <!-- JAXB dependencies -->
            <pathelement location="${lib.dir}/jakarta.activation.jar"/>
            <pathelement location="${lib.dir}/jakarta.xml.bind-api.jar"/>
            <pathelement location="${lib.dir}/jaxb-impl.jar"/>
        </classpath>    
    </taskdef>  
...

You’ll notice that the classpath of the taskdef now includes the JAXB related dependency jars. Now let’s rerun the build on Java 17, like previously:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

When you run this, you will now see something that starts like this:

BUILD FAILED
build.xml:17: java.lang.RuntimeException: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.bind.v2.ContextFactory]
    at org.myapp.HelloTask.execute(Unknown Source)
    

So the build still fails, but unlike previously where it had failed when trying to define the HelloTask itself, this time it fails in the execute() implementation of the HelloTask.

So why does it fail even with the JAXB jars in the classpath. This has to do with JAXB (and several other Java EE APIs), which rely on thread context classloader. Several of these APIs, including this call to:

final JAXBContext context = JAXBContext.newInstance(DataContainer.class);

relies on thread context classloader to find the JAXB related classes and resources. It expects the thread context classloader, whichever it is, to be able to load these JAXB classes. Thread context classloaders are specific to the thread that is currently executing the code. So let’s quickly see which classloaders are in play in the execute() method of the HelloTask. To see that, let’s add some trivial debug messages in the code, whose snippet will now look like:

    @Override
    public void execute() {
        System.out.println(HelloTask.class + " was loaded by classloader: " + HelloTask.class.getClassLoader());
        final ClassLoader tccl = Thread.currentThread().getContextClassLoader();
        System.out.println("Context classloader of current thread is " + tccl);
        if (tccl instanceof java.net.URLClassLoader) {
            // let's additionally print the classpath of the URLClassLoader
            final java.net.URL[] classpath = ((java.net.URLClassLoader) tccl).getURLs();
            System.out.println("Context classloader's classpath is " + java.util.Arrays.toString(classpath));
        }
        try {
            final JAXBContext context = JAXBContext.newInstance(DataContainer.class);
            System.out.println("Created JAXB context " + context);
        } catch (JAXBException e) {
            throw new RuntimeException(e);
        }

        System.out.println(this.message);
    }

So what we have done here is that added a few System.out.println messages which prints the classloader which loaded the HelloTask and also prints the current thread’s context classloader. Additionally if the context classloader is a java.net.URLClassLoader, we even print the classpath used by the URLClassLoader. The goal of these debug messages is to see what classloaders are in play when using that JAXB API. Let’s rerun the build again on Java 17 - it’s still expected to fail like previously, but this time we should see these debug messages:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

This fails with the same exception stacktrace as previously, but this time you should also see, something like:

[hello] class org.myapp.HelloTask was loaded by classloader: AntClassLoader[build/hellotask.jar:lib/jakarta.activation.jar:lib/jakarta.xml.bind-api.jar:lib/jaxb-impl.jar]
[hello] Context classloader of current thread is java.net.URLClassLoader@6d6f6e28
[hello] Context classloader's classpath is [file:/apache-ant-1.10.14/lib/ant-commons-net.jar, file:/apache-ant-1.10.14/lib/ant-xz.jar, file:/apache-ant-1.10.14/lib/ant-junit4.jar, file:/apache-ant-1.10.14/lib/ant-jai.jar, file:/apache-ant-1.10.14/lib/ant-apache-resolver.jar, file:/apache-ant-1.10.14/lib/ant-jdepend.jar, file:/apache-ant-1.10.14/lib/ant-apache-regexp.jar, file:/apache-ant-1.10.14/lib/ant-apache-log4j.jar, file:/apache-ant-1.10.14/lib/ant-javamail.jar, file:/apache-ant-1.10.14/lib/ant-apache-bcel.jar, file:/apache-ant-1.10.14/lib/ant.jar, file:/apache-ant-1.10.14/lib/ant-netrexx.jar, file:/apache-ant-1.10.14/lib/ant-swing.jar, file:/apache-ant-1.10.14/lib/ant-jsch.jar, file:/apache-ant-1.10.14/lib/ant-junitlauncher.jar, file:/apache-ant-1.10.14/lib/ant-jakartamail.jar, file:/apache-ant-1.10.14/lib/ant-junit.jar, file:/apache-ant-1.10.14/lib/ant-imageio.jar, file:/apache-ant-1.10.14/lib/ant-launcher.jar, file:/apache-ant-1.10.14/lib/ant-antlr.jar, file:/apache-ant-1.10.14/lib/ant-testutil.jar, file:/apache-ant-1.10.14/lib/ant-apache-oro.jar, file:/apache-ant-1.10.14/lib/ant-jmf.jar, file:/apache-ant-1.10.14/lib/ant-apache-xalan2.jar, file:/apache-ant-1.10.14/lib/ant-apache-bsf.jar, file:/apache-ant-1.10.14/lib/ant-commons-logging.jar]

You’ll see that the HelloTask was loaded using an instance of AntClassLoader (which is internal implementation detail of the Ant project) and this classloader has the relevant JAXB jars in its classpath (as seen in the message above). You’ll also notice in the log message that the thread’s context classloader is an instance of URLClassLoader:

[hello] Context classloader of current thread is java.net.URLClassLoader@6d6f6e28

and this instance of URLClassLoader has a classpath which has only Ant specific jars and nothing related to JAXB jars. Now when the specific call to JAXBContext.newInstance(...) gets made it ends up using the URLClassLoader (since it is the thread context classloader) which doesn’t have JAXB jars. Effectively, you end up seeing the classloading failures and the build fails.

So how do we fix this. The important bit here is that the thread context classloader should be the one which has the JAXB classes available, so that it can load them. Java’s java.lang.Thread class allows the context classloader to be changed/switched. In fact, in the Java EE ecosystem, frameworks, servers and other implementations (typically not the application code), switch the thread’s context classloader to a “relevant” classloader at the “right place” and then switch it back to the old context classloader when the operation completes. We will need a similar implementation here in the custom task’s execute() method. Here’s what the snippet will now look like (we no longer need the debug logging, so that’s now been removed):

@Override
    public void execute() {
        // get the current context classloader
        final ClassLoader tccl = Thread.currentThread().getContextClassLoader();
        try {
            // change the thread context classloader to this task's classloader
            // before using the JAXB API
            Thread.currentThread().setContextClassLoader(HelloTask.class.getClassLoader());
            final JAXBContext context = JAXBContext.newInstance(DataContainer.class);
            System.out.println("Created JAXB context " + context);
        } catch (JAXBException e) {
            throw new RuntimeException(e);
        } finally {
            // restore back the old context classloader
            Thread.currentThread().setContextClassLoader(tccl);
        }

        System.out.println(this.message);
    }

Notice that we get hold of the current context classloader and then before calling JAXBContext.newInstance(...) we change the context classloader to the HelloTask’s classloader. The HelloTask’s classloader, as we have seen so far, has the necessary JAXB jars (since we defined it in the classpath of the taskdef in the build.xml) from which it should be able to load the JAXB classes. Finally, and very importantly, in a finally block we restore the context classloader to whatever it was previously - that way rest of the code isn’t impacted by switching the thread context classloader. Let’s now build the project again on Java 17:

export JAVA_HOME=<path-to-JDK-17>
ant invoke-task

When you now run this, you should see:

invoke-task:
    [hello] Created JAXB context jar:file:/lib/jaxb-impl.jar!/com/sun/xml/bind/v2/runtime/JAXBContextImpl.class Build-Id: 2.3.8
    [hello] Classes known to this context:
    [hello]   [B
    [hello]   boolean
    [hello]   byte
    [hello]   char
    [hello]   com.sun.xml.bind.api.CompositeStructure
    [hello]   double
    [hello]   float
    [hello]   int
    ...
    [hello]   org.myapp.HelloTask$DataContainer
    [hello]   short
    [hello]   void
    [hello] 
    [hello] hello world

So the build now succeeds.

To summarize, recent versions of JDK have removed certain APIs from the JDK. Some of such APIs are available as external libraries which can be configured in the application classpath. In context of Ant, if a task that is shipped by Ant makes use of such APIs, then the Ant release note and the manual of that task will make a note of such change and will also note what needs to be done to get that task functional. In other cases, where custom tasks are involved and depend on APIs that are no longer part of the JDK, on most occasions they will need to update their build files to include those dependencies in their taskdef’s classpath. In some additional cases, like this very specific JAXBContext API usage, they might even have to do changes to the task’s code to use the right classloader. Do note that I decided to use this specific API of JAXB only to demonstrate the classloader change that would be needed in the task. Not all tasks that use JAXB would need this change in the task’s code - the build.xml classpath changes are expected. Also note that switching of classloaders shouldn’t be done blindly as it can cause other classloading issues. It should only be done when the specific API call in question has semantics which specify the use of a context classloader.

Some of you would be wondering if Ant itself should be doing a change where it sets the “right” thread context classloader before invoking the tasks. It wouldn’t be right for Ant to be doing such a change - depending on what the custom task does, there could be different answers to what is the “right” thread context classloader to use and for what duration. Answers to either of those questions are task specific and as such should be handled/implemented in the (custom) tasks.

Special characters in proxy configuration of docker daemon and systemd

 
This post is about some of the struggles I've had while dealing with configuring HTTP(s) proxy for docker daemon process. I note it here for myself as well as others who might run into this issue in future.

People familiar with Docker will know that it has a process (called docker daemon) which is responsible for handling requests from a docker client - like the docker CLI or language specific libraries like Python's docker package. This docker daemon process is responsible for handling these requests (for example for a docker image pull) and serving those requests. In that interaction, the docker daemon will communicate with docker registries either hosted internally or available over the Internet. If your system(s) where you have the docker daemon running have a proxy installed, then you will have to configure your docker daemon process to use those proxies.

The Docker documentation has a section which explains how to do that https://docs.docker.com/config/daemon/systemd/#httphttps-proxy. It's pretty straightforward as you see it there. You just configure the HTTP_PROXY and HTTPS_PROXY environment variables to point to the proxy URL.

Now imagine your proxy requires a username/password authentication. Configuring this detail too is pretty straightforward - you configure the HTTP_PROXY and HTTPS_PROXY environment variable just like before, but also include the username and password in the URL. For example, if your proxy host was 192.168.10.12 and the user name was "john" and password was "doe", you would configure the environment variable as follows:

Environment="HTTP_PROXY=http://john:doe@192.168.10.12:80"

in that configuration file noted in the Docker documentation.

Things are fine and straightforward so far. However, it gets interesting when the user name (or password) has a special character involved. For example, imagine the username is a Windows style user name where you specify the domain and the user name separated by a "\" (backslash) character. So imagine "john" being in the "dev" domain, so the username for authentication is "dev\john". Using this value literally as follows will not work:

Environment="HTTP_PROXY=http://dev\john:doe@192.168.10.12:80"

The value provided in HTTP_PROXY is used by docker daemon process and it expects it to be URL encoded. What that means is characters like the "\" (backslash) need to be handled specifically and are expected to be URL encoded. So the URL encoded version of that character is %5C. One would expect that setting that environment variable to include this encoded value would work. So:

Environment="HTTP_PROXY=http://dev%5Cjohn:doe@192.168.10.12:80"

But no - docker daemon's interaction with the docker registries, which is one of the places where the proxy gets used, kept failing. It wasn't clear why this was failing, however, looking at the logs that get generated in /var/log/messages, this message stood out:

Jan 1 04:00:12 localhost systemd[1]: /etc/systemd/system/docker.service.d/http-proxy.conf:2: Failed to resolve unit specifiers in HTTP_PROXY=http://dev%5Cjohn:doe@192.168.10.12:80, ignoring: Invalid slot

So clearly the value was being picked up but was being considered invalid and ignored. Thus leading to issues while dealing with the proxy on that system. It wasn't clear why it was considering that value invalid. After all, we had already URL encoded that special character.

It took a bit of time before I could spot a hint in that log message. If you look at that log message closely, you will notice that the log message is being logged by "systemd" not by the docker daemon. This is a sign that "systemd" is the one which is considering this value invalid. So how's systemd involved in this? In context of this issue, systemd is the process which manages "services" on your operating system. The docker daemon is just another such service. The file(s) we have been using to configure the HTTP_PROXY environment variable are infact configuration files that systemd parses and manages for the individual services. So clearly, systemd doesn't like this specified value for the HTTP_PROXY environment variable.

Reading through the systemd documentation, it became clear why it was having a problem with this value. Turns out for systemd configuration files (like the one we have here), % happens to be a special character and represents something called as a "specifier". So clearly, as noted by that warn message, it's trying to use %5 as a specifier and fails to understand what specifier it is. So we need to add another layer of escaping (although for a different tool - this time systemd) and we have to escape the % character to tell systemd not to interpret it as a specifier. To do that we use the % character, so it now becomes %%5C (as noted in https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers %% means use % literal).

So that now means, the final value for our HTTP_PROXY environment variable, as configured in the systemd service's configuration file is:

Environment="HTTP_PROXY=http://dev%%5Cjohn:doe@192.168.10.12:80"

(notice the use of %%)

Once this was done and the docker daemon restarted, things started working fine and the docker daemon was able to interact with the docker registries.

It's always tricky to get tools/libraries working when special characters are involved. As seen here, it gets even more trickier when multiple tools are involved in using the same value and each one has a different set of special characters.

I hope this post helps those who run into similar issues with docker daemon or systemd configurations in general.