Special characters in proxy configuration of docker daemon and systemd

 
This post is about some of the struggles I've had while dealing with configuring HTTP(s) proxy for docker daemon process. I note it here for myself as well as others who might run into this issue in future.

People familiar with Docker will know that it has a process (called docker daemon) which is responsible for handling requests from a docker client - like the docker CLI or language specific libraries like Python's docker package. This docker daemon process is responsible for handling these requests (for example for a docker image pull) and serving those requests. In that interaction, the docker daemon will communicate with docker registries either hosted internally or available over the Internet. If your system(s) where you have the docker daemon running have a proxy installed, then you will have to configure your docker daemon process to use those proxies.

The Docker documentation has a section which explains how to do that https://docs.docker.com/config/daemon/systemd/#httphttps-proxy. It's pretty straightforward as you see it there. You just configure the HTTP_PROXY and HTTPS_PROXY environment variables to point to the proxy URL.

Now imagine your proxy requires a username/password authentication. Configuring this detail too is pretty straightforward - you configure the HTTP_PROXY and HTTPS_PROXY environment variable just like before, but also include the username and password in the URL. For example, if your proxy host was 192.168.10.12 and the user name was "john" and password was "doe", you would configure the environment variable as follows:

Environment="HTTP_PROXY=http://john:doe@192.168.10.12:80"

in that configuration file noted in the Docker documentation.

Things are fine and straightforward so far. However, it gets interesting when the user name (or password) has a special character involved. For example, imagine the username is a Windows style user name where you specify the domain and the user name separated by a "\" (backslash) character. So imagine "john" being in the "dev" domain, so the username for authentication is "dev\john". Using this value literally as follows will not work:

Environment="HTTP_PROXY=http://dev\john:doe@192.168.10.12:80"

The value provided in HTTP_PROXY is used by docker daemon process and it expects it to be URL encoded. What that means is characters like the "\" (backslash) need to be handled specifically and are expected to be URL encoded. So the URL encoded version of that character is %5C. One would expect that setting that environment variable to include this encoded value would work. So:

Environment="HTTP_PROXY=http://dev%5Cjohn:doe@192.168.10.12:80"

But no - docker daemon's interaction with the docker registries, which is one of the places where the proxy gets used, kept failing. It wasn't clear why this was failing, however, looking at the logs that get generated in /var/log/messages, this message stood out:

Jan 1 04:00:12 localhost systemd[1]: /etc/systemd/system/docker.service.d/http-proxy.conf:2: Failed to resolve unit specifiers in HTTP_PROXY=http://dev%5Cjohn:doe@192.168.10.12:80, ignoring: Invalid slot

So clearly the value was being picked up but was being considered invalid and ignored. Thus leading to issues while dealing with the proxy on that system. It wasn't clear why it was considering that value invalid. After all, we had already URL encoded that special character.

It took a bit of time before I could spot a hint in that log message. If you look at that log message closely, you will notice that the log message is being logged by "systemd" not by the docker daemon. This is a sign that "systemd" is the one which is considering this value invalid. So how's systemd involved in this? In context of this issue, systemd is the process which manages "services" on your operating system. The docker daemon is just another such service. The file(s) we have been using to configure the HTTP_PROXY environment variable are infact configuration files that systemd parses and manages for the individual services. So clearly, systemd doesn't like this specified value for the HTTP_PROXY environment variable.

Reading through the systemd documentation, it became clear why it was having a problem with this value. Turns out for systemd configuration files (like the one we have here), % happens to be a special character and represents something called as a "specifier". So clearly, as noted by that warn message, it's trying to use %5 as a specifier and fails to understand what specifier it is. So we need to add another layer of escaping (although for a different tool - this time systemd) and we have to escape the % character to tell systemd not to interpret it as a specifier. To do that we use the % character, so it now becomes %%5C (as noted in https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers %% means use % literal).

So that now means, the final value for our HTTP_PROXY environment variable, as configured in the systemd service's configuration file is:

Environment="HTTP_PROXY=http://dev%%5Cjohn:doe@192.168.10.12:80"

(notice the use of %%)

Once this was done and the docker daemon restarted, things started working fine and the docker daemon was able to interact with the docker registries.

It's always tricky to get tools/libraries working when special characters are involved. As seen here, it gets even more trickier when multiple tools are involved in using the same value and each one has a different set of special characters.

I hope this post helps those who run into similar issues with docker daemon or systemd configurations in general.

Apache Ant 1.10.10 released - Better test result summary from junitlauncher task

Apache Ant 1.10.10 got released around a month back. Among the usual bug fixes, we added a new enhancement for the "junitlauncher" task.

For those of you who haven't used or know about "junitlauncher" task, it's a new task we introduced a few years back to allow projects using Ant, to be able to use the new JUnit5 testing framework. The previous (and still supported) "junit" task is meant to be used only if you want to continue using just JUnit4. If you plan to use JUnit5 (which also supports JUnit4 style testcases), then you will have to use the "junitlauncher" task.

This "junitlauncher" task has been around for a few years now and some users have reported that its "printSummary" feature isn't of much use. People familiar with the "junit" task will know that when a test gets run, the task prints an instantaneous summary like:

org.myapp.foo.bar.SimpleTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec

This is useful to see a quick summary of the tests being run.

The "junitlauncher" has a "printSummary" attribute which until Ant 1.10.10 version used to print a summary after all the tests had been executed. Furthermore, the printed summary was a summary that the JUnit5 framework generates by default, something like:

[junitlauncher]
[junitlauncher] Test run finished after 5103 ms
[junitlauncher] [         2 containers found      ]
[junitlauncher] [         0 containers skipped    ]
[junitlauncher] [         2 containers started    ]
[junitlauncher] [         0 containers aborted    ]
[junitlauncher] [         2 containers successful ]
[junitlauncher] [         0 containers failed     ]
[junitlauncher] [         1 tests found           ]
[junitlauncher] [         0 tests skipped         ]
[junitlauncher] [         1 tests started         ]
[junitlauncher] [         0 tests aborted         ]
[junitlauncher] [         1 tests successful      ]
[junitlauncher] [         0 tests failed          ]

As you can see, summary of this form isn't really useful. So some of the Ant users requested (https://bz.apache.org/bugzilla/show_bug.cgi?id=64836) this to be improved to provide a summary which resembled what we have with the "junit" task.

This Ant 1.10.10 release now consists that enhancement. When you use "printSummary=true" on the "junitlauncher" task, it will now print a more useful and immediate summary like the "junit" task does:

Running org.myapp.foo.bar.SimpleTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec


As usual, the release is available for download at the Ant downloads page https://ant.apache.org/bindownload.cgi. Please give this a try and if you have any suggestion or feedback on this release, please get in touch with us on our mailing lists https://ant.apache.org/mail.html or our issue tracker https://ant.apache.org/bugs.html.