Special characters in proxy configuration of docker daemon and systemd

 
This post is about some of the struggles I've had while dealing with configuring HTTP(s) proxy for docker daemon process. I note it here for myself as well as others who might run into this issue in future.

People familiar with Docker will know that it has a process (called docker daemon) which is responsible for handling requests from a docker client - like the docker CLI or language specific libraries like Python's docker package. This docker daemon process is responsible for handling these requests (for example for a docker image pull) and serving those requests. In that interaction, the docker daemon will communicate with docker registries either hosted internally or available over the Internet. If your system(s) where you have the docker daemon running have a proxy installed, then you will have to configure your docker daemon process to use those proxies.

The Docker documentation has a section which explains how to do that https://docs.docker.com/config/daemon/systemd/#httphttps-proxy. It's pretty straightforward as you see it there. You just configure the HTTP_PROXY and HTTPS_PROXY environment variables to point to the proxy URL.

Now imagine your proxy requires a username/password authentication. Configuring this detail too is pretty straightforward - you configure the HTTP_PROXY and HTTPS_PROXY environment variable just like before, but also include the username and password in the URL. For example, if your proxy host was 192.168.10.12 and the user name was "john" and password was "doe", you would configure the environment variable as follows:

Environment="HTTP_PROXY=http://john:doe@192.168.10.12:80"

in that configuration file noted in the Docker documentation.

Things are fine and straightforward so far. However, it gets interesting when the user name (or password) has a special character involved. For example, imagine the username is a Windows style user name where you specify the domain and the user name separated by a "\" (backslash) character. So imagine "john" being in the "dev" domain, so the username for authentication is "dev\john". Using this value literally as follows will not work:

Environment="HTTP_PROXY=http://dev\john:doe@192.168.10.12:80"

The value provided in HTTP_PROXY is used by docker daemon process and it expects it to be URL encoded. What that means is characters like the "\" (backslash) need to be handled specifically and are expected to be URL encoded. So the URL encoded version of that character is %5C. One would expect that setting that environment variable to include this encoded value would work. So:

Environment="HTTP_PROXY=http://dev%5Cjohn:doe@192.168.10.12:80"

But no - docker daemon's interaction with the docker registries, which is one of the places where the proxy gets used, kept failing. It wasn't clear why this was failing, however, looking at the logs that get generated in /var/log/messages, this message stood out:

Jan 1 04:00:12 localhost systemd[1]: /etc/systemd/system/docker.service.d/http-proxy.conf:2: Failed to resolve unit specifiers in HTTP_PROXY=http://dev%5Cjohn:doe@192.168.10.12:80, ignoring: Invalid slot

So clearly the value was being picked up but was being considered invalid and ignored. Thus leading to issues while dealing with the proxy on that system. It wasn't clear why it was considering that value invalid. After all, we had already URL encoded that special character.

It took a bit of time before I could spot a hint in that log message. If you look at that log message closely, you will notice that the log message is being logged by "systemd" not by the docker daemon. This is a sign that "systemd" is the one which is considering this value invalid. So how's systemd involved in this? In context of this issue, systemd is the process which manages "services" on your operating system. The docker daemon is just another such service. The file(s) we have been using to configure the HTTP_PROXY environment variable are infact configuration files that systemd parses and manages for the individual services. So clearly, systemd doesn't like this specified value for the HTTP_PROXY environment variable.

Reading through the systemd documentation, it became clear why it was having a problem with this value. Turns out for systemd configuration files (like the one we have here), % happens to be a special character and represents something called as a "specifier". So clearly, as noted by that warn message, it's trying to use %5 as a specifier and fails to understand what specifier it is. So we need to add another layer of escaping (although for a different tool - this time systemd) and we have to escape the % character to tell systemd not to interpret it as a specifier. To do that we use the % character, so it now becomes %%5C (as noted in https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers %% means use % literal).

So that now means, the final value for our HTTP_PROXY environment variable, as configured in the systemd service's configuration file is:

Environment="HTTP_PROXY=http://dev%%5Cjohn:doe@192.168.10.12:80"

(notice the use of %%)

Once this was done and the docker daemon restarted, things started working fine and the docker daemon was able to interact with the docker registries.

It's always tricky to get tools/libraries working when special characters are involved. As seen here, it gets even more trickier when multiple tools are involved in using the same value and each one has a different set of special characters.

I hope this post helps those who run into similar issues with docker daemon or systemd configurations in general.