Table of Contents
- The Basic Problem with Docker for a “Personal” User Account
- How User-Namespaces Addresses the Ownership Issue for Docker Containers
- How to Setup a Personal User-Namespace for Docker Containers
You may be thinking — Don! When are you going show us how to fire up containers from that great NVIDIA NGC docker registry you have been talking about? We are almost there, I promise.
If you have followed along with the previous posts, you will have a solid Ubuntu 16.04 base systems setup including Docker and Nvidia-Docker v2 installed and running. If you haven’t looked through those previous posts you might want to do that, How-To Setup NVIDIA Docker and NGC Registry on your Workstation – Part 1 Introduction and Base System Setup, and How-To Setup NVIDIA Docker and NGC Registry on your Workstation – Part 2 Docker and NVIDIA-Docker-v2.
About 1 year ago I wrote a post titled Docker and NVIDIA-docker on your workstation: Setup User Namespaces. I recommend giving that a read. It covers details about Linux Kernel Namespaces, the User-Namespace and how that applies to Docker. That post goes through a series of experiments illustrating why you would want to configure docker with User-Namespaces i.e. you want to run as “you” not as root! I will not go through as much detail in this post since the old post is a good reference.
Use this current post as your guide to setup User-Namspaces for Docker, since some things have changed since that older post. Most importantly, I will use the Docker JSON configuration file for setup rather than creating a systemd add-in file.
The Basic Problem with Docker for a “Personal” User Account
By default Docker containers run as root inside the container AND on your system!
If you are using Docker for the, perhaps more typical, case of firing off short lived micro-services on a server then the above statement may not bother you so much (but maybe it still should bother you). If this is your personal workstation and you are wanting to leverage Docker for ease-of-use managing application dependencies for your personal work-flow then you want to own your files and processes!
Illustration of docker container file-ownership problems
As part of the Docker setup I’ve outlined in these post I added my own user account to the “docker group”. That means I can start containers without being root. This is basically the point since we are looking at using Docker for a personal work-flow.
Lets see what file ownership looks like when you run Docker. I created a director in my home directory called projects/docker-test. In that directory I have created two files, one owned by me and one owned by root.
kinghorn@i9:~/projects/docker-test$ ls -l total 0 -rw-rw-r-- 1 kinghorn kinghorn 0 Feb 5 13:56 file-owned-by-me -rw-r--r-- 1 root root 0 Feb 5 13:56 file-owned-by-root
Now, I’m going to start up the (tiny) “alpine Linux” docker container and have it create a file in that same directory and then exit.
kinghorn@i9:~/projects/docker-test$ docker run --rm -it -v $HOME/projects/docker-test:/opt alpine /bin/touch /opt/file-created-in-container
I have used -v to bind the directory we are looking at to /opt in the container and used touch to create an empty file. Let’s look at that directory again,
kinghorn@i9:~/projects/docker-test$ ls -l total 0 -rw-r--r-- 1 root root 0 Feb 5 15:51 file-created-in-container -rw-rw-r-- 1 kinghorn kinghorn 0 Feb 5 13:56 file-owned-by-me -rw-r--r-- 1 root root 0 Feb 5 13:56 file-owned-by-root
That docker command created the file as owned by root.
If I list that directory from inside the container it looks like,
kinghorn@i9:~/projects/docker-test$ docker run --rm -it -v $HOME/projects/docker-test:/opt alpine /bin/ls -l /opt total 0 -rw-r--r-- 1 root root 0 Feb 7 20:47 file-created-in-container -rw-rw-r-- 1 1000 1000 0 Feb 7 20:29 file-owned-by-me -rw-r--r-- 1 root root 0 Feb 7 20:29 file-owned-by-root
You see in that output that inside the container my user account “kinghorn” doesn’t exist so file-owned-by-me is listed as belonging to userid “1000”.
This is not what I want! I want to own the files I create while inside a container and I want my files in any directory that I bind to the container to be owned by me (kinghorn) inside the container.
How User-Namespaces Addresses the Ownership Issue for Docker Containers
I’m going to do the same short test as above but this time it will be after implementing the User-Namespaces configuration. I will go through that configuration setup in the next section. You can compare the commands and output below to the previous section to see how things have changed with User-Namspaces enabled in docker.
Here’s the same file creation command I did above but with docker using my User-Namespace configuration,
kinghorn@i9:~/projects/docker-test$ docker run --rm -it -v $HOME/projects/docker-test:/opt alpine /bin/touch /opt/file-created-in-container
kinghorn@i9:~/projects/docker-test$ ls -l total 0 -rw-r--r-- 1 kinghorn kinghorn 0 Feb 7 12:31 file-created-in-container -rw-rw-r-- 1 kinghorn kinghorn 0 Feb 7 12:29 file-owned-by-me -rw-r--r-- 1 root root 0 Feb 7 12:29 file-owned-by-root
The file I created inside the container is now owned by me.
From inside the container it now looks very different too,
kinghorn@i9:~/projects/docker-test$ docker run --rm -it -v $HOME/projects/docker-test:/opt alpine /bin/ls -l /opt total 0 -rw-r--r-- 1 root root 0 Feb 7 22:05 file-created-in-container -rw-rw-r-- 1 root root 0 Feb 7 20:29 file-owned-by-me -rw-r--r-- 1 nobody nobody 0 Feb 7 20:29 file-owned-by-root
This is what you are seeing in the two ls outputs above;
- I setup docker to use my user (kinghorn) for a user-namespace remap. That means that on my system I own the docker containers that I start. So, if I create files (or run processes) inside a container I own them on my system rather than root owning them. However, inside the container my system user (kinghorn) is mapped to uid 1, a.k.a. root, inside the container. That is opposed to mapping root from my host system to root in the container.
- Also, you see that inside the container the file that I had created as root on my host is now owned by “nobody” rather than belonging to root in the container.
All of this is file ownership detail is important. If you are going to use docker for a personal work-flow you will want to have some user directory that is basically shared between containers you start and you host system. That could be working directories for job I/O and making data available to the container processes. That is what is happening with the part of the docker command line that looks like -v $HOME/projects/docker-test:/opt. You have control of the directories that you own on your host system and where they are being bound inside a container. This ownership applies to application processes too.
Caveat — By default you are root “inside the confines” of a container even with User-Namespaces enabled.
Is that a problem? Yes and No. From the host side things are much better. You own files and processes with containers. You are mostly “sand-boxed” in teh container. However, inside the container you are root (as always). This means if you bind a directory from your host machine into a container then from a shell in that container you could delete or modify “any” files in there.
You should only ever bind directories that “you” own into a container. That way your are just acting on those files with the same permissions you would have on the host side. I generally create a “projects” sub-directory in my account home directory and then mount a sub-directory of that into containers. I mount my own data directories in that sub-directory for things like machine learning jobs.
How to Setup a Personal User-Namespace for Docker Containers
I’m going to do a User-Namspaces setup that will have the same basic end-result as what I had done in a post last year, but this will be a new setup and configuration for the latest Docker version.
This User-Namespaces setup will achieve a more, “Workstation-User” friendly docker configuration. Your user account will “own” any docker containers and any created files. It will be nearly the same behavior for applications in docker containers as you would have if you had installed the application directly on your local system and run with your normal user account.
Setup subordinate ID’s
User-Namespaces use something called subordinate user ID’s. If you look at the file /etc/subuid you will see something similar to the following,
kinghorn@i9:~$ cat /etc/subuid lxd:100000:65536 root:100000:65536 kinghorn:165536:65536
We will change this file a little. The way it is, if I enabled user-namespace remap for docker and used “kinghorn” I would be mapping uid 1 in a container to uid 165536 on my system which is the first subordinate ID for kinghorn. (There is a range of 65536 ID’s available starting from that first one.) That’s OK, and it gets docker away from root on the host but it’s not useful for our case since my “real” host uid is actually 1000. That’s the user ID I want to own files created in containers. That user ID 165536 is arbitrary and I can actually easily add my real uid of 1000 as a subordinate ID. Here’s how to do that, ( You need to configure subordinate group ID’s in /etc/subgid too. )
sudo usermod -v 1000-1000 kinghorn
The 1000-1000 is the range of ID’s I’m adding, i.e. just one ID.
kinghorn@i9:~$ cat /etc/subuid lxd:100000:65536 root:100000:65536 kinghorn:165536:65536 kinghorn:1000:1
Now when I use user-namespace remap with Docker it will go with the lowest ID with is ID 1000 a.k.a. me (kinghorn)
You need to do the same thing for the subordinate group ID. The following command will add the gid to /etc/subgid the file will look identical to the subuid file.
sudo usermod -w 1000-1000 kinghorn
Configure Docker to use User-Namespaces
I will use the Docker JSON configuration file for setup rather than creating a systemd add-in file like I have done in the past. I didn’t really like adding systemd files for this in the past and now configuration with the JSON file seems to be working great. This is how the NVIDIA-Docker configuration gets added to Docker now too.
We will be making modifications to the /etc/docker/daemon.json file. Note you have have root privilege even to just look at this file (use sudo).
kinghorn@i9:~$ sudo cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
}
}
That is the daemon.json file with the NVIDIA runtime configuration information in it. If you do not have NVIDIA-Docker installed yet this file probably wont exist. You can just create it and add what I will show below or, better, you can go to my post last week and see how I installed NVIDIA-Docker.
I will need to add "userns-remap": "kinghorn" to that JSON configuration file. Of course you would want to add the user account that you are using.
I’m adding the userns-remap line as the “second” element in that JSON object file. The first entry is "runtimes": {...}. Use any editor you like but you will definitely need to use sudo to run it. It should look similar to the following when you are done.
{
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
},
"userns-remap": "kinghorn"
}
Don’t forget that comma!
Now all you need to do is restart docker,
sudo systemctl restart docker.service
Docker containers will now be running as you not as root on your host system.
If you now look at /var/lib/docker you will see a new directory in there for the container overlays created by your new primary docker user.
kinghorn@i9:~$ sudo ls -l /var/lib/docker total 52 drwx------ 14 kinghorn kinghorn 4096 Feb 7 16:56 1000.1000 drwx------ 2 root root 4096 Feb 1 15:39 builder drwx--x--x 3 root root 4096 Feb 1 15:39 containerd drwx------ 2 root root 4096 Feb 5 16:20 containers ...
When starting up new containers after changing to a User-Namespace configuration you will be pulling new overlays into that directory.
Note: If at any time you want to bypass the user-namespace remap you can do so on a individual container basis by adding the flag --userns=host to your docker command line.
We now have covered a core Ubuntu 16.04 install and desktop configuration, an up-to-date Docker install, installed NVIDIA-Docker, and added some “sanity” to the setup by using User-Namespaces in a way that make Docker much more usable on a workstation. Next up is how to get access and use that juicy NVIDIA NGC docker registry on your workstation!
Happy computing –dbk