Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/1460
Dr Donald Kinghorn (Scientific Computing Advisor )

How To Install Docker and NVIDIA-Docker on Ubuntu 19.04

Written on May 7, 2019 by Dr Donald Kinghorn
Share:

Introduction

Being able to get Docker and the NVIDIA-Docker runtime working on Ubuntu 19.04 makes this new and (currently) mostly unsupported Linux distribution a lot more useful. In this post I'll go through the steps that I used to get everything working nicely.

First my "standard'ish" disclaimer for unsupported install hacks,

Docker is a wonderful computing tool. It's displacing tradition servers and virtual machines in the data-center. It is also great on your workstation! It makes it possible to work with a variety of applications and frameworks that might be very difficult, incompatible or not even possible to directly install on your workstation.

I've been writing posts about docker and nvidia-docker since Feb. 2017. That was when NVIDIA launched NGC (NVIDIA GPU Cloud). I wrote a series of 5 posts in starting in Feb 2018 that form a guide to configuring docker and nvidia-docker for workstation use. Those posts are a good source for more detailed discussion on the steps we'll go through in this post. The 5th post in that series has links back to the earlier posts so for more details check them out.

How-To Setup NVIDIA Docker and NGC Registry on your Workstation - Part 5 Docker Performance and Resource Tuning

These are the steps we'll go through for the install and setup,

  • Setup the nvidia-docker repository
  • Setup the docker repository
  • Install and test docker
  • Install and test the nvidia-docker2 runtime
  • Configure docker group and "User Namespaces"
  • Tune docker for good performance with nvidia-docker compute jobs

Setup the nvidia-docker2 repository

To install docker and nvidia-docker on Ubuntu 19.04 we will need to do some manual repository configuration. There is a reason for this that you will see below.

nvidia-docker and docker-ce have to be in sync. the most recent nvidia-docker repositories are for Ubuntu 18.04 (and that makes sense). The code-base however is keep updated and built against the most recent docker build.

I'm going to start by configuring the NVIDIA-docker repositories first so we can look at the version to see what what docker repository and version will be needed.

1) Go to https://github.com/NVIDIA/nvidia-docker to see the latest supported Ubuntu version.

As of this writing supported Ubuntu versions are 14.04, 16,04, and 18.04 (all long term support versions). We will need to configure repos for Ubuntu 18.04 since it is the most recent.

2) Add the repo key,

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -

3) Configure the needed repos by hand.

If you try to use the script on the install page it will fail because it will return 19.04 as VERSION_ID. It's easy to just add the repos by hand.

Using sudo and your editor of choice create the following file,

/etc/apt/sources.list.d/nvidia-docker.list

add the following lines to it,

deb https://nvidia.github.io/libnvidia-container/ubuntu18.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-docker/ubuntu18.04/$(ARCH) /

Save that file and refresh the package list,

sudo apt-get update

4) Check the installable versions of nvidia-docker2.

This is what I got when I wrote this post,

kinghorn@u19:~$ apt-cache madison nvidia-docker2​​​​​​​

nvidia-docker2 | 2.0.3+docker18.09.5-3 | https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
nvidia-docker2 | 2.0.3+docker18.09.5-2 | https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
nvidia-docker2 | 2.0.3+docker18.09.4-1 | https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
...
...
...
nvidia-docker2 | 2.0.3+docker18.03.1-1 | https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
nvidia-docker2 | 2.0.3+docker17.12.1-1 | https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages

The "+docker ..." part of those lines is the corresponding docker version that needs to be installed for that version of nvidia-docker. The most recent version in this case is "docker18.09.5-3". Keep in mind that this is from a repo for Ubuntu 18.04, also know as "bionic". That will be important.

Setup the docker repository, Install and check docker

We'll mostly follow the install described here https://docs.docker.com/install/linux/docker-ce/ubuntu/ except that we will be specific about the version and Ubuntu release.

1) Install some required packages, (they may already be installed)

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

2) Add and check the docker key,

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo apt-key fingerprint 0EBFCD88

The fingerprint should match "9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88"

3) Now add the repository for "bionic"

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

Note: I explicitly used "bionic" rather than $(lsb_release -cs) (that would return "disco"). We have to use "bionic" in the repository to keep apt-get from failing during the nvidia-docker2 install.

4) Check the installable docker versions,

kinghorn@u19:~$ apt-cache madison docker-ce

docker-ce | 5:18.09.5~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
docker-ce | 5:18.09.4~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
...
...
...
docker-ce | 18.06.0~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
docker-ce | 18.03.1~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages

You see that the most recent version is tagged as "5:18.09.5~3-0~ubuntu-bionic". That will match up with the nvidia-docker2 version "2.0.3+docker18.09.5-3".


5) Install docker (with the specified version numbers).

sudo apt-get install docker-ce=5:18.09.5~3-0~ubuntu-bionic docker-ce-cli=5:18.09.5~3-0~ubuntu-bionic containerd.io

Note: "containerd.io" is independent of docker but included in that repository.

6) Check docker,

sudo docker run --rm hello-world

That should pull and run the hello-world container from Docker Hub. We will add user configuration later.

Install and check nvidia-docker2

1) Install nvidia-docker2 (using the version that will match with the docker install)

sudo apt-get install nvidia-docker2=2.0.3+docker18.09.5-3

2) Restart docker to enable the configuration for the nvidia-docker2 runtime.

sudo systemctl restart docker.service

3) Check nvidia-docker2

sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

That will pull the latest nvidia cuda container from Docker Hub, run the nvidia-smi command and exit.

(Recommended) Disable the repositories you installed from

If your docker build version and nvidia-docker build version get out of sync because of automatic system updates then your install will break. You can use the "apt hold" mechanism or simply comment out the repository lines at the end of /etc/apt/sources.list and in /etc/apt/sources.list.d/nvidia-docker.list

Of course, this means that if there are new releases that you want to upgrade to you will need to revert the repository configuration and recheck version numbers. But, this would be a good thing!

Setup docker for your personal user account with User Namespaces and add performance optimizations

1) Add your user name to the docker group

sudo usermod -aG docker your-user-name

Note: You will need to logout and back in for this to take effect.

3) Add your user and group id's as "subordinate id's".

See How-To Setup NVIDIA Docker and NGC Registry on your Workstation - Part 3 Setup User-Namespaces.

- Add a subordinate id

sudo usermod -v 1000-1000 kinghorn

- Add a subordinate group

sudo usermod -w 1000-1000 kinghorn

Note: Of course, use your own user-name and id. You can use the id command to check your user and group id.

4) Setup the docker configuration file for your User Namespace, the nvidia-docker runtime, and compute performance optimizations.

You cannot even access the docker configuration directory unless you are "root" so before you edit the file /etc/docker/daemon.json do,

sudo -s

cd /etc/docker

to give yourself a root shell. Then save a copy of the docker daemon.json config file and create a new one with the following JSON.

Note: Be sure to to change that "kinghorn" to your own user name!

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "userns-remap": "kinghorn",
    "default-shm-size": "1G",
    "default-ulimits": {
	     "memlock": { "name":"memlock", "soft":  -1, "hard": -1 },
	     "stack"  : { "name":"stack", "soft": 67108864, "hard": 67108864 }
    }
}

Docker will read this file on startup and add the nvidia-runtime so you can use your NVIDIA GPU with docker.

This config then sets up your "user" in the docker User Namespace. That means that even though you see a root prompt in a docker container any files you create will belong to you in your system namespace. Please reed the post I mentioned in step 3) of this section to understand the full meaning of this.

The last part ot the config changes the (pitiful) default performance parameters so the you can run large GPU accelerated compute jobs. The post I linked in the "Introduction" has details of that setup.

5) Restart docker

sudo systemctl restart docker.service

Done!

You now have what you will hopefully find to be a very useful docker configuration on your Ubuntu 19.04 workstation.

If you want to use some of the nice containers that NVIDIA has put together on NGC then go to there site and register for access. You could then, for example start up their TensorFlow container with something like,

docker run --runtime=nvidia --rm -it -v $HOME/projects:/projects nvcr.io/nvidia/tensorflow:19.02-py3

Note: The -v $HOME/projects:/projectspart of that line is mapping a directory named "projects" in your users home directory to the directory "/projects" in the container. Any files created there will be owned by your user and remain there after you exit from the container instance.


I hope this gets you going with docker on Ubuntu 19.04. If you have trouble and decide to go back to Ubuntu 18.04 the procedure in this post will work for that too! (And, it would still be good to be careful about the version number matching.)

Happy computing! --dbk


Looking for a
Scientific Compute System?

Do you have a project that needs serious compute power, and you don't know where to turn? Puget Systems offers a range of HPC workstations and servers tailored for both CPU and GPU workloads.

Why Choose Puget Systems?


Built specifically for you

Rather than getting a generic workstation, our systems are designed around your unique workflow and are optimized for the work you do every day.

Fast Build Times

By keeping inventory of our most popular parts, and maintaining a short supply line to parts we need, we are able to offer an industry leading ship time of 7-10 business days on nearly all our system orders.

We're Here, Give Us a Call!

We make sure our representatives are as accessible as possible, by phone and email. At Puget Systems, you can actually talk to a real person!

Lifetime Support/Labor Warranty

Even when your parts warranty expires, we continue to answer your questions and even fix your computer with no labor costs.

Click here for even more reasons!

Tags: Ubuntu 19.04, Docker, NVIDIA-docker
Darrow

Thanks a lot for this wonderful post and also the 5 part blog post on NVIDIA Docker & NGC setup. I really appreciate the time you took in compiling all this.

Posted on 2019-06-07 02:27:30
Laurie O

I've got nvidia-docker to work with the disco (Ubuntu 19.04) docker-ce by uncompiling the nvidia-docker.deb (with dpkg-deb), removing the version in the docker-ce requirement, recompiling, installing 'nvidia-container-runtime', then installing the recompiled package

Posted on 2019-06-24 05:37:06
Donald Kinghorn

Ha ha, I have "adjusted" version numbers before too :-) Cheers mate!

Posted on 2019-06-24 16:08:29
twyeld

Does it matter what nVidia card is installed on one's machine? ie it might be worth mentioning what card you are using - some may be using low end 700 series cards or 1050 etc... :)

Posted on 2019-06-24 22:45:23
Donald Kinghorn

Good point! I generally assume people will be using GeForce 900 series or newer GPU's. I don't recommend anything less than that but they may work perfectly fine. Low-end 900 1000 or 2000 (or equivalent Quadro) should be OK for almost anything. I use my laptop with a 1060 in it a lot. What works will be software dependent. More recent builds may have more recent "compute capability" restrictions. However, code that has been around for awhile may support all the way back to 2.0 or earlier. A lot of code still has full support for "Fermi" i.e. GTX 590 and 2nd gen Tesla.
The thing that determines if a program will work or not is the supported "compute capability" Take a look at the wikipedia CUDA page for a list of cards and capabilities. https://en.wikipedia.org/wi...

There is a possible gotcha, some low-end cards are just really made badly! I've seen a lot of problems from things like GTX 730's that had non-compliant firmware.

Posted on 2019-06-25 17:49:33
twyeld

At last (after 10s of attempts on earlier versions of ubuntu/nvidia-docker etc!) - this routine actually works (at least up to the part where one runs nvidia-smi - I haven't gone beyond that yet...). However, it did hangup at "Install and check nvidia-docker2" 1) Install nvidia-docker2 (using the version that will match with the docker install) sudo apt-get install nvidia-docker2=2.0.3+docker18.09.5-3"

the nvidia-docker-runtime package conflicts with the other 2 so this part should be:

"sudo apt-get install nvidia-docker2=2.0.3+docker18.09.5-3 nvidia-docker-runtime2=2.0.0+docker18.09.5-3"

note the zero on the end of the version number for the runtime package

at least this is what I needed to do to get past this point.

Also, for others that might strike the same problem, I am running a very cheap GeForce 710 card - just for experimenting on (and, yes, it needed a firmware update - but that happens when you specify the driver and reboot) - you need to select the 418 driver (in the applications & updates > alternative drivers tab) rather than the 390 driver for the runtime to work.

Finally, there may even be a conflict in getting Ubuntu to restart after all this - if it hangs use the esc key to get into the log screen then cycle through ctrl+alt+f1/2/3/4/5/6/7 and it will continue along its merry way to the login screen - brutal, I know, but it works!

Will update if I can get Donald's tensorflow py script to run...

Posted on 2019-06-26 05:16:13
Donald Kinghorn

Hey twyeld you really should look at getting a better video card :-) but you already know that :-)

interesting that you had to change that version number to 2.0.0 I just checked on my sys here,

sudo apt-get update then
apt-cache madison nvidia-docker2 gives at the top of the list
nvidia-docker2 | 2.0.3+docker18.09.6-3 | https://nvidia.github.io/nv... Packages

nvidia-docker2 shows to be at 2.0.3 and is tagged with deps for docker18.09.6 ??? Not sure why you had to change that. But, as long as everything is working it should be OK though.

I should note that I left my repos enabled and it looks like my system updated OK without breaking any version dependencies (which has happened to me in the past)
kinghorn@i9:~$ docker --version
Docker version 18.09.6, build 481bc77

I'm getting ready to do some config work for a "Data Science Workstation" setup. I'll pay extra attention to what's going on with the version numbers and note any strangeness.

Post back if you hit any more snags ... I'll try to get confirmation on an updated setup done on 19.04 again --Don

Posted on 2019-06-26 15:42:08
twyeld

Thanks again Donald - your version of the nvidia-docker2 package is correct - rather, its that I had to explicitly add the nvidia-docker-runtime2 package as version 2.0.0 :)

Posted on 2019-06-26 22:48:52