Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/915
Dr Donald Kinghorn (Scientific Computing Advisor )

Docker and NVIDIA-docker on your workstation: Using Graphical Applications

Written on March 21, 2017 by Dr Donald Kinghorn
Share:

This is my fourth post about "Docker and NVIDIA-Docker on your Workstation". In this post I will go through how to attach your X server display to a Docker container so that you can run application that have GUIs or other graphical display output.

It is assumed that you have already installed the host OS, Docker-Engine, and NVIDIA-Docker as described in the "installation post". I also highly recommend that you have setup Docker to use Kernel User-Namespaces.

Here is a list of the first three posts in this series for reference,

I recommend that you do not use X-Window with your Docker containers unless you have configured Docker with User-Namespaces as I have outlined in previous posts. If you do this then you are using X-Window with Docker as your normal log-in user account rather than as root.


How to Run a GUI application from a Docker Container

The most common use of Docker is for servers and other "command-line" applications. What if you want to run an application that needs your desktop display? You can do this. The relevant Docker command flags are,

-v /tmp/.X11-unix:/tmp/.X11-unix
-e DISPLAY=$DISPLAY

The flag -v /tmp/.X11-unix:/tmp/.X11-unix gives the container access to your X socket and -e DISPLAY=$DISPLAY sets the DISPLAY environment variable in the container the same as that of your desktop.

Note: You can add sound capability to a container with --device /dev/snd

Security: I believe that using X-Window with Docker in this way is as secure as if you were running any X-Window application with your normal user account. The reason for this is, if you are running Docker configured with User-Namespaces the way I have outlined then you are running with your normal user account from the perspective of the host!

Example: Compile and run CUDA nbody with OpenGL display output

This example is the kind of thing that I really wanted to be able to do easily with a Docker container. I'll start with the default NVIDIA CUDA 8 image and add the "freeglut3-dev" package to pull the needed dependencies into the container. Then compile nbody in the container and run it with openGL display output.

I have created a directory docker/cuda8 in my home directory and added put the CUDA "samples" source there. I will use this directory in the container.

  1. Start the container

nvidia-docker run -it -v $HOME/docker:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY nvidia/cuda
  1. cd to the nbody source directory and try to run make

cd projects/cuda8/samples/5_Simulations/nbody
...
>>> WARNING - libGL.so not found, ...
 
  1. No GL lib and a lot of other missing libs so lets install "freeglut3-dev" which will pull the dependencies we need.

apt-get update
apt-get install freeglut3-dev

Note: That will add a lot of packages to the image! They will add as a new "layer". We will save this modified container to a new image later.

  1. Try make again

make clean
make
...
Success!
  1. Now run nbody

./nbody

You should be greeted with something like this;
nbody OpenGL GUI

That is screenshot of the running nbody application using the GPU for CUDA acceleration and using your X display to show the openGL output. ... From a Docker container!


How to Save a Modified Container to a new Image

We just made a significant modification to the container we started from the default NVIDIA cuda8-dev image. This will persist as long as we don't remove the container. You can exit from the container and then re-start it and all of your changes will still be there. However, adding the openGL support to the container is a useful thing so saving that as a new image so you (and others) could create new containers from it is a reasonable thing to do.

There are a couple of ways to save your modified container as a new image. I'll show you how to use commit. In a later post I'll go through how to create a Dockerfile and use that to build a new image with the same changes that we apply in the container.

Using docker commit

If you are still connected to the modified container go ahead and exit, then run docker ps -a which will output something like,

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
9fc3da506758        nvidia/cuda         "/bin/bash"         19 hours ago        Exited (0) 18 seconds ago                       awesome_payne

Make note of the container ID. In this case it is 9fc3da506758. To save this as a new image do,

docker commit 9fc3da506758 cuda8-dev-opengl

That will output a sha256 ID for the new image and save it to your image directory with the new name. Do docker images and you should see your saved image. For example,

docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
cuda8-dev-opengl    latest              3022f6beb3ef        18 seconds ago      1.75 GB
alpine              latest              4a415e366388        2 weeks ago         3.99 MB
nvidia/cuda         latest              1cce8839a2c5        2 weeks ago         1.62 GB

Note You can add tags then you commit an image. I could have used something like docker commit 9fc3da506758 cuda8-dev:opengl. Tags are usually used for versioning but you can use naming schemes to suit your taste.

Note You only need to use enough characters from the ID hash to uniquely identify the container or image. For example using 9fc for the example container above would be sufficient.

You can now remove the old modified container with docker rm 9fc You can verify that the container was remove with docker ps -a

To start a new container with the new image we committed do

nvidia-docker run -it -v $HOME/docker:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY cuda8-dev-opengl

That is just scratching the surface of what you can do! It is possible to run many GUI applications including application that use GPU compute acceleration.

I think it is good to keep something in mind when you start playing around with running GUI applications with Docker -- Just because you can do something doesn't mean that you should do it! You could get carried away trying to run all of your normal desktop application in containers, but why would you want to do that? The idea here is to have a good "Single-User-Docker-Workstation" that you can use wisely to manage application environments that could be problematic or complex to setup otherwise.

Happy computing --dbk

Tags: Docker, NVIDIA-docker, GPU
vv111y

I usually need to do : xhost + , before it will work.

Posted on 2017-07-29 22:03:39
Pines

I am finding this impossible to do from a remote host. Have you tried to access what you created on a remote host? I understand there are some limitations/problems with X11.
The output using your example is this error:

CUDA error at bodysystemcuda_impl.h:183 code=30(cudaErrorUnknown) "cudaGraphicsGLRegisterBuffer(&m_pGRes[i], m_pbo[i], cudaGraphicsMapFlagsNone)"

I have looked and looked and nothing. Have you ever faced a similar problem?

Posted on 2018-12-08 01:17:07
Donald Kinghorn

Hi Pines, That's interesting and it's an obvious thing to want to do! ... but I haven't tried it. ... Oh wait , I have too sort of. It's not so much an X problem as it is a openGL problem. It can be a real pain to get GL working remotely. You have to have it setup on your local X server. I don't remember how to do it but that will get you on the right track. GL has to execute on your machine ... First thing to do is get something like glxgears working

Posted on 2018-12-10 17:00:39
Alex

Hey Donald,

Thanks for the awesome article.
I'm actually facing an error running the second make command (after I installed freeglut3-dev) which seems like:

make: /usr/local/cuda-10.0/bin/nvcc: Command not found
Makefile:304: recipe for target 'bodysystemcuda.o' failed
make: *** [bodysystemcuda.o] Error 127

Although when I run "nvcc" in the container shell it founds the executable.

I'm using Ubuntu18.

Thanks in advance,
AlexZ

Posted on 2019-06-11 13:16:27
Donald Kinghorn

Hi Alex, first this post is pretty old :-) however, really, this stuff should still work. I was thinking about this a couple of days ago, I haven't done this X binding for ages!

... let me check something ... OK, found the problem. The latest nvidia/cuda container image is at cuda 10.1 The Makefile that you have in the samples is configured for 10.0 You could get a newer copy of the cuda samples and try that ... I edited the nbody Makefile in the 10.0 samples that I had and changed
CUDA_PATH ?= /usr/local/cuda-10.0 to just /usr/local/cuda The code built with the 10.1 nvcc. It ran fine with nbody --benchmark but when I tried it with the GL display if dumped a bunch of openGL errors.

I think the first thing to try ti to get a copy of the cuda samples for 10.1 and try that

... let me try something else ... I tried using the tag :10.0-devel that also built the code OK but failed to run with GL errors.

I'm on Ubuntu 19.04 with the latest nvidia driver, I may be seeing a versioning problem or just one of those annoying openGL problems.

This should get you on the right track but if you hit the GL errors that I did it may be a pain to fix

Posted on 2019-06-12 21:41:56
twyeld

Hi again Donald - I am making good progress. I am able to get this running using CUDA 10.1 in Ubuntu 19.04 but not inside a docker container. Even after installing the missing glut dependencies I get the following errors:

root@c70027bab98c:/cuda8/nbody/cuda8/nbody# make
/usr/local/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -ftz=true -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o bodysystemcuda.o -c bodysystemcuda.cu
bodysystemcuda.cu:12:10: fatal error: helper_cuda.h: No such file or directory
#include <helper_cuda.h>

it can't find <helper_cuda.h> but as below shows, it is clearly in the make folder.

root@c70027bab98c:/cuda8/nbody/cuda8/nbody# ls
Makefile bodysystem.h bodysystemcpu_impl.h bodysystemcuda.h doc galaxy_20K.bin nbody.cpp render_particles.cpp tipsy.h
NsightEclipse.xml bodysystemcpu.h bodysystemcuda.cu bodysystemcuda_impl.h findgllib.mk helper_cuda.h readme.txt render_particles.h

and I did set these paths before building/running the container:

export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64\ ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

any ideas?

Cheers, Ted:)

Posted on 2019-06-27 14:14:58
Donald Kinghorn

Hey Ted, This is from some hard-coded version info in the source ... yes, that's annoying! I've hit it several times ... there are some hard-coded paths and compiler versions.

It looks like that container is cuda8
If you look at the tag list tab in docker-hub you can find a huge selection of images. (unfortunately I don't think any of them have the code samples)

You can use one of the official cuda install ".run" files to get just the samples directory for a specific version. It's possible to grep through the samples directory and find paths and such and then edit that. The code will usually build. I gave up on that though and just try to get the samples directory that matches up directly with the cuda version I'm using.

Posted on 2019-06-27 16:14:45
twyeld

Thanks Donald - yes, the cuda8 is a bit confusing - I started out thinking I would need to use the cuda8 package but instead ended up getting it mostly working in cuda10.1.

I solved the missing <helper_cuda.h> by simply making sure the common/inc was in the same tree as the fetch directory - duh!

Now I discover glut is no longer supported in the nvidia-docker2 version! argh

new error:

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
X Error of failed request: GLXBadContext

root@25ea3061be57:/cuda10.1/nbody/cuda10.1/nbody# ldconfig -p | grep -i gl.so
libOpenGL.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libOpenGL.so.0
libOpenGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libOpenGL.so
libGL.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so.1
libGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so
libEGL.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libEGL.so.1
libEGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libEGL.so

removing libGL.so.1 doesn't help...

Any ideas on a getaround?

I know they have implemented virtualGL in place of openGL but there's gotta be a better way?

I can get glxgears running from inside a container but its not clear if it is simply accessing the x-host to run:

#Dockerfile
FROM ubuntu:18.04

# nvidia-container-runtime
ENV NVIDIA_VISIBLE_DEVICES ${NVIDIA_VISIBLE_DEVICES:-all}
ENV NVIDIA_DRIVER_CAPABILITIES ${NVIDIA_DRIVER_CAPABILITIES:+$NVIDIA_DRIVER_CAPABILITIES,}graphics

# install GLX-Gears
RUN apt update && apt install -y --no-install-recommends mesa-utils x11-apps && rm -rf /var/lib/apt/lists/*

#build and run Dockerfile:

xhost +
docker build -t gl_test_gears_ins .

docker run --runtime=nvidia --rm -it --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" gl_test_gears_ins glxgears

But when I run nvidia-smi I don't see any processes using the GPU...

As a simple test, it would be cool to get glxgears using the gpu to both process and run, and then monitoring this as a process from nvidia-smi or (gpustats?)..

your thoughts?

Ted:)

Posted on 2019-06-28 03:39:51
Donald Kinghorn

You are hitting all of my "favorite" problems :-) openGL drives me crazy ... I don't understand it well enough to ever "really" know what is going wrong. But, my understanding is that GL runs on the "local" machine, like when you try to run a remote app over X that is using GL graphics. That is even more confusing when you are running docker because you are mounting your local X socket into the container.

You are right about changes in nvidia-docker2 ( I remember talking with the devs about it ) but again I don't understand the details well enough to sort it out. I have "hacked" my way around it before but I never seem to remember what I did to get things to work so end up fighting with it every time.

This is a good problem to sort out. I'm pretty sure the answer is going to be getting environment variables set correctly (probably in the container and host both) . The end result should be the GPU code running in the container and the GL display code running on the host. That would make a good blog post ... one of my favorite things about writing posts is that they become "external memory" for when I forget how to do something :-) I'm adding this to my list of "short topics" for posts but I don't have time to do it right now.

If you want to hack around with it my best advise is to look at the various env settings for openGL , I think that's the key. --Don

Posted on 2019-06-28 15:44:41
twyeld

haha, you are quite right about posts as "external memory" - I go so far as to create a PDF library of really good one's (like yours) in case the URL disappears!

If I can sort out the GL glitching I will certainly post here again...

Cheers

Posted on 2019-06-29 01:23:57
Donald Kinghorn

Hey Ted I found the secret sauce :-) I'm going to write this up in external memory ... I mean a blog post...
docker run --runtime=nvidia --rm -it -v $HOME/projects:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e XAUTHORITY -e NVIDIA_DRIVER_CAPABILITIES=all nvidia/cuda

That's my usual startup for nvidia-docker2 with X display but I set the environment variables -e DISPLAY and -e XAUTHORITY to get access to the display and the "secret" -e NVIDIA_DRIVER_CAPABILITIES=all this last one did the trick. You still have to install freeglut3 and -dev etc. But this seems to do the trick.

Turns out there are two important env vars for the "nvidia-container-runtime"
NVIDIA_DRIVER_CAPABILITIES
NVIDIA_VISIBLE_DEVICES

With the last one you can set which GPU's are visible to the container 0,1 etc..

For capabilities there is,

Supported driver capabilities
compute: required for CUDA and OpenCL applications.
compat32: required for running 32-bit applications.
graphics: required for running OpenGL and Vulkan applications.
utility: required for using nvidia-smi and NVML.
video: required for using the Video Codec SDK.
display: required for leveraging X11 display.

using "all" just sets everything ... the reason we couldn't get nbody going is because "graphics" is not set by default

Happy 4th of July!

Posted on 2019-07-05 02:17:04
twyeld

nicely solved Donald...

I found this routine which works (using cudagl for the nbody sim and relies on rollback deps from ubuntu 16.04) but makes a mess of my system

https://devblogs.nvidia.com...

So, I am keen to try your method again...

Just need to finish off some other stuff first - back on this next week...

Cheers and bests for the 4th! (its already the 5th here...)

Posted on 2019-07-05 02:36:52