Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/915
Dr Donald Kinghorn (Scientific Computing Advisor )

Docker and NVIDIA-docker on your workstation: Using Graphical Applications

Written on March 21, 2017 by Dr Donald Kinghorn
Share:

This is my fourth post about "Docker and NVIDIA-Docker on your Workstation". In this post I will go through how to attach your X server display to a Docker container so that you can run application that have GUIs or other graphical display output.

It is assumed that you have already installed the host OS, Docker-Engine, and NVIDIA-Docker as described in the "installation post". I also highly recommend that you have setup Docker to use Kernel User-Namespaces.

Here is a list of the first three posts in this series for reference,

I recommend that you do not use X-Window with your Docker containers unless you have configured Docker with User-Namespaces as I have outlined in previous posts. If you do this then you are using X-Window with Docker as your normal log-in user account rather than as root.


How to Run a GUI application from a Docker Container

The most common use of Docker is for servers and other "command-line" applications. What if you want to run an application that needs your desktop display? You can do this. The relevant Docker command flags are,

-v /tmp/.X11-unix:/tmp/.X11-unix
-e DISPLAY=$DISPLAY

The flag -v /tmp/.X11-unix:/tmp/.X11-unix gives the container access to your X socket and -e DISPLAY=$DISPLAY sets the DISPLAY environment variable in the container the same as that of your desktop.

Note: You can add sound capability to a container with --device /dev/snd

Security: I believe that using X-Window with Docker in this way is as secure as if you were running any X-Window application with your normal user account. The reason for this is, if you are running Docker configured with User-Namespaces the way I have outlined then you are running with your normal user account from the perspective of the host!

Example: Compile and run CUDA nbody with OpenGL display output

This example is the kind of thing that I really wanted to be able to do easily with a Docker container. I'll start with the default NVIDIA CUDA 8 image and add the "freeglut3-dev" package to pull the needed dependencies into the container. Then compile nbody in the container and run it with openGL display output.

I have created a directory docker/cuda8 in my home directory and added put the CUDA "samples" source there. I will use this directory in the container.

  1. Start the container

nvidia-docker run -it -v $HOME/docker:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY nvidia/cuda
  1. cd to the nbody source directory and try to run make

cd projects/cuda8/samples/5_Simulations/nbody
...
>>> WARNING - libGL.so not found, ...
 
  1. No GL lib and a lot of other missing libs so lets install "freeglut3-dev" which will pull the dependencies we need.

apt-get update
apt-get install freeglut3-dev

Note: That will add a lot of packages to the image! They will add as a new "layer". We will save this modified container to a new image later.

  1. Try make again

make clean
make
...
Success!
  1. Now run nbody

./nbody

You should be greeted with something like this;
nbody OpenGL GUI

That is screenshot of the running nbody application using the GPU for CUDA acceleration and using your X display to show the openGL output. ... From a Docker container!


How to Save a Modified Container to a new Image

We just made a significant modification to the container we started from the default NVIDIA cuda8-dev image. This will persist as long as we don't remove the container. You can exit from the container and then re-start it and all of your changes will still be there. However, adding the openGL support to the container is a useful thing so saving that as a new image so you (and others) could create new containers from it is a reasonable thing to do.

There are a couple of ways to save your modified container as a new image. I'll show you how to use commit. In a later post I'll go through how to create a Dockerfile and use that to build a new image with the same changes that we apply in the container.

Using docker commit

If you are still connected to the modified container go ahead and exit, then run docker ps -a which will output something like,

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
9fc3da506758        nvidia/cuda         "/bin/bash"         19 hours ago        Exited (0) 18 seconds ago                       awesome_payne

Make note of the container ID. In this case it is 9fc3da506758. To save this as a new image do,

docker commit 9fc3da506758 cuda8-dev-opengl

That will output a sha256 ID for the new image and save it to your image directory with the new name. Do docker images and you should see your saved image. For example,

docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
cuda8-dev-opengl    latest              3022f6beb3ef        18 seconds ago      1.75 GB
alpine              latest              4a415e366388        2 weeks ago         3.99 MB
nvidia/cuda         latest              1cce8839a2c5        2 weeks ago         1.62 GB

Note You can add tags then you commit an image. I could have used something like docker commit 9fc3da506758 cuda8-dev:opengl. Tags are usually used for versioning but you can use naming schemes to suit your taste.

Note You only need to use enough characters from the ID hash to uniquely identify the container or image. For example using 9fc for the example container above would be sufficient.

You can now remove the old modified container with docker rm 9fc You can verify that the container was remove with docker ps -a

To start a new container with the new image we committed do

nvidia-docker run -it -v $HOME/docker:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY cuda8-dev-opengl

That is just scratching the surface of what you can do! It is possible to run many GUI applications including application that use GPU compute acceleration.

I think it is good to keep something in mind when you start playing around with running GUI applications with Docker -- Just because you can do something doesn't mean that you should do it! You could get carried away trying to run all of your normal desktop application in containers, but why would you want to do that? The idea here is to have a good "Single-User-Docker-Workstation" that you can use wisely to manage application environments that could be problematic or complex to setup otherwise.

Happy computing --dbk

Tags: Docker, NVIDIA-docker, GPU
vv111y

I usually need to do : xhost + , before it will work.

Posted on 2017-07-29 22:03:39
Pines

I am finding this impossible to do from a remote host. Have you tried to access what you created on a remote host? I understand there are some limitations/problems with X11.
The output using your example is this error:

CUDA error at bodysystemcuda_impl.h:183 code=30(cudaErrorUnknown) "cudaGraphicsGLRegisterBuffer(&m_pGRes[i], m_pbo[i], cudaGraphicsMapFlagsNone)"

I have looked and looked and nothing. Have you ever faced a similar problem?

Posted on 2018-12-08 01:17:07
Donald Kinghorn

Hi Pines, That's interesting and it's an obvious thing to want to do! ... but I haven't tried it. ... Oh wait , I have too sort of. It's not so much an X problem as it is a openGL problem. It can be a real pain to get GL working remotely. You have to have it setup on your local X server. I don't remember how to do it but that will get you on the right track. GL has to execute on your machine ... First thing to do is get something like glxgears working

Posted on 2018-12-10 17:00:39
Alex

Hey Donald,

Thanks for the awesome article.
I'm actually facing an error running the second make command (after I installed freeglut3-dev) which seems like:

make: /usr/local/cuda-10.0/bin/nvcc: Command not found
Makefile:304: recipe for target 'bodysystemcuda.o' failed
make: *** [bodysystemcuda.o] Error 127

Although when I run "nvcc" in the container shell it founds the executable.

I'm using Ubuntu18.

Thanks in advance,
AlexZ

Posted on 2019-06-11 13:16:27
Donald Kinghorn

Hi Alex, first this post is pretty old :-) however, really, this stuff should still work. I was thinking about this a couple of days ago, I haven't done this X binding for ages!

... let me check something ... OK, found the problem. The latest nvidia/cuda container image is at cuda 10.1 The Makefile that you have in the samples is configured for 10.0 You could get a newer copy of the cuda samples and try that ... I edited the nbody Makefile in the 10.0 samples that I had and changed
CUDA_PATH ?= /usr/local/cuda-10.0 to just /usr/local/cuda The code built with the 10.1 nvcc. It ran fine with nbody --benchmark but when I tried it with the GL display if dumped a bunch of openGL errors.

I think the first thing to try ti to get a copy of the cuda samples for 10.1 and try that

... let me try something else ... I tried using the tag :10.0-devel that also built the code OK but failed to run with GL errors.

I'm on Ubuntu 19.04 with the latest nvidia driver, I may be seeing a versioning problem or just one of those annoying openGL problems.

This should get you on the right track but if you hit the GL errors that I did it may be a pain to fix

Posted on 2019-06-12 21:41:56
twyeld

Hi again Donald - I am making good progress. I am able to get this running using CUDA 10.1 in Ubuntu 19.04 but not inside a docker container. Even after installing the missing glut dependencies I get the following errors:

root@c70027bab98c:/cuda8/nbody/cuda8/nbody# make
/usr/local/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -ftz=true -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o bodysystemcuda.o -c bodysystemcuda.cu
bodysystemcuda.cu:12:10: fatal error: helper_cuda.h: No such file or directory
#include <helper_cuda.h>

it can't find <helper_cuda.h> but as below shows, it is clearly in the make folder.

root@c70027bab98c:/cuda8/nbody/cuda8/nbody# ls
Makefile bodysystem.h bodysystemcpu_impl.h bodysystemcuda.h doc galaxy_20K.bin nbody.cpp render_particles.cpp tipsy.h
NsightEclipse.xml bodysystemcpu.h bodysystemcuda.cu bodysystemcuda_impl.h findgllib.mk helper_cuda.h readme.txt render_particles.h

and I did set these paths before building/running the container:

export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64\ ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

any ideas?

Cheers, Ted:)

Posted on 2019-06-27 14:14:58
Donald Kinghorn

Hey Ted, This is from some hard-coded version info in the source ... yes, that's annoying! I've hit it several times ... there are some hard-coded paths and compiler versions.

It looks like that container is cuda8
If you look at the tag list tab in docker-hub you can find a huge selection of images. (unfortunately I don't think any of them have the code samples)

You can use one of the official cuda install ".run" files to get just the samples directory for a specific version. It's possible to grep through the samples directory and find paths and such and then edit that. The code will usually build. I gave up on that though and just try to get the samples directory that matches up directly with the cuda version I'm using.

Posted on 2019-06-27 16:14:45
twyeld

Thanks Donald - yes, the cuda8 is a bit confusing - I started out thinking I would need to use the cuda8 package but instead ended up getting it mostly working in cuda10.1.

I solved the missing <helper_cuda.h> by simply making sure the common/inc was in the same tree as the fetch directory - duh!

Now I discover glut is no longer supported in the nvidia-docker2 version! argh

new error:

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
X Error of failed request: GLXBadContext

root@25ea3061be57:/cuda10.1/nbody/cuda10.1/nbody# ldconfig -p | grep -i gl.so
libOpenGL.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libOpenGL.so.0
libOpenGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libOpenGL.so
libGL.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so.1
libGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so
libEGL.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libEGL.so.1
libEGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libEGL.so

removing libGL.so.1 doesn't help...

Any ideas on a getaround?

I know they have implemented virtualGL in place of openGL but there's gotta be a better way?

I can get glxgears running from inside a container but its not clear if it is simply accessing the x-host to run:

#Dockerfile
FROM ubuntu:18.04

# nvidia-container-runtime
ENV NVIDIA_VISIBLE_DEVICES ${NVIDIA_VISIBLE_DEVICES:-all}
ENV NVIDIA_DRIVER_CAPABILITIES ${NVIDIA_DRIVER_CAPABILITIES:+$NVIDIA_DRIVER_CAPABILITIES,}graphics

# install GLX-Gears
RUN apt update && apt install -y --no-install-recommends mesa-utils x11-apps && rm -rf /var/lib/apt/lists/*

#build and run Dockerfile:

xhost +
docker build -t gl_test_gears_ins .

docker run --runtime=nvidia --rm -it --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" gl_test_gears_ins glxgears

But when I run nvidia-smi I don't see any processes using the GPU...

As a simple test, it would be cool to get glxgears using the gpu to both process and run, and then monitoring this as a process from nvidia-smi or (gpustats?)..

your thoughts?

Ted:)

Posted on 2019-06-28 03:39:51
Donald Kinghorn

You are hitting all of my "favorite" problems :-) openGL drives me crazy ... I don't understand it well enough to ever "really" know what is going wrong. But, my understanding is that GL runs on the "local" machine, like when you try to run a remote app over X that is using GL graphics. That is even more confusing when you are running docker because you are mounting your local X socket into the container.

You are right about changes in nvidia-docker2 ( I remember talking with the devs about it ) but again I don't understand the details well enough to sort it out. I have "hacked" my way around it before but I never seem to remember what I did to get things to work so end up fighting with it every time.

This is a good problem to sort out. I'm pretty sure the answer is going to be getting environment variables set correctly (probably in the container and host both) . The end result should be the GPU code running in the container and the GL display code running on the host. That would make a good blog post ... one of my favorite things about writing posts is that they become "external memory" for when I forget how to do something :-) I'm adding this to my list of "short topics" for posts but I don't have time to do it right now.

If you want to hack around with it my best advise is to look at the various env settings for openGL , I think that's the key. --Don

Posted on 2019-06-28 15:44:41
twyeld

haha, you are quite right about posts as "external memory" - I go so far as to create a PDF library of really good one's (like yours) in case the URL disappears!

If I can sort out the GL glitching I will certainly post here again...

Cheers

Posted on 2019-06-29 01:23:57
Donald Kinghorn

Hey Ted I found the secret sauce :-) I'm going to write this up in external memory ... I mean a blog post...
docker run --runtime=nvidia --rm -it -v $HOME/projects:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e XAUTHORITY -e NVIDIA_DRIVER_CAPABILITIES=all nvidia/cuda

That's my usual startup for nvidia-docker2 with X display but I set the environment variables -e DISPLAY and -e XAUTHORITY to get access to the display and the "secret" -e NVIDIA_DRIVER_CAPABILITIES=all this last one did the trick. You still have to install freeglut3 and -dev etc. But this seems to do the trick.

Turns out there are two important env vars for the "nvidia-container-runtime"
NVIDIA_DRIVER_CAPABILITIES
NVIDIA_VISIBLE_DEVICES

With the last one you can set which GPU's are visible to the container 0,1 etc..

For capabilities there is,

Supported driver capabilities
compute: required for CUDA and OpenCL applications.
compat32: required for running 32-bit applications.
graphics: required for running OpenGL and Vulkan applications.
utility: required for using nvidia-smi and NVML.
video: required for using the Video Codec SDK.
display: required for leveraging X11 display.

using "all" just sets everything ... the reason we couldn't get nbody going is because "graphics" is not set by default

Happy 4th of July!

Posted on 2019-07-05 02:17:04
twyeld

nicely solved Donald...

I found this routine which works (using cudagl for the nbody sim and relies on rollback deps from ubuntu 16.04) but makes a mess of my system

https://devblogs.nvidia.com...

So, I am keen to try your method again...

Just need to finish off some other stuff first - back on this next week...

Cheers and bests for the 4th! (its already the 5th here...)

Posted on 2019-07-05 02:36:52
twyeld

Well I am back in earnest after a lengthy hiatus from this project.
I will need to redo all of the above to re familiarise myself...

Anyway, the long and short is that I have now been asked to demonstrate how a compiled/linux Unity3D application can run inside a docker container which:
1) accesses and uses the host GPUs for the parallel processing parts - CPU for the sequential...
2) uses the host GPUs also for graphics acceleration - ie - it ports a video stream
3) spin up multiple container versions of the Unity game compile with different seed values
4) be able to monitor CPU and GPU use stats on the server
5) different containers support different platforms (PC, mobile, HMD etc...) with synchronous streaming, with a callback to the main server to change camera angle which updates the video stream, all from a single base image

The compiled game level uses a static database to simulate a combat scenario with different seed values

I have access to a small HPC cluster with nVidia cards running linux and has Unity3D installed and working...

Sounds impossible? I think so!

But, are you prepared to come along for the ride Donald? You are perhaps the only person with the broad understanding of all the necessary technologies to be able to even contemplate such a task...

Cheers, Ted:)

Posted on 2019-09-02 04:59:38
Donald Kinghorn

Hey Ted,
First thing to note is there is some changes on the NVIDIA side. They have deprecated nvidia-docker2. The GPU stuff is now a more integral runtime. This is not a huge change really but the install is a little different and the docker run command has different syntax (they have not migrated their own stuff on NGC to this yet)

This is becoming a very interesting project! I am interested ... but lets see ... here's my initial thoughts
1) doable
2) doable (OpenGL ... maybe tricky but should be doable)
3) easy-ish
4) can monitor and restrict resource allocation per container (no multi-tenancy on GPU) ... should be some open tools for easy monitoring
5) Should be OK ... I think ... there could be gotchas in there (more so than the other parts)

It sounds like quite a bit of work and I'm sure there are gotchas but, yes, I think it is doable. :-)

Posted on 2019-09-04 15:12:51
twyeld

Thanks Donald - will keep you posted!

Posted on 2019-09-07 01:22:56
twyeld

Hi Donald - so this is as far as I have got:
1) built a level using some rudimentary RL/DQN
2) build as both standalone linux and headless
3) created a docker container using gableroux's template
4) run it using: xvfb-run --auto-servernum --server-args='-screen 0 640x480x24' /myBuild/standalone/myproject.x86_64
5) traceback for the headless version:
Found path: /myBuild/headless/ban2heliIndAgentsHeadless.x86_64
Mono path[0] = '/myBuild/headless/ban2heliIndAgentsHeadless_Data/Managed'
Mono config path = '/myBuild/headless/ban2heliIndAgentsHeadless_Data/MonoBleedingEdge/etc'
Preloaded 'libgrpc_csharp_ext.x64.so'
Logging to /root/.config/unity3d/Unity Technologies/Unity Environment/Player.log
6) traceback for the non-headless version:
Found path: /myBuild/standalone/ban2heliIndAgents.x86_64
Mono path[0] = '/myBuild/standalone/ban2heliIndAgents_Data/Managed'
Mono config path = '/myBuild/standalone/ban2heliIndAgents_Data/MonoBleedingEdge/etc'
Preloaded 'libgrpc_csharp_ext.x64.so'
Unable to preload the following plugins:
ScreenSelector.so
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies/Unity Environment
Logging to /root/.config/unity3d/Unity Technologies/Unity Environment/Player.log

My problem now is how to view the actual game level running in the docker container - I need to connect a vnc.

All I can find on this show to run a Unity3D editor in a Docker container - not a standalone game level...

Any ideas?

Ted

Posted on 2019-09-16 05:49:12
Donald Kinghorn

nice progress! I really want to try this but haven't started yet ...
I think you are close. A red flag that goes up for me is PlayerPrefs writing to container /root/ you might hit some snags there??
As far as viewing the level I would think this would work the same way that the editor did.?? This is where you can get snagged up with OpenGL on remote display ... but I would have assumed the editors was a GL app too(?)
First thing I would try/dheck is to run glperf or something like that you make sure you have all of those gotchas worked out. There is an environment variable that is important look at https://www.mesa3d.org/envv...
LIBGL_ALWAYS_INDIRECT

That all I can think of ...

Posted on 2019-09-16 15:49:24
twyeld

OK I have got very close to having a GUI launch from inside a docker container but something is missing.

So far I have followed lots of tutorials and examples (gableroux, jessfraz, mviereck and so on) - and I can get many GUIs working from a docker install - but not a standalone unity3d game level

here is what I have done so far (well this is only 1/100th!):

1) start xhost
xhost +local:root;
2) create a display to use
Xvfb :2 -screen 0 1024x768x16
3) check that the display is running
ps aux | grep X
4) run my docker session
docker run -it --rm -e DISPLAY=:2 \
-v /tmp/.X11-unix:/tmp/.X11-unix <docker_imageid> /bin/bash
5) from inside docker:
root@<containerid>:/app# ./<my_game>.x86_64

from the traceback you can see it is running but no screen pops up?:
Set current directory to /app
Found path: /app/<my_game>.x86_64
Mono path[0] = '/app/<my_game>_Data/Managed'
Mono config path = '/app/<my_game>_Data/MonoBleedingEdge/etc'
Preloaded 'libgrpc_csharp_ext.x64.so'
Unable to preload the following plugins:
ScreenSelector.so
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies/Unity Environment
Logging to /root/.config/unity3d/Unity Technologies/Unity Environment/Player.log

any ideas where I am going wrong?

Posted on 2019-09-17 11:40:08
Donald Kinghorn

something to check ... use ldd on your executable to see the lib dependencies. There may be something missing ?? It's tricky because there may be dependencies in the container or local or both.

I did a quick search on
" Unable to preload the following plugins:
ScreenSelector.so"
Looks like that itself is not a problem, but it brought up some links where other folks had problems that look the same, ... I didn't see any fixes

I'm curious to try this too but not sure when I'll get to it ...

Posted on 2019-09-17 21:59:50
twyeld

I have got a bit closer now using xephyr:

use ps aux | grep X to check what display is free (in this case I am using :2)

Displaynumber=2

Xephyr :$Displaynumber -screen 1280x1024 -extension MIT-SHM -extension XTEST &

docker run --rm \
-e DISPLAY=:$Displaynumber \
-v /tmp/.X11-unix/X$Displaynumber:/tmp/.X11-unix/X$Displaynumber:rw \
--user $(id -u):$(id -g) \
--cap-drop=ALL \
--security-opt=no-new-privileges \
<image_name> ./<my_standalone_game_compile>.x86_64

you get a popup xephyr window and game is definitely running (can ignore "Unable to preload the following plugins: ScreenSelector.so" not used) but no graphics from the game itself...

Posted on 2019-09-18 01:19:29
twyeld

ok - have it working in an xterm - but very slow fps because only CPU - I need to find a way to use the nVidia card

#xterm dockerfile

FROM ubuntu:18.04

RUN apt-get update && apt-get install -y xterm
RUN useradd -ms /bin/bash xterm
USER xterm
WORKDIR /home/xterm
COPY standalone /home/app/

docker build -t xterm .

Xephyr :2 -screen 1280x1024 -extension MIT-SHM

then open another terminal and:
docker run --rm -e DISPLAY=:2 -v /tmp/.X11-unix:/tmp/.X11-unix xterm xterm

inside the xterminal - to find what directory you are in:
pwd

CD ../ to whatever directory your game exec is in and launch:
./<my_game.x86_64>

Posted on 2019-09-19 09:37:08
Donald Kinghorn

OK, that still looks like openGL is being a problem. like it's dropping back to CPU rendering ... I would back off on getting the game to work and first be sure that you can run a simple openGL app (glxgears) on your local display from THAT container. It seems like it should be similar to getting nbody working???
You are obviously close to getting it working ... I'm pretty sure I'm missing something because I'm not in it up to my elbows like you are ...

I would be digging through message logs on both container and host. It seems like a missing lib, permission, or environment var to me ??
I am really curious to try this too but have been busy...

Posted on 2019-09-19 16:52:25