Docker and NVIDIA-docker on your workstation: Using Graphical Applications
Written on March 21, 2017 by Dr Donald KinghornThis is my fourth post about "Docker and NVIDIA-Docker on your Workstation". In this post I will go through how to attach your X server display to a Docker container so that you can run application that have GUIs or other graphical display output.
It is assumed that you have already installed the host OS, Docker-Engine, and NVIDIA-Docker as described in the "installation post". I also highly recommend that you have setup Docker to use Kernel User-Namespaces.
Here is a list of the first three posts in this series for reference,
I recommend that you do not use X-Window with your Docker containers unless you have configured Docker with User-Namespaces as I have outlined in previous posts. If you do this then you are using X-Window with Docker as your normal log-in user account rather than as root.
How to Run a GUI application from a Docker Container
The most common use of Docker is for servers and other "command-line" applications. What if you want to run an application that needs your desktop display? You can do this. The relevant Docker command flags are,
The flag -v /tmp/.X11-unix:/tmp/.X11-unix
gives the container access to your X socket and -e DISPLAY=$DISPLAY
sets the DISPLAY environment variable in the container the same as that of your desktop.
Note: You can add sound capability to a container with --device /dev/snd
Security: I believe that using X-Window with Docker in this way is as secure as if you were running any X-Window application with your normal user account. The reason for this is, if you are running Docker configured with User-Namespaces the way I have outlined then you are running with your normal user account from the perspective of the host!
Example: Compile and run CUDA nbody with OpenGL display output
This example is the kind of thing that I really wanted to be able to do easily with a Docker container. I'll start with the default NVIDIA CUDA 8 image and add the "freeglut3-dev" package to pull the needed dependencies into the container. Then compile nbody in the container and run it with openGL display output.
I have created a directory docker/cuda8
in my home directory and added put the CUDA "samples" source there. I will use this directory in the container.
Start the container
cd
to the nbody source directory and try to runmake
No GL lib and a lot of other missing libs so lets install "freeglut3-dev" which will pull the dependencies we need.
Note: That will add a lot of packages to the image! They will add as a new "layer". We will save this modified container to a new image later.
Try
make
again
Now run
nbody
You should be greeted with something like this;
That is screenshot of the running nbody application using the GPU for CUDA acceleration and using your X display to show the openGL output. ... From a Docker container!
How to Save a Modified Container to a new Image
We just made a significant modification to the container we started from the default NVIDIA cuda8-dev image. This will persist as long as we don't remove the container. You can exit from the container and then re-start it and all of your changes will still be there. However, adding the openGL support to the container is a useful thing so saving that as a new image so you (and others) could create new containers from it is a reasonable thing to do.
There are a couple of ways to save your modified container as a new image. I'll show you how to use commit
. In a later post I'll go through how to create a Dockerfile
and use that to build
a new image with the same changes that we apply in the container.
Using docker commit
If you are still connected to the modified container go ahead and exit
, then run docker ps -a
which will output something like,
Make note of the container ID. In this case it is 9fc3da506758
. To save this as a new image do,
That will output a sha256 ID for the new image and save it to your image directory with the new name. Do docker images
and you should see your saved image. For example,
Note You can add tags then you commit
an image. I could have used something like docker commit 9fc3da506758 cuda8-dev:opengl
. Tags are usually used for versioning but you can use naming schemes to suit your taste.
Note You only need to use enough characters from the ID hash to uniquely identify the container or image. For example using 9fc for the example container above would be sufficient.
You can now remove the old modified container with docker rm 9fc
You can verify that the container was remove with docker ps -a
To start a new container with the new image we committed do
That is just scratching the surface of what you can do! It is possible to run many GUI applications including application that use GPU compute acceleration.
I think it is good to keep something in mind when you start playing around with running GUI applications with Docker -- Just because you can do something doesn't mean that you should do it! You could get carried away trying to run all of your normal desktop application in containers, but why would you want to do that? The idea here is to have a good "Single-User-Docker-Workstation" that you can use wisely to manage application environments that could be problematic or complex to setup otherwise.
Happy computing --dbk
I usually need to do : xhost + , before it will work.
I am finding this impossible to do from a remote host. Have you tried to access what you created on a remote host? I understand there are some limitations/problems with X11.
The output using your example is this error:
CUDA error at bodysystemcuda_impl.h:183 code=30(cudaErrorUnknown) "cudaGraphicsGLRegisterBuffer(&m_pGRes[i], m_pbo[i], cudaGraphicsMapFlagsNone)"
I have looked and looked and nothing. Have you ever faced a similar problem?
Hi Pines, That's interesting and it's an obvious thing to want to do! ... but I haven't tried it. ... Oh wait , I have too sort of. It's not so much an X problem as it is a openGL problem. It can be a real pain to get GL working remotely. You have to have it setup on your local X server. I don't remember how to do it but that will get you on the right track. GL has to execute on your machine ... First thing to do is get something like glxgears working
Hey Donald,
Thanks for the awesome article.
I'm actually facing an error running the second make command (after I installed freeglut3-dev) which seems like:
make: /usr/local/cuda-10.0/bin/nvcc: Command not found
Makefile:304: recipe for target 'bodysystemcuda.o' failed
make: *** [bodysystemcuda.o] Error 127
Although when I run "nvcc" in the container shell it founds the executable.
I'm using Ubuntu18.
Thanks in advance,
AlexZ
Hi Alex, first this post is pretty old :-) however, really, this stuff should still work. I was thinking about this a couple of days ago, I haven't done this X binding for ages!
... let me check something ... OK, found the problem. The latest nvidia/cuda container image is at cuda 10.1 The Makefile that you have in the samples is configured for 10.0 You could get a newer copy of the cuda samples and try that ... I edited the nbody Makefile in the 10.0 samples that I had and changed
CUDA_PATH ?= /usr/local/cuda-10.0 to just /usr/local/cuda The code built with the 10.1 nvcc. It ran fine with nbody --benchmark but when I tried it with the GL display if dumped a bunch of openGL errors.
I think the first thing to try ti to get a copy of the cuda samples for 10.1 and try that
... let me try something else ... I tried using the tag :10.0-devel that also built the code OK but failed to run with GL errors.
I'm on Ubuntu 19.04 with the latest nvidia driver, I may be seeing a versioning problem or just one of those annoying openGL problems.
This should get you on the right track but if you hit the GL errors that I did it may be a pain to fix
Hi again Donald - I am making good progress. I am able to get this running using CUDA 10.1 in Ubuntu 19.04 but not inside a docker container. Even after installing the missing glut dependencies I get the following errors:
root@c70027bab98c:/cuda8/nbody/cuda8/nbody# make
/usr/local/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -ftz=true -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o bodysystemcuda.o -c bodysystemcuda.cu
bodysystemcuda.cu:12:10: fatal error: helper_cuda.h: No such file or directory
#include <helper_cuda.h>
it can't find <helper_cuda.h> but as below shows, it is clearly in the make folder.
root@c70027bab98c:/cuda8/nbody/cuda8/nbody# ls
Makefile bodysystem.h bodysystemcpu_impl.h bodysystemcuda.h doc galaxy_20K.bin nbody.cpp render_particles.cpp tipsy.h
NsightEclipse.xml bodysystemcpu.h bodysystemcuda.cu bodysystemcuda_impl.h findgllib.mk helper_cuda.h readme.txt render_particles.h
and I did set these paths before building/running the container:
export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64\ ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
any ideas?
Cheers, Ted:)
Hey Ted, This is from some hard-coded version info in the source ... yes, that's annoying! I've hit it several times ... there are some hard-coded paths and compiler versions.
It looks like that container is cuda8
If you look at the tag list tab in docker-hub you can find a huge selection of images. (unfortunately I don't think any of them have the code samples)
You can use one of the official cuda install ".run" files to get just the samples directory for a specific version. It's possible to grep through the samples directory and find paths and such and then edit that. The code will usually build. I gave up on that though and just try to get the samples directory that matches up directly with the cuda version I'm using.
Thanks Donald - yes, the cuda8 is a bit confusing - I started out thinking I would need to use the cuda8 package but instead ended up getting it mostly working in cuda10.1.
I solved the missing <helper_cuda.h> by simply making sure the common/inc was in the same tree as the fetch directory - duh!
Now I discover glut is no longer supported in the nvidia-docker2 version! argh
new error:
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
X Error of failed request: GLXBadContext
root@25ea3061be57:/cuda10.1/nbody/cuda10.1/nbody# ldconfig -p | grep -i gl.so
libOpenGL.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libOpenGL.so.0
libOpenGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libOpenGL.so
libGL.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so.1
libGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so
libEGL.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libEGL.so.1
libEGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libEGL.so
removing libGL.so.1 doesn't help...
Any ideas on a getaround?
I know they have implemented virtualGL in place of openGL but there's gotta be a better way?
I can get glxgears running from inside a container but its not clear if it is simply accessing the x-host to run:
#Dockerfile
FROM ubuntu:18.04
# nvidia-container-runtime
ENV NVIDIA_VISIBLE_DEVICES ${NVIDIA_VISIBLE_DEVICES:-all}
ENV NVIDIA_DRIVER_CAPABILITIES ${NVIDIA_DRIVER_CAPABILITIES:+$NVIDIA_DRIVER_CAPABILITIES,}graphics
# install GLX-Gears
RUN apt update && apt install -y --no-install-recommends mesa-utils x11-apps && rm -rf /var/lib/apt/lists/*
#build and run Dockerfile:
xhost +
docker build -t gl_test_gears_ins .
docker run --runtime=nvidia --rm -it --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" gl_test_gears_ins glxgears
But when I run nvidia-smi I don't see any processes using the GPU...
As a simple test, it would be cool to get glxgears using the gpu to both process and run, and then monitoring this as a process from nvidia-smi or (gpustats?)..
your thoughts?
Ted:)
You are hitting all of my "favorite" problems :-) openGL drives me crazy ... I don't understand it well enough to ever "really" know what is going wrong. But, my understanding is that GL runs on the "local" machine, like when you try to run a remote app over X that is using GL graphics. That is even more confusing when you are running docker because you are mounting your local X socket into the container.
You are right about changes in nvidia-docker2 ( I remember talking with the devs about it ) but again I don't understand the details well enough to sort it out. I have "hacked" my way around it before but I never seem to remember what I did to get things to work so end up fighting with it every time.
This is a good problem to sort out. I'm pretty sure the answer is going to be getting environment variables set correctly (probably in the container and host both) . The end result should be the GPU code running in the container and the GL display code running on the host. That would make a good blog post ... one of my favorite things about writing posts is that they become "external memory" for when I forget how to do something :-) I'm adding this to my list of "short topics" for posts but I don't have time to do it right now.
If you want to hack around with it my best advise is to look at the various env settings for openGL , I think that's the key. --Don
haha, you are quite right about posts as "external memory" - I go so far as to create a PDF library of really good one's (like yours) in case the URL disappears!
If I can sort out the GL glitching I will certainly post here again...
Cheers
Hey Ted I found the secret sauce :-) I'm going to write this up in external memory ... I mean a blog post...
docker run --runtime=nvidia --rm -it -v $HOME/projects:/projects -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e XAUTHORITY -e NVIDIA_DRIVER_CAPABILITIES=all nvidia/cuda
That's my usual startup for nvidia-docker2 with X display but I set the environment variables -e DISPLAY and -e XAUTHORITY to get access to the display and the "secret" -e NVIDIA_DRIVER_CAPABILITIES=all this last one did the trick. You still have to install freeglut3 and -dev etc. But this seems to do the trick.
Turns out there are two important env vars for the "nvidia-container-runtime"
NVIDIA_DRIVER_CAPABILITIES
NVIDIA_VISIBLE_DEVICES
With the last one you can set which GPU's are visible to the container 0,1 etc..
For capabilities there is,
Supported driver capabilities
compute: required for CUDA and OpenCL applications.
compat32: required for running 32-bit applications.
graphics: required for running OpenGL and Vulkan applications.
utility: required for using nvidia-smi and NVML.
video: required for using the Video Codec SDK.
display: required for leveraging X11 display.
using "all" just sets everything ... the reason we couldn't get nbody going is because "graphics" is not set by default
Happy 4th of July!
nicely solved Donald...
I found this routine which works (using cudagl for the nbody sim and relies on rollback deps from ubuntu 16.04) but makes a mess of my system
https://devblogs.nvidia.com...
So, I am keen to try your method again...
Just need to finish off some other stuff first - back on this next week...
Cheers and bests for the 4th! (its already the 5th here...)
Well I am back in earnest after a lengthy hiatus from this project.
I will need to redo all of the above to re familiarise myself...
Anyway, the long and short is that I have now been asked to demonstrate how a compiled/linux Unity3D application can run inside a docker container which:
1) accesses and uses the host GPUs for the parallel processing parts - CPU for the sequential...
2) uses the host GPUs also for graphics acceleration - ie - it ports a video stream
3) spin up multiple container versions of the Unity game compile with different seed values
4) be able to monitor CPU and GPU use stats on the server
5) different containers support different platforms (PC, mobile, HMD etc...) with synchronous streaming, with a callback to the main server to change camera angle which updates the video stream, all from a single base image
The compiled game level uses a static database to simulate a combat scenario with different seed values
I have access to a small HPC cluster with nVidia cards running linux and has Unity3D installed and working...
Sounds impossible? I think so!
But, are you prepared to come along for the ride Donald? You are perhaps the only person with the broad understanding of all the necessary technologies to be able to even contemplate such a task...
Cheers, Ted:)
Hey Ted,
First thing to note is there is some changes on the NVIDIA side. They have deprecated nvidia-docker2. The GPU stuff is now a more integral runtime. This is not a huge change really but the install is a little different and the docker run command has different syntax (they have not migrated their own stuff on NGC to this yet)
This is becoming a very interesting project! I am interested ... but lets see ... here's my initial thoughts
1) doable
2) doable (OpenGL ... maybe tricky but should be doable)
3) easy-ish
4) can monitor and restrict resource allocation per container (no multi-tenancy on GPU) ... should be some open tools for easy monitoring
5) Should be OK ... I think ... there could be gotchas in there (more so than the other parts)
It sounds like quite a bit of work and I'm sure there are gotchas but, yes, I think it is doable. :-)
Thanks Donald - will keep you posted!
Hi Donald - so this is as far as I have got:
1) built a level using some rudimentary RL/DQN
2) build as both standalone linux and headless
3) created a docker container using gableroux's template
4) run it using: xvfb-run --auto-servernum --server-args='-screen 0 640x480x24' /myBuild/standalone/myproject.x86_64
5) traceback for the headless version:
Found path: /myBuild/headless/ban2heliIndAgentsHeadless.x86_64
Mono path[0] = '/myBuild/headless/ban2heliIndAgentsHeadless_Data/Managed'
Mono config path = '/myBuild/headless/ban2heliIndAgentsHeadless_Data/MonoBleedingEdge/etc'
Preloaded 'libgrpc_csharp_ext.x64.so'
Logging to /root/.config/unity3d/Unity Technologies/Unity Environment/Player.log
6) traceback for the non-headless version:
Found path: /myBuild/standalone/ban2heliIndAgents.x86_64
Mono path[0] = '/myBuild/standalone/ban2heliIndAgents_Data/Managed'
Mono config path = '/myBuild/standalone/ban2heliIndAgents_Data/MonoBleedingEdge/etc'
Preloaded 'libgrpc_csharp_ext.x64.so'
Unable to preload the following plugins:
ScreenSelector.so
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies/Unity Environment
Logging to /root/.config/unity3d/Unity Technologies/Unity Environment/Player.log
My problem now is how to view the actual game level running in the docker container - I need to connect a vnc.
All I can find on this show to run a Unity3D editor in a Docker container - not a standalone game level...
Any ideas?
Ted
nice progress! I really want to try this but haven't started yet ...
I think you are close. A red flag that goes up for me is PlayerPrefs writing to container /root/ you might hit some snags there??
As far as viewing the level I would think this would work the same way that the editor did.?? This is where you can get snagged up with OpenGL on remote display ... but I would have assumed the editors was a GL app too(?)
First thing I would try/dheck is to run glperf or something like that you make sure you have all of those gotchas worked out. There is an environment variable that is important look at https://www.mesa3d.org/envv...
LIBGL_ALWAYS_INDIRECT
That all I can think of ...
OK I have got very close to having a GUI launch from inside a docker container but something is missing.
So far I have followed lots of tutorials and examples (gableroux, jessfraz, mviereck and so on) - and I can get many GUIs working from a docker install - but not a standalone unity3d game level
here is what I have done so far (well this is only 1/100th!):
1) start xhost
xhost +local:root;
2) create a display to use
Xvfb :2 -screen 0 1024x768x16
3) check that the display is running
ps aux | grep X
4) run my docker session
docker run -it --rm -e DISPLAY=:2 \
-v /tmp/.X11-unix:/tmp/.X11-unix <docker_imageid> /bin/bash
5) from inside docker:
root@<containerid>:/app# ./<my_game>.x86_64
from the traceback you can see it is running but no screen pops up?:
Set current directory to /app
Found path: /app/<my_game>.x86_64
Mono path[0] = '/app/<my_game>_Data/Managed'
Mono config path = '/app/<my_game>_Data/MonoBleedingEdge/etc'
Preloaded 'libgrpc_csharp_ext.x64.so'
Unable to preload the following plugins:
ScreenSelector.so
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies
PlayerPrefs - Creating folder: /root/.config/unity3d/Unity Technologies/Unity Environment
Logging to /root/.config/unity3d/Unity Technologies/Unity Environment/Player.log
any ideas where I am going wrong?
something to check ... use ldd on your executable to see the lib dependencies. There may be something missing ?? It's tricky because there may be dependencies in the container or local or both.
I did a quick search on
" Unable to preload the following plugins:
ScreenSelector.so"
Looks like that itself is not a problem, but it brought up some links where other folks had problems that look the same, ... I didn't see any fixes
I'm curious to try this too but not sure when I'll get to it ...
I have got a bit closer now using xephyr:
use ps aux | grep X to check what display is free (in this case I am using :2)
Displaynumber=2
Xephyr :$Displaynumber -screen 1280x1024 -extension MIT-SHM -extension XTEST &
docker run --rm \
-e DISPLAY=:$Displaynumber \
-v /tmp/.X11-unix/X$Displaynumber:/tmp/.X11-unix/X$Displaynumber:rw \
--user $(id -u):$(id -g) \
--cap-drop=ALL \
--security-opt=no-new-privileges \
<image_name> ./<my_standalone_game_compile>.x86_64
you get a popup xephyr window and game is definitely running (can ignore "Unable to preload the following plugins: ScreenSelector.so" not used) but no graphics from the game itself...
ok - have it working in an xterm - but very slow fps because only CPU - I need to find a way to use the nVidia card
#xterm dockerfile
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y xterm
RUN useradd -ms /bin/bash xterm
USER xterm
WORKDIR /home/xterm
COPY standalone /home/app/
docker build -t xterm .
Xephyr :2 -screen 1280x1024 -extension MIT-SHM
then open another terminal and:
docker run --rm -e DISPLAY=:2 -v /tmp/.X11-unix:/tmp/.X11-unix xterm xterm
inside the xterminal - to find what directory you are in:
pwd
CD ../ to whatever directory your game exec is in and launch:
./<my_game.x86_64>
OK, that still looks like openGL is being a problem. like it's dropping back to CPU rendering ... I would back off on getting the game to work and first be sure that you can run a simple openGL app (glxgears) on your local display from THAT container. It seems like it should be similar to getting nbody working???
You are obviously close to getting it working ... I'm pretty sure I'm missing something because I'm not in it up to my elbows like you are ...
I would be digging through message logs on both container and host. It seems like a missing lib, permission, or environment var to me ??
I am really curious to try this too but have been busy...