Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/1097
Dr Donald Kinghorn (Scientific Computing Advisor )

The Best Way To Install Ubuntu 16.04 with NVIDIA Drivers and CUDA

Written on January 19, 2018 by Dr Donald Kinghorn
Share:

In this post I'll be going over details of Installing Ubuntu 16.04 including the NVIDIA display driver and, optionally, NVIDIA CUDA. I have found the method presented here to be the most likely to succeed no matter what hardware configuration you are installing onto.

I usually don't like to start a blog post with the words "The Best". However, I have to confess that I often do searches myself that start with those words. Installing Linux can sometimes be frustrating especially if you are installing onto new "bleeding edge" hardware or something with an unusual configuration. I have been installing Linux since it has existed. The early days were very challenging but great fun. Modern Linux distributions, unfortunately, can sometimes still be challenging but with out the fun part. Linux is usually very easy to install and configure but when it's not it can be very frustrating. The installation method I present here is, in general, my preferred method for installing Linux. In particular this post details how I nearly always install Ubuntu. I'm doing Ubuntu 16.04 because it is a very solid base platform for a lot of applications and is widely used. I will be using this base install for the docker plus NVIDIA-docker and NGC docker registry series of posts I will be doing soon.

If you have run into difficulties doing an Ubuntu 16.04 (or any Linux install) you might want to try what I outline in this post. I've used this to install on a large variety of hardware configurations and it is nearly always successful, -- quick and trouble free. Note, that I said "nearly always" :-) I never cease to be amazed at how troublesome a seemingly simple task can become when computers are involved.


Installing a Desktop with Ubuntu 16.04 from the "Server" base image

Install Ubuntu from the server base. You might be asking yourself -- if we are going to do a desktop setup why are we installing from the server media which will give us a text based console? Here are some reasons for doing it this way,

Why install from server?

  • Server install image is small and downloads easily. (826MB vs 1.7GB for a typical desktop image).

  • A minimal server install is fast and easy and it's simple to add any desktop environment you want on top of that.

  • A small server install is fast and easy to update to current application versions. The software you install after that will be up-to-date rather than the old versions that would be on the desktop install media that would need to be updated.

  • It's fairly simple to script a complete install that starts from a simple server base. [I will do this in the first post on doing a docker setup to use the NVIDIA NCG registry.]

  • It's a reasonable base for docker or VM images that you may create later so it's a good idea to be familiar with it.

  • A server install is "usually" much more likely to work if you are using "bleeding edge" hardware or unusual configurations.

That last reason is actually why I do it this way! I often have a pile of steaming fresh ... hardware ... to test. That hardware may be pre-production samples etc.. I've installed Linux thousands of times, and that's not counting automated provisioning. I usually have best luck with installs by keeping things simple and then build on that. [ I had an Ubuntu 18.04 pre alpha 1 install running Tensorflow test jobs on pre release hardware with a full docker and NVIDIA-docker v2 configuration in about an hour ... and none of that is supported yet! ("don't try this at home" :-) ]

Note! Installing from the full desktop live media can be very simple "when it works" and it often does work just fine. It can sometimes be frustrating trying to get proprietary graphics drivers installed and working right etc.. Don't hesitate to try the install that way, it may work great but, if it doesn't, then knowing how to fall back to a simple base install can save you some grief and frustration.

Step 0) Prepare your hardware and make backups if needed.

I'm assuming that you are doing a clean install. That could be installing on new hardware or reinstalling over a systems that you don't need to save any data from. If you have anything important on the disk you are going to use for the install please back it up! If you have secondary drives or data drives that you want to preserve and use in the new install it is a good idea to disconnect them during your fresh install so that you don't accidentally overwrite them. You can reattach and mount them in your new install later.

Step 1) Get Ubuntu

You can get the Ubuntu server image from Canonical's site. Use the 16.04.3 LTS download. I do not recommend 17.10. I have used it but it has caused trouble for some people and it will go away soon. 16.04 LTS is solid and well supported and will remain that way for several more years. If you use the HWE kernel it will work fine with recent hardware. Ubuntu is released as an LTS (long term support) version every 2 years. The next LTS will be 18.04 due to be released at the end of March. It will be a while before that is fully supported after it's released. I will do a post on how to update or reinstall after it's out.

Step 2) Boot the install image in UEFI mode

I recommend you use UEFI for your install. If you have been doing this for a while you may be comfortable with legacy MBR installs but it's probably time to try UEFI. If you are using new hardware some of it may not even work correctly if you don't boot to UEFI. Be sure you BIOS has "secure boot" disabled. You might be able to get your install to work with it enabled but you will have mysterious errors when you try to install the NVIDIA driver kernel modules.

To start your install you will need to know the magic key to press at startup to get a boot device select screen. Unfortunately that's not standard among different motherboard manufacturers so you will need to look it up if you don't know what it is. When you get to that boot selection screen you may have a choice of boot partitions on the install device. One of them should be labeled UEFI.

Step 3) Install with the HWE kernel

I recommend using the HWE kernel (Hardware Enhanced). This will be a more recent stable kernel with better support for newer hardware (remember the 16 in 16.04 stands for the year 2016). Currently the kernel from the install will be a 4.10 version but will update after the install to an LTS 4.13 kernel. That has the latest patches for various Intel flaws and security issues. [I'm picking on Intel obviously but, this applies to AMD and ARM CPU's too.] The 4.13 kernel seems to be very good. I've observed that it gives better performance for most systems than the 4.10 or 4.4 kernels (at least on current hardware that I've tested). The server install will be an HWE 4.10 kernel that will update to 4.13.

HWE Kernel boot

Step 4.1) Proceed with the install

Note: if you start the install and have corrupted video then you may need to do what I suggest in Step 5) below to get the install started correctly.

Start install
Once you have started the install you will be guided though several question that will be pretty obvious ... until, you get to "Partition disks"

Step 4.2) Disk partitions

I recommend that you do manual disk partitioning. It's not hard and and if you let the installer do it manually you may get some surprises if you are not careful! One insane thing that the Ubuntu installer will do with automatic partitioning is to create a swap partition equal to the size of your physical memory. Years ago when a system may have 4-8GB of memory that was a recommend practice. I recently did a quick install on a system that had 128GB of memory and in my rush I selected automatic partitioning and didn't check what it had configured. It created a 128GB swap file on my 256GB SSD!

Disk partitioning is an area where people seem to often disagree. You used to be able to tell a systems that I had installed by the partition layout, it was almost a signature. That was the old days! On a modern system I highly recommend that you use an SSD device for your OS install and keep the partitioning dirt simple. You can add, format and mount data disks and such later if needed. Just keep thinks simple. You will need a small EFI partition, a small swap partition is good, and the rest of the disk can be your system root partition. This is in my opinion the best way to layout a modern SSD.

Manual partitioning

The partitioner on the server install is pretty good and easy to use. Select the disk you are using for the install and, delete any old partitions if needed.

  • If it's a new device selecting it will ask if you want to make a new partition table -- do that.

  • Then select "Free space" and create your EFI partition (for the the UEFI boot information). Select partition type EFI. Add it to the beginning of the disk and I recommend a size of 256MB.

  • Next select the "Free space" again and add a partition of type swap to the end of the disk space. I used to normally add a 16GB swap partition regardless of how much physical memory is in the system but now I think it is fine to just create a 2GB swap partition just for those occasions that the systems expects to have one. [Note: It's not required to have a swap partition but I think it is still good practice. I noticed that Ubuntu 18.04 adds a 2GB swap file by default with automatic partitioning. That's a big improvement over the old installer! A swap file can be created and mounted after the install if you think you need it for some reason.]

  • Next, select the "Free space" again and use the remaining storage space for your OS root partition (/). For this partition you can use the default EXT4 type and mount point /

  • Then "Finish partitioning and write changes to disk"

Partitions

At this point the installation will will proceed with just a couple more simple questions. When you get to the "Software selection" screen I recommend that you add the OpenSSH server.

Add ssh server

After this the install will finish and you will be ready to boot into the OS.

Step 5) Potential problem number 1!

You might need "nomodeset" until you get GPU drivers installed! You might have this problem during the install too. What I describe here will "usually" take care of this.

When you reboot the system may come up into a corrupted screen or it may hang! This is because the proprietary driver is not installed for the NVIDIA card (which I presume you have in the system). This is most often caused by the system seeing your great video card and then trying to bring up a console in a fancy "frame-buffer". RANT-ON There is no reason for that on a server install! ... RANT-OFF

Here's how you get around this if you encounter it,

When you get to the Grub selection screen type e That will let you edit the kernel boot parameter line (for this boot only). You need to find the line that starts "Linux" go to the end of that line and add nomodeset Then press F10 to boot. That should get you to a login screen. We will have everything installed including the NVIDIA driver before the next boot so this should be the only time you have to do this and there should be no reason to have to add nomodeset permanently.

nomodeset

nomodeset end

You should now have a login prompt waiting for you!

Step 6) Run updates, add your desktop of choice

Login to your system. Now we'll add a bunch of the good stuff!

First run updates,

sudo apt-get update
sudo apt-get dist-upgrade

You will probably get a kernel update but don't reboot yet!

Select and install your desktop of choice

We will use a very handy command that is is installed by default with a server install.

sudo tasksel

With tasksel you you have a lot of options available. It's a really nice tool! The image below show some of the tasksel options. I have selected "Ubuntu MATE desktop". That's my current favorite Linux desktop.
tasksel image

After you make your selection hit return. In the screen-shot below you will see that it is installing 1420 new packages!
1420 packages

When those packages finish downloading and installing you will be able to reboot into your new desktop, but ...

Don't reboot yet! We want to get the NVIDIA display driver installed BEFORE we boot to a GUI desktop.

Step 7) Install the NVIDIA display driver

There are a few methods for installing the proprietary NVIDIA display driver. For Ubuntu installs the one I recommend is to install the latest driver by adding the the graphics driver ppa repository. This is well maintained and you will be able to get the latest driver (or older ones if needed). I recommend you use this method even if you want to do a NVIDIA CUDA install as I describe in a section near the end of this post.

The command to add graphics-drivers ppa is,

sudo add-apt-repository ppa:graphics-drivers/ppa

Before you install the driver is good to be certain dkms (dynamic kernel module support) is installed. I also install a few extra packages that I know I will want now too.

sudo apt-get install dkms synaptic emacs build-essential

I think Synaptic is a great package management interface, build-essential will install a good base of development packages and yes, I like emacs! :-)

Now to install the driver,

sudo apt-get update
sudo apt-get install nvidia-387

That would install the latest driver version 387 (as of this writing). That will work with NVIDIA Volta and lower cards.

This install should be robust. New kernels or drivers should automatically rebuild with dkms.

Now you can reboot!

sudo shutdown -r Now

When your system comes up you should be greeted with a graphical login manager. You will notice that you get a lot of messages on the screen during boot. That's because we installed from server. A typical desktop install form live media will be configured to hide all of that information from you. Personally I like to see it! You will sometimes see some warnings or errors. Those are almost always artifacts of startup ordering or some such no concerning events. However, if there is a real problem on the system those messages can give you useful information for diagnoses.

Step 8) Optional: enable NetworkManager on install interface

One artifact of doing an install from server is that the network interface that is configured during the install will not be using NetworkManager. This is because the server install doesn't have NetworkManager! That's a good thing for a server because you most likely want to manually configure your network. NetworkManager can be useful, especially for systems like laptops where there may be frequent interface switching. NetworkManager is a good system tool and it can be used to give a user other than root the ability to turn on and off the interfaces and it handles wifi well. You can switch your system to NetworkManager control with the following commands (I'm using sed to edit the files but you could do this with a "normal" editor but you do have be root i.e. use sudo)

Move manual network config that happened from server install to NetworkManager control.
This sed line comments out the primary nic interface.

sudo sed -i '/The primary network interface/,/^$/ s/^/#/' /etc/network/interfaces

This line enables NetworkManager for everything

sudo sed -i 's/managed=false/managed=true/' /etc/NetworkManager/NetworkManager.conf

You can reboot after this to check that your network comes up OK.

Step 9) Optional: NVIDIA CUDA install

The NVIDIA display driver in the CUDA 9.1 install repository is nvidia-387 which is the current driver as of this writing. It is the same major version as the driver we installed in Step 7) above. This means that we can do an easy CUDA install from the NVIDIA CUDA repositories even though it will reinstall the display driver. This will make the CUDA install very simple. [Note: that I tried using the CUDA install instead of the ppa driver method in Step 7) and the driver failed to install correctly. I recommend you do your base install as I outlined above including the NVIDIA driver install from the ppa before you do the CUDA install.]

Note: When I am scripting an automated system install I add CUDA manually i.e. using the "run" files. That is a convenient way to install multiple version of CUDA on the same system too. However, going forward I will be recommending using docker to support multiple CUDA versions!

Install and configure NVIDIA CUDA

If you have followed the Ubuntu install method in this post you will have a GUI desktop on your system that is working correctly with the NVIDIA display driver. You can do the install easily from your desktop.

  • Get the latest CUDA repo install deb for Ubuntu 16.04 . I recommend using the "deb(network)" package. If you download this file with firefox it should offer to open the file with "GDebi Package Installer". That is a great utility, go ahead and use it. [If you don't want to use GDebi or don't have the option, then you can just download the deb file and install with dpkg i.e.

    sudo dpkg -i cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
    ]

  • Next you will want to download and install the signing key for the repo.

On the NVIDIA CUDA download page there are instructions that show this

sudo apt-key adv --fetch-keys http://developer...
Uggg! that is using http not https for downloading the keys. That kind of bugs me! That command uses curl by default and curl is a little out of date on Ubuntu 16.04 I recommend that you install gnupg-curl. This will allow that key download command to work with https.
sudo apt-get install gnupg-curl

then do
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

  • Now install CUDA

sudo apt-get update
sudo apt-get install cuda
  • Configure CUDA for use on your system

    • I like to configure CUDA for all users. To do this create the file (use sudo and a text editor of your choice)

    /etc/profile.d/cuda.sh

    with the following content,

    export PATH=$PATH:/usr/local/cuda/bin
    export CUDADIR=/usr/local/cuda
    export GLPATH=/usr/lib/

    Also create the file,

    /etc/ld.so.conf.d/cuda.conf

    and add the line,

    /usr/local/cuda/lib64

    Then run

    sudo ldconfig

The next time you login CUDA will be on your path and ready to use. If you want to load that environment right now without logging out the just do,

source /etc/profile.d/cuda.sh
.


Happy computing --dbk

Tags: Ubuntu 16.04, NVIDIA, CUDA, Install
ep5tv

Propitiatory??? Not proprietary?

Posted on 2018-01-28 12:38:39
Donald Kinghorn

I keep forgetting to monitor posts! .... not sure what you are asking exactly ... You are always free to use anything that I write ... NVIDIA is mixed, some of their stuff is released under an open license and some of it is proprietary in a very bad way. I'm thinking the use restriction in the driver EULA ... If there are restrictions on things that I write about imposed by third parties I try to make it clear.

I'm a strong believer in being open. I'm a scientist and a Linux enthusiast and an ex-hippy :-). But, I don't begrudge anyone from making a living by charging for their work and I will pay for software if I like and it's useful to me. I wish there was more commercial software with Linux support ... rant, rant ...

Posted on 2018-03-19 16:24:59
ep5tv

Not asking, actually. You used the word, "propitiatory", which relates to atonement or apology. Clearly, the intent was "proprietary", as in the nature of property. And we all know where that mindset leads...

I, too, believe in openness. My charitable foundation produces educational content for radio and television and charges for none of it. None at all. Examples can be seen at www.ep5.org, if you'll forgive the proprietary plug!

As for open software, here's a radical idea. Trash what's in place now and begin over with a single hardware platform, a single operating system, a single programming language, etc. Instead of dozens of competing adversaries all striving to attain mediocrity and falling short, the entire economy benefits from an IT ecosystem that is the best that the human mind can create. No one in his right or wrong mind would ever truly argue that what we have now is the very best of which we are capable. Most of it . . . well . . . would you want Windoze to be remembered as the human species' finest achievement?

Posted on 2018-03-19 16:34:35
Donald Kinghorn

Ha! I never did catch that spelling oops, even when you pointed it out, I'm sure it was from auto-correct ... It just never registered :-) Fixed ... Thanks!

... and thumbs up on the ep5.org

Posted on 2018-03-20 18:25:32
James Toast

I have been trying for weeks with no success. I was finally able to install CUDA 9.1 on my Xubuntu desktop by following your steps 7 and 9.

Thank you very much Donald!

Posted on 2018-04-04 03:35:41
Donald Kinghorn

You are most welcome! It can be surprisingly difficult to get things working right sometimes. I've been having a lot of trouble lately because of motherboard manufactures making a mess of their BIOS with the (broken) patches from Intel.

Enjoy your CUDA setup! --Don

Posted on 2018-04-04 15:43:56
coarist

I found your post by searching the web. Before I found your post I had big problem installing latest Nvidia driver and Cuda toolkit spending a lot of time on different iterations, steps and rebooting. The method you posted is, to me,

(1) most up-to-date driver and Cuda for Ubuntu 16.04

(2) without having to modify configuration file to disable display card <and rebooting="" into="" lower="" graphics="" mode="">

(3) Pulls from Ubuntu repository so risk is lowest.

Thanks a lot for putting it together. Very useful for me.

Posted on 2018-04-12 14:32:21
coarist

Best guide to follow for quickly and painlessly getting Nvidia driver and Nvidia Cuda up and running on Ubuntu 16.

Posted on 2018-04-12 15:04:18
Ricjon Su

created an account just to say you're the man! Spent hours googling how to properly install nvidia driver and cuda to no avail, but yours worked! thanks!

Posted on 2018-05-27 13:29:17
George Avirappattu

Best post: I spent my whole weekend trying to get NVIDIA (GTX 1080 Ti) drivers and CUDA installed on the newly installed Ubuntu OS for my HP-Omen (Intel 8th generation processor) with NVIDIA GPU. After looking around on all different posts and tutorials on the Web and unsuccessful attempts. Then followed this posting almost verbatim (disappointed I can't have this figured on a Ubuntu 18.04 or nvidia-396, but none of those combinations successfully installed for me). Thank you so much Dr. Kinghorn.

-George

Posted on 2018-06-11 17:10:16
Donald Kinghorn

I got a CUDA 9.2 setup on Ubuntu 18.04 today ... I should have a post up by the end of the week

Posted on 2018-06-13 05:15:13
Donald Kinghorn

Thanks (everybody!) :-) I'm glad it's been helpful. I did a very similar post for Ubuntu 18.04
https://www.pugetsystems.co...

I did not do CUDA install for 18.04 since it is not officially supported by NVIDIA yet, ...however ... I have started working on that as an unofficial install. I already hit a couple of snags but I may have it posted by Friday or Monday. I like 18.04 (and I am using it) but it might be better to wait until it hits 18.04.1 before committing to it in a "production" environment. Best wishes to all --Don

Posted on 2018-06-13 00:31:01
jpvm

Just curious if I should be worried about the output of ldconfig after making the cuda.conf file.

I get:

/sbin/ldconfig.real: /usr/lib/libgsl.so.23 is not a symbolic link

/sbin/ldconfig.real: /usr/lib/libgslcblas.so.0 is not a symbolic link

Posted on 2018-06-21 19:37:16
Donald Kinghorn

Interesting ... a couple of things; (That's the GNU scientific library)
First, I wouldn't worry about it too much but remember it in case you run into something strange related to the gsl libs later on.

It's interesting that you are seeing reference to version 23 The version for Ubuntu 16.04 is 19 (installed from package libgsl2) version 23 is default in Ubuntu 18.04 (installed from package libgsl23) ... It's also interesting that it's referring to the /usr/lib path since you generally find this in /usr/lib/x86_64-linux-gnu

In any case it is likely not a big deal. It may go away the next time you run updates ... as long as everything you are doing is working fine then I wouldn't be concerned. But, like I said above remember it in case you do run into some strange error with something. (in general) If you have a program that doesn't want to start and complains about a missing library you can use the ldd command on it to see what dynamic libraries it is trying to link to.

I hope that sets your mind at ease a bit. --Don

Posted on 2018-06-22 17:45:45
jpvm

Thanks for your quick response Don, I appreciate it. For the most part I have not had to worry about it with the things that I am doing. In setting up Ubuntu 16.04 I went about installing gsl-2.5 on my own maybe thats the cause?

Posted on 2018-06-22 18:42:30
REH

Great article! Thanks for sharing. I have a somewhat related issue that you might be able to provide some guidance on I hope. I'm attempting a bare metal install of Ubuntu 16.04 LTS server ISO on a new Dell R640 server with a PERC H740p RAID controller. The driver for the controller is not included in the install iso. No controller means no storage drive mounted to write the boot image to. I've attempted to use all of the Megaraid drivers but they all fail. I've been able to install U17.10 sever and upgrade it to U18.04 with no problem. However, the application I need to load on the server requires 16.04 LTS. ANy recommendations on how I can 1) locate the compatible driver from DSell or LSI and 2) add it to the install build during the installation?

Posted on 2018-06-22 22:05:36
Donald Kinghorn

I'm not crazy about hardware raid controllers because they seem to always be trouble (sooner or later) but I understand the need for it. Looks like you are hitting one of those "sooner" problems.

There are a couple of things I can think of; First, make sure you are using install media built from the latest installer. That should probably be 16.04.4. Then be sure to select HWE Kernel during the install. That is the Hardware Enhanced kernel. It has been maintained pretty well and I would expect it to include most backported modules from 17.10 ?? It is a 4.13 kernel and it's there to keep driver modules up to date.

If you are still not getting any joy then you might want to reconsider your plan for file system layout. i.e. could you get away with using a dedicated SSD for the base install and then configure and mount your raid partitions where you need them. That way you could pull the kernel modules from 17.10 *after* your install ...

I feel your pain! Install time hardware issues are the worst. I wish you the best! --Don

Posted on 2018-06-25 16:22:12
REH

Thank you, thank you, thank you! I have been searching for a separate HWE install ISO with no luck. It never occurred to me that it is included in the LTS install ISO! Reran the install with HWE support enabled and viola, there it is. As is often the case, it is last last little piece of information that is the key to the solution. Laying right there under my nose the entire time. You are truly a master. Thanks again.

Posted on 2018-06-25 22:15:38
jpvm

I had a quick question as to how I should go about having both cuda 8 and cuda 9.2 installed on my Ubuntu 16.04 LTS desktop. Should I just follow the steps you outlined in this previous post? (https://www.pugetsystems.co...

Thanks in advance.

Posted on 2018-06-25 20:14:24
Donald Kinghorn

That post should be a good guide for you but you might want to keep things a little simpler. Basically you want to do the installs from the .run files and not install the driver. You can have multiple versions of the toolkit installed. In that post I did version switching by changing the cuda symbolic link. You might prefer using an environment script in your home directory to switch back and forth and not do the setup with /etc/profile.d/cuda.sh or /etc/ld.so.conf.d/cuda.conf i.e just set your PATH and LD_LIBRARY_PATH locally as needed. I talk about this a bit in the post I did for installing cuda 9.2 on Ubuntu 18.04 https://www.pugetsystems.co...

You do have to take a little extra care when you are working to be sure you have everything pointing to the right places but it shouldn't be too bad because the .run file installs should localize everything to the individual directories under /usr/local

Posted on 2018-06-26 17:57:51
jpvm

Thanks again Don I will try that. Also quick question, will updating my Ubuntu 16.04 LTS kernel from 4.4.0-128-generic to 4.7.0 effect my cuda install?

Posted on 2018-06-26 18:55:12
Donald Kinghorn

that should be fine ...
since you are installing cuda 9.2 you will want to be sure you have the 396 NVIDIA display driver installed. That should be OK with cuda 8 too. Also, be sure you have dkms installed so that your drivers rebuild when you change the kernel. ... also, don't "auto remove" the old kernel so it's still there if you have any trouble with 4.7

Posted on 2018-06-27 21:54:59