Dr Donald Kinghorn (HPC and Scientific Computing)

GTX 1080 CUDA performance on Linux (Ubuntu 16.04) preliminary results (nbody and NAMD)

Written on May 27, 2016 by Dr Donald Kinghorn
Share:

I got my hands on a GTX 1080 today and decided to see if I could get it working on Linux and do some quick CUDA testing. I succeeded! These are preliminary results and will no doubt change when official drivers and CUDA 8 is released. Also, this is a new architecture so there are bound to be problems with some existing CUDA accelerated programs. Take these results for what they are, just a quick hack of a setup to see if things work. Later when official drivers and CUDA are are released we will get a much better picture of how compute is on the new GeForce cards.

I'll get right to the first CUDA results since they are stunning.

CUDA 7.5 samples nbody simulation (benchmark single precision) 

GTX 1080 --> 5816 GFLOP/s

GTX 980  --> 2572 GFLOP/s

I would expect your jaw to drop right now, mine did! I also, ran a "real" program, NAMD molecular dynamics simulation.

NAMD 2.11 500 time steps of stmv (satellite tobacco mosaic virus - million atom)

GTX 1080 --> 54 sec  .81 day/ns

GTX 980   --> 71 sec  1.21 day/ns

33% speedup with 1080 which is very good for this program!

Testing setup

Puget Systems Peak Single

  • Intel Xeon E5-1650 v3 6-core 3.5 GHz (3.6 GHz all-core-turbo)
  • 64GB DDR 4 2133 REG ECC
  • GTX 980
  • GTX 1080

Puget Systems Peak Dual

  • 2 x Intel Xeon E5-2687W v4 12-core (Broadwell) 3.0 GHz (3.2 GHz all-core-turbo)
  • 256 GB DDR4 2133 REG ECC
  • GTX 980
  • GTX 1080

Software/OS

  • Ubuntu MATE 16.04
  • NVIDIA display driver 367.18 installed from ppa:graphics-drivers/ppa The GTX 1080 is detected as "Graphics Device" with this driver :-)
  • CUDA 7.5-18  I used my usual hacks to get CUDA working on Ubuntu 16.04.

  • NAMD 2.11  Note: I used the Peak Dual for NAMD since the Single was CPU bound i.e. NAMD uses both the CPU and GPU and I needed more CPU to balance the performance of the GTX 1080.
Please note that getting all this working on Ubuntu 16.04 requires some sysadmin skill and it's not supported by anyone! It may be possible to get all of this working in a more sensible setup with Ubuntu 14.04. I will try that and do another post with details. I may do another detailed post on getting everything working on Ubuntu 16.04 after CUDA 8 is out and a "release" driver for the 1080 is released. (both of those I expect to see soon) 

Here's some raw output so you can see a few more details about the card.

kinghorn@u16-1080:~/testing/samples/bin/x86_64/linux/release$ nvidia-smi Thu May 26 14:04:51 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 367.18     Driver Version: 367.18         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980     Off  | 0000:05:00.0      On |                  N/A |
| 26%   37C    P8    15W / 180W |     80MiB /  4037MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Graphics Device     Off  | 0000:09:00.0     Off |                  N/A |
| 27%   30C    P8     6W / 180W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1113    G   /usr/lib/xorg/Xorg                              79MiB |
+-----------------------------------------------------------------------------+


kinghorn@u16-1080:~/testing/samples/bin/x86_64/linux/release$ ./nbody -benchmark -numbodies=256000 -device=0

> Windowed mode
> Simulation data stored in video memory
> Single precision floating point simulation
> 1 Devices used for simulation
gpuDeviceInit() CUDA Device [0]: "Graphics Device
> Compute 6.1 CUDA device: [Graphics Device]
number of bodies = 256000
256000 bodies, total time for 10 iterations: 2259.061 ms
= 290.103 billion interactions per second
= 5802.057 single-precision GFLOP/s at 20 flops per interaction
kinghorn@u16-1080:~/testing/samples/bin/x86_64/linux/release$ 


kinghorn@u16-1080:~/testing/samples/bin/x86_64/linux/release$ ./bandwidthTest --device=0
[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: Graphics Device
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			12147.9

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			12866.1

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			278829.1

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.


kinghorn@u16-1080:~/testing/samples/bin/x86_64/linux/release$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 2 CUDA Capable device(s)

Device 0: "Graphics Device"
  CUDA Driver Version / Runtime Version          8.0 / 7.5
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 8113 MBytes (8507555840 bytes)
MapSMtoCores for SM 6.1 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 6.1 is undefined.  Default to use 128 Cores/SM
  (20) Multiprocessors, (128) CUDA Cores/MP:     2560 CUDA Cores
  GPU Max Clock rate:                            1734 MHz (1.73 GHz)
  Memory Clock rate:                             5005 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 2097152 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 9 / 0
  Compute Mode:
      Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) 

Device 1: "GeForce GTX 980"
  CUDA Driver Version / Runtime Version          8.0 / 7.5
  CUDA Capability Major/Minor version number:    5.2
  Total amount of global memory:                 4037 MBytes (4233297920 bytes)
  (16) Multiprocessors, (128) CUDA Cores/MP:     2048 CUDA Cores
  GPU Max Clock rate:                            1216 MHz (1.22 GHz)
  Memory Clock rate:                             3505 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 2097152 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 5 / 0
  Compute Mode:
      Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) 
 Peer access from Graphics Device (GPU0) -> GeForce GTX 980 (GPU1) : No
 Peer access from GeForce GTX 980 (GPU1) -> Graphics Device (GPU0) : No

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 7.5, NumDevs = 2, Device0 = Graphics Device, Device1 = GeForce GTX 980
Result = PASS

Happy computing! --dbk

Tags: GTX 1080, NVIDIA, CUDA, Performance, HPC
ProPhoto Stl

Can you add Titan X to this comparison? If you don't have it, would to assume that it would perform similar to GTX 980?

Posted on 2016-05-28 00:26:45
20 ezressé' kommentelni

>GTX 1080 --> 54 sec .81 day/ns
>GTX 980 --> 71 sec 1.21 day/ns
>50% speedup with 1080 which is very good for this program!

that's 31%

Posted on 2016-05-28 22:33:04
Donald Kinghorn

yea, lets call it 33% -> 1.21 - (1.21 * .33) = .817 ... fixed ...

Different programs are going to to show different results. These MD programs need to have a good system balance CPU/GPU to get best results. With the E5 1650v3 6-core the results for the 980 and 1080 were basically the same i.e. CPU bound. Bottom line though is that these GTX 10xx cards are going to be worth trying given the price/performance. There could be programs that break from the architecture change though so some caution is in order.
Take care -Don

Posted on 2016-05-31 15:28:53
Donald Kinghorn

As you all probably know by now CUDA 8 is released as dev RC and it does have support for gcc > 4.8 i.e. Ubuntu 16.04 should work out of the box. I'll do some more testing and keep you posted. --Don

Posted on 2016-05-31 15:42:59
Chrisyphus

Would you happen to know the performance difference between the 970 and the 980, or the 1070 and the 1080 for these kinds of applications? I can only ever find gaming related benchmarks. I am thinking of picking up a 1080 to replace a 970 on a purely GPU number crunching machine, but am trying to see if it would justify the price.

Posted on 2016-07-10 06:42:55