Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/1266
Article Thumbnail

How to Enable and Test NVIDIA NVLink on Quadro and GeForce RTX Cards in Windows 10

Written on October 26, 2018 by William George
Share:

Introduction

NVLink is a technology from NVIDIA for creating a high bandwidth link between two of their video cards. It can be used for many things, from basic SLI for faster gaming to potentially pooling GPU memory for rendering large and complex scenes. What NVLink can be used for depends on how software developers write their applications, and there is a lot of exciting potential for this technology.

However, as of the time we are publishing this, there is no easy way to tell if NVLink is enabled and functioning within Windows 10. Further, the way to enable NVLink is different depending on what sort of video cards you have. We have put together this guide to help: covering the different ways to enable NVLink, along with a small utility to easily test and make sure it is working as expected.

If you have already set up your system for NVLink, you can skip ahead to download and run the NVLinkTest utility.

Installing the Physical NVLink Bridge

Regardless of what cards you are using, for NVLink to function they must be connected by a physical NVLink bridge (or in some cases, two bridges). The bridge needs to be the right size to reach between the cards, which can potentially be 2, 3, or 4 PCI-E slots apart.

In addition to being the right size, the bridge you use also needs to be compatible with the video cards. For example, older Quadro GP100 bridges do not work on the newer GeForce RTX series cards. On the other hand, the GeForce bridges do seem to work on the Quadro GP100 cards - except that they are physically larger, so you can only fit one bridge instead of the two that the GP100 is designed to use. The safest option is to stick with bridges specifically designed for the video cards you are using. Some other combinations may work, but are not likely to be officially supported by NVIDIA.

Here are examples of what NVLink bridges can look like, as well as what it looks like to have them installed on cards:

NVLink Bridges from Quadro GP100 and GeForce RTX cards

NVLink Bridges from Quadro GP100 and GeForce RTX cards

NVLink Bridge installed on two GeForce RTX 2080 cards

NVLink Bridge installed on two GeForce RTX 2080 cards

Enabling NVLink on GeForce and Quadro RTX Video Cards

Once the physical link is securely installed, it is quite simple - though not obvious - how to enable NVLink on GeForce and Quadro RTX series cards. While NVLink itself is never mentioned in the NVIDIA Control Panel (so far as we could find) all you need to do is enable SLI. To do so, open the NVIDIA Control Panel, go to "Configure SLI, Surround, and PhysX" under "3D Settings" in the menu tree on the left, select "Maximize 3D performance" under "SLI configuration", and click Apply. Here is what that should look like, once SLI is enabled:

NVIDIA Control Panel Screenshot Showing SLI Enabled on GeForce RTX 2080 Video Cards

SLI itself is mostly used for gaming, and in the past we have recommended avoiding it for other applications - but if you want to use NVLink, this is the way to enable it on GeForce RTX 2080 and 2080 Ti cards, as well as the Titan RTX and Quadro RTX 5000, 6000, and 8000 models. Once enabled, you can test to make sure it is working.

Enabling NVLink on Quadro GP100 and GV100 Cards

NVLink originally debuted on the Quadro GP100 video cards, and is also found on the GV100. Both of these models require a pair of NVLink bridges for full performance, along with a more complex setup process than simply turning on SLI:

1) A third video card needs to be installed - ideally a Quadro from the same generation or newer than the cards you are bridging. This is required for video output, because enabling NVLink on the GP100 and GV100 cards will turn off their video outputs.

2) With both physical bridges installed and the third card connected to your monitor(s), open up the Windows command line. An easy way to do this is to right-click on the Start icon (not the normal left-click) and select Command Prompt.

3) Once that is open, navigate to "C:\Program Files\NVIDIA Corporation\NVSMI"

4) Run "nvidia-smi.exe -L" to see which numbers (starting with 0) are assigned to the two video cards you want to bridge.

5) For each of those cards, run "nvidia-smi.exe -i # -dm TCC", where # is the number of the GPU you wish to have in NVLink.

6) Once you have successfully run that command on both cards, reboot the system and test to see if NVLink is working. As mentioned before, TCC mode disables direct video output from those GPUs - so you need to have any monitors hooked up to additional cards.

TCC Mode Being Enabled on Quadro GP100 Video Cards

This is how TCC is enabled on Quadro GP100s via the command line in Windows 10.

Download NVLinkTest to Verify NVLink Functionality

As of the publication of this article, there is no way to check NVLink status in the NVIDIA Control Panel. However, NVIDIA does supply some sample code in their CUDA Toolkit which can check for the peer-to-peer communication that NVLink enables and even measure bandwidth between video cards. You can download that toolkit, install Visual Studio, compile the sample code, and then run it - or...

We have compiled one of those sample programs and put together a simple GUI to make it easy to run in Windows 10. Just click the Download button below, save the linked ZIP file, and extract its contents. Inside you will find three files: a Readme.txt, NVIDIA's p2pBandwidthLatencyTest.exe, and our NVLinkTest.exe. Run that last program and it will report whether NVLink is working or not.

Download

NVLinkTest Screenshots

Here are screenshots showing what NVLinkTest looks like while running. The first shows the intro screen, the second shows a successful run - where NVLink is enabled - and the remaining two show the errors you will get if CUDA is not available (which means you don't have a NVIDIA GPU or the NVIDIA drivers are not correctly installed) or if you do not have any active NVLink connections.

NVLinkTest.exe intro screen with basic instructions

Intro screen with basic instructions

NVLinkTest.exe NVLink enabled result screen

NVLink enabled - success!

NVLinkTest.exe CUDA not available error screen

Error if CUDA is not available

NVLinkTest.exe NVLink not enabled result screen

Result if NVLink is not enabled

Additional Resources

If you want more info about NVLink in Windows, check out another article we published on the topic.

If you are interested in NVLink, Linux, and machine learning, check out Dr Kinghorn's HPC Blog.

For a list of all articles where we've talked about NVLink, use our website's search function.

Need help with your Puget Systems PC?

If something is wrong with your Puget Systems PC. We are readily accessible, and our support team comes from a wide range of technological backgrounds to better assist you!

Contact Puget Systems Support

Looking for more support guides?

If you are looking for a solution to a problem you are having with your PC, we also have a number of other support guides that may be able to assist you with other issues.

Puget Systems Online Help Guides

Tags: NVIDIA, NVLink, Enabled, Disabled, SLI, Windows 10, GeForce, RTX, 2080, 2080 Ti, Quadro, GP100, GV100, Bandwidth, Test, NVLinkTest, p2pBandwidthLatencyTest, CUDA, Screenshot, How to, Enable, Verify
Perry White

I've been watching the p2pBandwidthLatencyTest.exe process run for 15 minutes or so. How long should this run before results are given?

Posted on 2019-01-30 17:35:13

Only a few seconds. Did you run it from the utility I linked to above, or directly from the command line? Also, do you have the latest NVIDIA drivers installed?

Posted on 2019-01-30 18:02:34
Dima

I have asus rtx 2080 and 2080 super I do NVlink and is not enable WHY ?

Posted on 2019-12-26 18:01:19

The 2080 and 2080 SUPER are different cards, and I think NVLink requires a pair of identical cards.

Posted on 2019-12-26 19:50:38
dima

It's a simple card car with little improvements.
It's really stupid .

Posted on 2019-12-26 20:21:40
Niko

can I nvlink 2x RTX 2080 super ?

Posted on 2019-12-27 22:21:05

Yes, any two of the same model card that supports NVLink (which the 2080 SUPER does). I recommend using single fan, blower-style cards for multi-GPU configurations.

Posted on 2019-12-27 22:35:02
DrNooob

not sure if someone here can help me but i installed my two gpus, enabled nvlink, and confirmed with the nvlink test program. but when i play games, particularly rainbow six siege which according to several nvlink tests and reviews appears to scale very well, i actually get worse frames on nvlink setup compared to single gpu setup. is there some additional settings that i'm missing? or how can i troubleshoot this so i can get nvlink to work properly?

Posted on 2020-01-09 07:48:35

Hmm, it has been so long since I've actually done any game-focused testing that I am not sure what to recommend in this situation. I play games myself, but have never owned a dual GPU / SLI / NVLink setup. Maybe try asking in more gaming-centric online forums, like on Reddit, or on Twitter or YouTube channels where dual GPU is featured regularly... like JayzTwoCents?

Posted on 2020-01-09 18:43:07
DrNooob

thank you for the reply, i've been looking all over trying to find a forum where this might be more appropriate but to no avail, so i figured i'd give it a shot here since the review article was thorough and a great read. given the plethora of reviews out there citing such great scaling with NVLink i feel like it would be a simple error on my part, either in setting it up in nvidia control panel or something but alas, perhaps it was not meant to be :) thank you for your reply, greatly appreciated!

Posted on 2020-01-09 22:53:07
Niko

Hey this is me again.
I bought two RTX 2080 super and nvlink and it doesn't sync .
what can I do ?

Posted on 2020-01-15 18:59:46

Can you describe what you mean by "it doesn't sync"? What steps have you gone through - both in terms of hardware setup / installation and drivers / settings? And what OS are you running?

Posted on 2020-01-15 19:02:15
Niko

I have a windows 10 .
I have a motherboard ASUS Z390 Rog Maximus XI Hero ATX .
2 RTX 2080 super and nvlink everything is connected and work .
I checked for updates to Nvidia and windows (No updates)
I go to control panel and no setting appears sli.
I downloaded the software for sticky and it seems not to work .
try aura sync and all sync but 1 of card no sync .

Posted on 2020-01-15 19:32:46
Niko

It's like not seeing the second card

Posted on 2020-01-15 19:34:11

Hmm, do two cards show up in Windows' Device Manager? And in the NVIDIA Control Panel?

Do both cards have their own power connections from the power supply?

Posted on 2020-01-15 19:47:01
Niko

Both cards have connections for power supply
My power supply 1000 w
Can't see the card in Control Pane / task manger .

Posted on 2020-01-15 20:04:50

Have you installed the latest NVIDIA drivers, from their website? If not, I would recommend doing that - and then checking device manager again to see what shows up. If both cards are properly seated in their PCI-E slots, powered up, and working then there should be two entries in Device Manager, under the Display Adapters section. Maybe take a screenshot of that after making sure the latest drivers are installed and post it here?

Download link: https://www.nvidia.com/Down...

Posted on 2020-01-15 20:09:06
Ahmad

Make sure the the NVLink is snapped on firmly. Make sure you actually see lights on the nVlink adapter

Posted on 2020-02-04 21:39:27

It is worth noting that Quadro NVLink bridges don't have lights, and if you use a GeForce or Titan bridge on a pair of Quadros I don't think it will light up (I am not certain, but that is what sticks in my head from our previous testing).

Posted on 2020-02-04 21:59:33
Niko

but if I turn my gtx 1080 he see aboth

Posted on 2020-01-15 20:08:00

I'm sorry, but I didn't understand this message - can you rephrase it?

Posted on 2020-01-15 20:11:30