NVIDIA GeForce RTX 2080 & 2080 Ti Do NOT Support Full NVLink in Windows 10Written on October 5, 2018 by William George
When NVIDIA announced the GeForce RTX product line in August 2018, one of the things they pointed out was that the old SLI connector used for linking multiple video cards had been dropped. Instead, RTX 2080 and 2080 Ti cards would use the NVLink connector, found on the high-end Quadro GP100 and GV100 cards. This caused much excitement since one of the features of NVLink on Quadros is the ability to combine the video memory on both cards and share it between them. This is extremely helpful in applications that can be memory-limited, like GPU based rendering, and having it available on GeForce cards seemed like a great boon. Afterward, though, NVIDIA only spoke of it using terms like "SLI over NVLink" - leading many to surmise that the GeForce RTX cards would not support the full NVLink feature set, and thus might not be able to pool memory at all. To clear this up we decided to investigate...
What is NVLink?
At its core, NVLink is a high-speed interconnect designed to allow multiple video cards (GPUs) to communicate directly with each other - rather than having to send data over the slower PCI-Express bus. It debuted on the Quadro GP100 and has been featured on a few other professional NVIDIA cards like the Quadro GV100 and Tesla V100.
What Can NVLink on Quadro Cards Do?
As originally implemented on the Quadro GP100, NVLink allows bi-directional communication between two identical video cards, including sharing access to the memory built onto each card. This allows video cards in such configurations to tackle larger projects than they could alone. In a larger implementation, it could connect multiple GPUs in a mesh network, with similar capabilities.
What Are the Requirements to Use NVLink on Quadros?
Special setup is necessary to use NVLink on compatible Quadro cards. Two NVLink bridges are required to connect them, and a third video card is needed to handle actual display output. Linked GPUs are then put in TCC mode, which turns off their outputs (hence the third card). Application-level support is also needed to enable memory pooling.
Do GeForce RTX 2080 and 2080 Ti Video Cards Have NVLink Connectors?
Technically, yes: there is a single NVLink connector on both the RTX 2080 and 2080 Ti cards (compared to two on the Quadro GP100 and GV100). If you look closely, though, you will see that the connectors on the RTX cards face the opposite direction of those on the Quadro cards. Check out the pictures below:
Are the GeForce RTX and Quadro NVLink Bridges the Same?
No, there are several differences between the NVLink bridges sold for the GeForce RTX cards and older ones built for Quadro GPUs. For example, they differ in both appearance and size - with the Quadro bridges designed to connect adjacent cards while the GeForce RTX bridges require leaving a slot or two between the connected video cards.
Are GeForce RTX and Quadro NVLink Bridges Interchangeable?
In our testing, the Quadro bridges physically fit but would not work on GeForce RTX 2080s. The GeForce bridge did work on a pair of Quadro GP100 cards, with some caveats. Due to its larger size, only one GeForce bridge could be installed on the pair of GP100s - meaning only half the potential bandwidth was available between them.
How Does NVLink on the GeForce RTX 2080 Compare to NVLink on the Quadro GP100?
After testing many different combinations of cards and NVLink bridges, we were unable to find any way to turn on TCC mode for the GeForce RTX cards. That means they cannot handle the "peer-to-peer" communication which is needed for full NVLink functionality. Even if they could, bandwidth would be much more limited due to fewer links.
The chart above shows the results we found when using different combinations of video cards and NVLink bridges, including which combinations supported SLI and whether TCC could be enabled. Click to expand and see additional notes about each configuration.
These screenshots from the Windows command line show peer-to-peer bandwidth across cards with different types of NVLink bridges installed. The first three are pairs of GP100s with no bridge, the GeForce RTX bridge, and then dual Quadro bridges - while the last screenshot shows that the RTX 2080 cards did not support peer-to-peer communication at all (regardless of what bridge was installed).
Will GeForce RTX Cards Support Memory Pooling in Windows?
At this point, all the data we have points to "No". Between the limited bandwidth that a single link offers, compared to four links on the Quadro GP100, and the fact that TCC mode cannot be enabled and thus peer-to-peer communication is not fully functional, it looks like GeForce RTX cards will not support memory pooling as many people hoped.
Does NVLink Work on GeForce RTX Cards in Linux?
My colleague Dr. Don Kinghorn conducted similar tests in Ubuntu 18.04, and he found that peer-to-peer communication over NVLink did work on RTX 2080 cards in that operating system. This functionality in Linux does not depend on a different driver mode like TCC, so with that hurdle removed the hardware link itself seems to work properly.
So What is NVLink on GeForce RTX Cards Good For?
While they do not appear to support the full NVLink feature set in Windows, even the single link that the RTX 2080 and 2080 Ti have is far faster than the older SLI interconnect. That seems to be the focus on these mainstream, gaming-oriented cards: implementing SLI over a faster NVLink connection. That goal is accomplished, as shown in benchmarks elsewhere.
Will GeForce RTX Cards Gain More NVLink Functionality in the Future?
Future driver updates from NVIDIA could change the situation dramatically, and might unlock capabilities that are currently unavailable. Additionally, the 2.5 Geeks Webcast interviewed a NVIDIA engineer who indicated that NVLink capabilities on these cards will be exposed via DirectX APIs - a different approach than TCC that Quadro cards have used, which is what we tested here.