Puget Systems print logo
https://www.pugetsystems.com
Read this article at https://www.pugetsystems.com/guides/2074
Article Thumbnail

NVIDIA NVLink 2021 Update and Compatibility Chart

Written on February 24, 2021 by William George
Share:

Introduction

NVLink is a technology from NVIDIA for creating a high bandwidth link between two compatible video cards. That high-speed data connection could be used for many things, such as pooling GPU memory for rendering large and complex scenes. What NVLink will be used for depends on how software developers write their applications, though, and there is a lot of exciting potential for this technology.

However, NVLink requires a physical bridge between the cards in order to enable these capabilities - and NVIDIA isn't transparent with regard to which models of the NVLink bridge will work with which video cards. The older GeForce 20-Series and Quadro RTX video card families included many models with NVLink support, but in the newest generation the list of supported cards is much smaller. As of the first publication of this article, there are only two GPUs (GeForce RTX 3090 & RTX A6000) and three bridges available... but as more models are announced, that number may grow. If that happens, we will try to keep this article updated with the latest information.

Dual NVIDIA RTX A6000 Video Cards Connected by a GeForce RTX 3090 4-slot NVLink Bridge

Dual NVIDIA RTX A6000s with a GeForce 4-slot NVLink Bridge

If you have already set up a system with NVLink, you can find instructions for enabling and testing it in another article.

Compatibility Matrix

Here is a chart of NVIDIA Ampere-based NVLink compatible video cards and bridges. Please note that this chart does not include the previous generation, as they use a physically different NVLink connector; for info about older cards and bridges, check out this article.

NVLink Bridge →
↓ Video Card
GeForce RTX 3090
4-Slot Bridge
RTX A6000
3-Slot Bridge
RTX A6000
2-Slot Bridge
GeForce RTX 3090 Works (Tested) Should Work* Works* (Tested)
RTX A6000 Works (Tested) Works (Design) Works (Tested)

For the combinations indicated as "Works", we specified whether we have actually tested it or it is just assumed to work because NVIDIA designed it to. For example, the RTX A6000 3-slot NVLink Bridge is clearly designed to work on the RTX A6000, but we haven't tested it because we don't currently have a bridge of that size. "Should Work" indications are used where we believe the combination will work based on other testing we have done, but we don't have samples on hand to verify for sure. "Does Not Work" isn't used in this chart, yet, but may show up later on if additional cards / bridges come out as part of this generation and are found to be incompatible.

* Please note that this chart is only concerned with the functionality of the bridges, and not physical slot spacing. Most GeForce RTX 3090 video cards are 3 or more slots wide, so a 2-slot NVLink bridge will obviously not work with those. We conducted our 2-slot bridge testing with Gigabyte GeForce RTX 3090 TURBO blower-style cards, which are only two slots wide but have sadly been discontinued.

NVIDIA GeForce RTX 3090 and RTX A6000 NVLink Bridges (Top View)

NVIDIA GeForce RTX 3090 and RTX A6000 NVLink Bridges (Bottom View)

NVIDIA GeForce RTX 3090 4-slot and RTX A6000 2-slot NVLink Bridges

Insights

The main takeaways from our testing with this generation of NVLink are as follows:

  • The GeForce RTX 3090 and RTX A6000 use a new NVLink connection, which is not compatible with the bridges from the 20-Series and Quadro RTX cards.
  • RTX A6000 and GeForce RTX 3090 NVLink bridges may look different on the outside, but they appear to be functionally identical. We've tested both types of cards with both styles of bridges, and they worked interchangeably.
  • If you are building your own system with NVLink, make sure you select the right size of bridge to fit the PCI-Express slot spacing on your motherboard.
  • NVLink on Windows is still enabled by turning on SLI, even though NVIDIA has sounded the death-knell for that technology.
  • We conducted our testing with driver version 461.40, but we have seen driver updates alter SLI / NVLink behavior in the past - so if you try this out and have trouble, switch to a different driver and see if that resolves it. In particular, we recommend using NVIDIA's Studio Driver releases (rather than their Game Ready drivers) for GeForce cards.

Conclusion

The good news here is that, if you are using hardware from the Ampere-based generation (GeForce RTX 30 Series and RTX A-series), you can use the right slot-width bridge for your motherboard without needing to worry about which specific video card it is branded for. That is particularly helpful since they come in different sizes: 2- and 3-slot for RTX A-series and only 4-slot for GeForce.

I wish NVIDIA had just made all of the slot sizes without any particular visual design or branding, but given they did the same thing last generation it wasn't a surprise. What was a surprise, though, is that the RTX A-series bridges seem to be priced higher this time around.

Additional Resources

If you want more info about NVLink in Windows, check out another article we published on the topic.

If you need a walk through on enabling NVLink in Windows, or want to make sure it is work, we have an article covering those topics too.

If you are interested in NVLink, Linux, and machine learning, check out Dr Kinghorn's HPC Blog.

If you want to see the previous version of the comparison chart above, for GeForce 20-Series and Quadro RTX cards, click here.

For a list of all articles where we've talked about NVLink, use our website's search function.

Tags: NVIDIA, NVLink, Bridge, SLI, Windows 10, GeForce, RTX, RTX A6000, RTX 3090
Ryan Gesner

Where to buy correct 2 + 3 slot varients?
I have already purchased a four slot version and I was under the impression I would have to get my soldering iron out however if I can purchase a two and or three slot version then I can return or resell the four slot version... Thanks.

Posted on 2021-03-13 18:20:26

With how new the A6000 is, finding the bridges may be a bit tricky at the moment. Using Google (search phrase: 'pny A6000 nvlink') I was able to find several sites listing them, like this, but I cannot tell on some of them if they have stock on hand or what prices are like:

2-slot: https://www.exxactcorp.com/...

3-slot: https://www.exxactcorp.com/...

From those pages, though, I was able to get the actual PNY part numbers: RTXA6000NVLINK-KIT and RTXA6000NVLINK3S-KIT. Searching for those might be even better. Good luck!

Posted on 2021-03-15 18:13:21
bf3magic

Does 3090 nvlink support memory pooling for deep learning tasks?

Posted on 2021-03-20 20:08:09

NVLink, in and of itself, does not do memory pooling (and never has). It is purely a high-speed connection between two video cards, allowing for much higher bandwidth than if the cards communicated through the PCI-Express bus (which is the default, and really only option, without NVLink). That is true of GeForce RTX 3090 cards in NVLink, as well as older models.

What software developers choose to do with that high speed data pathway is up to them. Many developers / applications don't utilize it at all! But there are some programs which have been designed to use it for sharing of information in the cards' VRAM, which is what some folks have characterized as memory pooling. I am not aware of any comprehensive lists of which programs offer that, but if you are writing your own code for deep learning then it is something you could potentially take advantage of.

Posted on 2021-03-22 17:33:00
Ernst Stavro Blofeld

Useful reply, thanks. What about speed? Do two 3090s linked with the A* bridge achieve 112 gb/s communication? Thanks!

Posted on 2021-04-22 14:58:04

Hmm, I don't think I kept the results file (showing P2P bandwidth) from my last round of NVLink testing - so I can't say for sure. Unfortunately, I also no longer have access to the blower-style RTX 3090 cards (which have been discontinued) so I can't re-test with the 2-slot A6000 bridge. I could try a pair of Founders Edition RTX 3090 cards with the 4-slot bridge at some point, if you would find that info helpful.

Posted on 2021-04-22 16:54:25
Ernst Stavro Blofeld

Yes thanks, if you can do it, I think it would be useful to know. Note that AFAIK the 4-slot bridge is limited to 50gb/s. The 3-slot for A6000, instead, is 112 gb/s. So, if you can, do test the two FEs with the 3-slot bridge for A6000!

Posted on 2021-04-22 18:40:56