Puget Systems print logo
https://www.pugetsystems.com
Read this article at https://www.pugetsystems.com/guides/2074
Article Thumbnail

NVIDIA NVLink 2021 Update and Compatibility Chart

Written on February 24, 2021 by William George
Share:

Introduction

NVLink is a technology from NVIDIA for creating a high bandwidth link between two compatible video cards. That high-speed data connection could be used for many things, such as pooling GPU memory for rendering large and complex scenes. What NVLink will be used for depends on how software developers write their applications, though, and there is a lot of exciting potential for this technology.

However, NVLink requires a physical bridge between the cards in order to enable these capabilities - and NVIDIA isn't transparent with regard to which models of the NVLink bridge will work with which video cards. The older GeForce 20-Series and Quadro RTX video card families included many models with NVLink support, but in the newest generation the list of supported cards is much smaller. As of the first publication of this article, there are only two GPUs (GeForce RTX 3090 & RTX A6000) and three bridges available... but as more models are announced, that number may grow. If that happens, we will try to keep this article updated with the latest information.

Dual NVIDIA RTX A6000 Video Cards Connected by a GeForce RTX 3090 4-slot NVLink Bridge

Dual NVIDIA RTX A6000s with a GeForce 4-slot NVLink Bridge

If you have already set up a system with NVLink, you can find instructions for enabling and testing it in another article.

Compatibility Matrix

Here is a chart of NVIDIA Ampere-based NVLink compatible video cards and bridges. Please note that this chart does not include the previous generation, as they use a physically different NVLink connector; for info about older cards and bridges, check out this article.

NVLink Bridge →
↓ Video Card
GeForce RTX 3090
4-Slot Bridge
RTX A6000
3-Slot Bridge
RTX A6000
2-Slot Bridge
GeForce RTX 3090 Works (Tested) Works (Tested) Works* (Tested)
RTX A6000 Works (Tested) Works (Design) Works (Tested)

For the combinations indicated as "Works", we specified whether we have actually tested it or it is just assumed to work because NVIDIA designed it to. For example, the RTX A6000 3-slot NVLink Bridge is clearly designed to work on the RTX A6000, but we haven't tested it because we don't currently have a bridge of that size. "Should Work" indications are used where we believe the combination will work based on other testing we have done, but we don't have samples on hand to verify for sure. "Does Not Work" isn't used in this chart, yet, but may show up later on if additional cards / bridges come out as part of this generation and are found to be incompatible.

* Please note that this chart is only concerned with the functionality of the bridges, and not physical slot spacing. Most GeForce RTX 3090 video cards are 3 or more slots wide, so a 2-slot NVLink bridge will obviously not work with those. We conducted our 2-slot bridge testing with Gigabyte GeForce RTX 3090 TURBO blower-style cards, which are only two slots wide but have sadly been discontinued.

NVIDIA GeForce RTX 3090 and RTX A6000 NVLink Bridges (Top View)

NVIDIA GeForce RTX 3090 and RTX A6000 NVLink Bridges (Bottom View)

NVIDIA GeForce RTX 3090 4-slot and RTX A6000 2-slot NVLink Bridges

Insights

The main takeaways from our testing with this generation of NVLink are as follows:

  • The GeForce RTX 3090 and RTX A6000 use a new NVLink connection, which is not compatible with the bridges from the 20-Series and Quadro RTX cards.
  • RTX A6000 and GeForce RTX 3090 NVLink bridges may look different on the outside, but they appear to be functionally identical. We've tested both types of cards with both styles of bridges, and they worked interchangeably.
  • If you are building your own system with NVLink, make sure you select the right size of bridge to fit the PCI-Express slot spacing on your motherboard.
  • NVLink on Windows is still enabled by turning on SLI, even though NVIDIA has sounded the death-knell for that technology.
  • We conducted our testing with driver version 461.40, but we have seen driver updates alter SLI / NVLink behavior in the past - so if you try this out and have trouble, switch to a different driver and see if that resolves it. In particular, we recommend using NVIDIA's Studio Driver releases (rather than their Game Ready drivers) for GeForce cards.

Conclusion

The good news here is that, if you are using hardware from the Ampere-based generation (GeForce RTX 30 Series and RTX A-series), you can use the right slot-width bridge for your motherboard without needing to worry about which specific video card it is branded for. That is particularly helpful since they come in different sizes: 2- and 3-slot for RTX A-series and only 4-slot for GeForce.

I wish NVIDIA had just made all of the slot sizes without any particular visual design or branding, but given they did the same thing last generation it wasn't a surprise. What was a surprise, though, is that the RTX A-series bridges seem to be priced higher this time around.

Additional Resources

If you want more info about NVLink in Windows, check out another article we published on the topic.

If you need a walk through on enabling NVLink in Windows, or want to make sure it is work, we have an article covering those topics too.

If you are interested in NVLink, Linux, and machine learning, check out Dr Kinghorn's HPC Blog.

If you want to see the previous version of the comparison chart above, for GeForce 20-Series and Quadro RTX cards, click here.

For a list of all articles where we've talked about NVLink, use our website's search function.

Tags: NVIDIA, NVLink, Bridge, SLI, Windows 10, GeForce, RTX, RTX A6000, RTX 3090
Ryan Gesner

Where to buy correct 2 + 3 slot varients?
I have already purchased a four slot version and I was under the impression I would have to get my soldering iron out however if I can purchase a two and or three slot version then I can return or resell the four slot version... Thanks.

Posted on 2021-03-13 18:20:26

With how new the A6000 is, finding the bridges may be a bit tricky at the moment. Using Google (search phrase: 'pny A6000 nvlink') I was able to find several sites listing them, like this, but I cannot tell on some of them if they have stock on hand or what prices are like:

2-slot: https://www.exxactcorp.com/...

3-slot: https://www.exxactcorp.com/...

From those pages, though, I was able to get the actual PNY part numbers: RTXA6000NVLINK-KIT and RTXA6000NVLINK3S-KIT. Searching for those might be even better. Good luck!

Posted on 2021-03-15 18:13:21
Simon Dowsett

Still tricky to find/buy and they don't even come bundled with the high-end A-series cards. Remember when every graphics-card and every decent motherboard you bought came with an SLI-link when SLI was a terrible waste of money? The irony.

Posted on 2021-06-02 11:24:22

Yeah :(

I'm not sure what all is involved in making them, but NVLink bridges have always been a lot more expensive - even compared to some of the rigid, aftermarket SLI bridges back in the day. I'm assuming they must be a lot more than a simple ribbon cable (which is all SLI needed)... probably combined with them being made in smaller quantities (which drives per-unit prices up). That's my best guess, but it does suck that they are so costly and so rare.

Posted on 2021-06-03 21:31:55
bf3magic

Does 3090 nvlink support memory pooling for deep learning tasks?

Posted on 2021-03-20 20:08:09

NVLink, in and of itself, does not do memory pooling (and never has). It is purely a high-speed connection between two video cards, allowing for much higher bandwidth than if the cards communicated through the PCI-Express bus (which is the default, and really only option, without NVLink). That is true of GeForce RTX 3090 cards in NVLink, as well as older models.

What software developers choose to do with that high speed data pathway is up to them. Many developers / applications don't utilize it at all! But there are some programs which have been designed to use it for sharing of information in the cards' VRAM, which is what some folks have characterized as memory pooling. I am not aware of any comprehensive lists of which programs offer that, but if you are writing your own code for deep learning then it is something you could potentially take advantage of.

Posted on 2021-03-22 17:33:00
Ernst Stavro Blofeld

Useful reply, thanks. What about speed? Do two 3090s linked with the A* bridge achieve 112 gb/s communication? Thanks!

Posted on 2021-04-22 14:58:04

Hmm, I don't think I kept the results file (showing P2P bandwidth) from my last round of NVLink testing - so I can't say for sure. Unfortunately, I also no longer have access to the blower-style RTX 3090 cards (which have been discontinued) so I can't re-test with the 2-slot A6000 bridge. I could try a pair of Founders Edition RTX 3090 cards with the 4-slot bridge at some point, if you would find that info helpful.

Posted on 2021-04-22 16:54:25
Ernst Stavro Blofeld

Yes thanks, if you can do it, I think it would be useful to know. Note that AFAIK the 4-slot bridge is limited to 50gb/s. The 3-slot for A6000, instead, is 112 gb/s. So, if you can, do test the two FEs with the 3-slot bridge for A6000!

Posted on 2021-04-22 18:40:56

I found some of my old results from the testing for this article, with a 4-slot bridge on pairs of both A6000s and 3090s. Here is the measured bi-directional bandwidth:

2 x RTX 3090s - 101.45 GB/s
2 x RTX A6000s - 101.36 GB/s

That sounds like it lines up pretty well with a theoretical 112 GB/s (since actual measured speeds are rarely 100% of the theoretical maximum). I don't currently have access to A6000s or a 3-slot bridge to compare, and the 3090 FE cards I have here won't work with a 2-slot bridge... but I'm pretty sure, based on the numbers above, that the 4-slot bridge is not limited in any way.

Posted on 2021-05-27 21:30:49
Ernst Stavro Blofeld

Thanks! That's great since I have the 4-slot variant ;)

Posted on 2021-05-28 13:24:51
Simon Dowsett

RTX A5000 and up for memory-pooling. Although an RTX A5000 is slower than and RTX3090 but more power efficient.

Posted on 2021-06-02 11:29:07
Simon Dowsett

Not sure this test is accurate as I used an RTX3090 NVlink on two RTXA5000 and although the downloadable app said they were in SLI, the bandwidth between them was about half the amount it should have been and there was no automatic memory-pooling.
Also, even when "SLI" was software-disabled, the two RTXA5000s took longer to render the Blender BMW scene than they did previously with no link. With the link they took about 11 seconds and without the link they take around 7:39 seconds. Going to buy the proper link when it is available.

Posted on 2021-06-03 16:14:14

We haven't tested A5000s yet - we have not had any arrive at our facility, to my knowledge - but I hope to give it a try when we get them.

I have also heard that, in general, NVLink / memory pooling can slow down performance when you don't actually need the extra VRAM. In effect, that it is really only beneficial when the memory on a single card is insufficient for the scene you are rendering. I have never tested that myself, instead having tried to focus on evaluating simple functionality rather than real-world performance... largely because no standard rendering benchmark needs / benefits from NVLink at this time. As such, I have been using NVIDIA's provided P2P Bandwidth tools to evaluate when NVLink is functioning... but I would love to find a better approach!

Posted on 2021-06-03 21:37:25
Simon Dowsett

Sorry, I tried to delete my comment because I tried it again and it worked and with the full bandwidth. However, the thing is that you can't turn SLI off via software. As soon as you attach the NVLink bridge, performance takes a massive nose-dive with the specs I quoted. When you turn SLI on via software, the performance comes back again and is only slightly slower than no connected NVLink bridge no SLI.
I think the distance travelled via NVlink also matters.

Edit: disregard this. Maybe the first time the NVlink wasn't seated properly which was causing performance problems. Nvlink can be disable via software just fine now. Although you might need to restart windows explorer via the task manager.

Posted on 2021-06-04 10:30:41

Thank you for posting those updates! I'm glad to hear that it is working better now (or at least, not misbehaving anymore). This SLI / NVLink stuff is definitely a bit messy as far as implementation goes, and I wish NVIDIA would make it a bit cleaner and more straight-forward.

Posted on 2021-06-04 16:10:16