Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/1330
Article Thumbnail

NVIDIA NVLink Bridge Compatibility Chart

Written on January 15, 2019 by William George
Share:

Introduction

NVLink is a technology from NVIDIA for creating a high bandwidth link between two compatible video cards. It can be used for many things, from basic SLI for faster gaming to potentially pooling GPU memory for rendering large and complex scenes. What NVLink can be used for depends on how software developers write their applications, and there is a lot of exciting potential for this technology.

However, NVLink requires a physical bridge between the cards in order to enable these capabilities - and NVIDIA isn't transparent as to which models of the NVLink bridge will work with which video cards. Several Quadro video cards support NVLink, along with some of the new GeForce RTX 20-series cards and the Titan RTX. NVLink bridges are available with Quadro or GeForce branding in various sizes.

If you have already set up a system with NVLink, you can find instructions for enabling and testing it on another article.

Compatibility Matrix

Here is a chart of all the NVLink compatible video cards, as of publishing time, along with the various NVLink bridges that are available:

NVLink Bridge →
↓ Video Card
Quadro GP100
2-Slot Pair
Quadro GV100
2-Slot Pair
Quadro RTX
5000 2/3-Slot
Quadro RTX
6000 2/3-Slot
GeForce RTX
3/4-Slot
Titan RTX
3/4-Slot
Quadro GP100 Works (Tested) Should Work Does Not Work Does Not Work Does Not Work Does Not Work
Quadro GV100 Does Not Work Works (Design) Does Not Work Does Not Work Does Not Work Does Not Work
Quadro RTX 5000 Does Not Work Does Not Work Works (Design) Does Not Work Does Not Work Does Not Work
Quadro RTX 6000 Does Not Work Should Work Does Not Work Works (Tested) Works (Tested) Should Work
Quadro RTX 8000 Does Not Work Should Work Does Not Work Works (Design) Should Work Should Work
Titan RTX Does Not Work Should Work Does Not Work Works (Tested) Works (Tested) Works (Design)
GeForce RTX 2080 Does Not Work Should Work Does Not Work Works (Tested) Works (Tested) Should Work
GeForce RTX 2080 Ti Does Not Work Should Work Does Not Work Works (Tested) Works (Tested) Should Work

For the combinations indicated as "Works", we specified whether we have actually tested it or it is just assumed to work because NVIDIA designed it to. For example, the Quadro 6000 RTX NVLink Bridge is clearly designed to work on the Quadro RTX 6000, as well as the upcoming 8000 - but we only have RTX 6000s in-house to test currently. Likewise, the RTX 5000 has its own bridge designed for its smaller NVLink connector - so it should work, by design. "Should Work" indications are used where we believe the combination will work based on other testing we have done, but we don't have samples on hand to verify for sure. "Does Not Work" should be self explanatory.

Insights

The main takeaways from our testing are as follows:

  • The older Quadro GP100 cards were equipped with the first generation of NVLink, and had less bandwidth per link than newer models. As such, the Quadro GP100 bridges do not work with newer cards.
  • The Quadro GV100 cards use the second generation of NVLink, and like the GP100 have two NVLink connectors per card. The newer RTX series bridges can fit on one set of connectors, but their plastic housings are physically larger and cannot be paired up to fully connect both sets of NVLink connectors. That means newer RTX bridges won't fully work on GP100 or GV100 cards.
  • Quadro RTX 5000 cards have a smaller NVLink connector than all the other cards, requiring a completely separate set of NVLink bridges. Because of this they do not interchange with any of the other existing video cards, whether RTX generation or older. This seems to be a design decision by NVIDIA, but is strange since the GeForce RTX 2080 has the same lower NVLink bandwidth (50 GB/s total, versus 100 GB/s on the RTX 6000, 2080 Ti, and Titan RTX) but uses the same full-length connector.
  • Quadro RTX 6000 and GeForce RTX NVLink bridges may look different on the outside, but they appear to be functionally identical. We've tested both types and found they work across most Quadro, Titan, and GeForce cards in this generation. We cannot yet verify that on the Quadro RTX 8000, because they are not yet shipping, but we have tested on a pair of RTX 6000s and all four bridge variants work. Likewise, we have tested them on GeForce RTX 2080, 2080 Ti, and Titan RTX successfully.
  • It is worth noting, however, that the LED logo on GeForce RTX bridges does not light up when used on Quadro RTX cards. The Titan RTX bridges also appear to have a NVIDIA logo, but as we don't have them in hand I am not sure if it lights up.

Conclusion

The good news here is that, if you are using hardware from the RTX generation - except for the Quadro RTX 5000 - it looks like you can use a GeForce, Titan, or Quadro branded bridge without needing to worry about compatibility. That is particularly helpful since they come in different sizes: 2- and 3-slot for Quadro and 3- and 4-slot for GeForce / Titan. I'm not sure why NVIDIA didn't just make 2-, 3-, and 4-slot solutions that were brand agnostic... but at least there don't seem to be any weird limitations on which bridges work with which cards. They are also all the same price, as of publishing time, if purchased directly from NVIDIA. The older GP100 and GV100 are a different story, and if you are using those you should just stick with the paired bridges built for each model, respectively.

Additional Resources

If you want more info about NVLink in Windows, check out another article we published on the topic.

If you need a walk through on enabling NVLink in Windows, or want to make sure it is work, we have an article covering those topics too.

If you are interested in NVLink, Linux, and machine learning, check out Dr Kinghorn's HPC Blog.

For a list of all articles where we've talked about NVLink, use our website's search function.

Tags: NVIDIA, NVLink, Bridge, SLI, Windows 10, GeForce, RTX, 2080, 2080 Ti, Titan, Quadro, 5000, 6000, 8000, GP100, GV100

I just found out that NVIDIA has gone and made Titan RTX NVLink bridges as well - which look just like the GeForce ones, but gold instead of silver (to match the color of the Titan RTX's cooler). I thought about updating the chart above, but if I add any more columns it will not fit in the same width... so I'm just adding a note here that I fully expect it to work in all the same places the GeForce RTX NVLink bridge does - which is to say, across all current RTX series cards.

Posted on 2019-01-25 22:05:55

I also just found out that NVIDIA pulled a stupid and made the Quadro RTX 5000's NVLink connector a smaller size. We knew it was already going to be a 1-link (50 GB/s) connection, rather than 2-link (100 GB/s) like the RTX 6000 and 8000. However, that same split exists on the GeForce side: RTX 2080 is 50 GB/s and 2080 Ti has the full 100 GB/s... but they have the same connector and share the same bridges. I don't know what led NVIDIA to change that for the Quadro RTX 5000, especially since it means yet another set of NVLink bridges (with the smaller connector). I am going to go ahead and update the chart above with this, since it is a pretty major issue.

Posted on 2019-01-25 23:49:39
mathieu soleil

I think it would be worth it to update the paper with those comments as well as the chart (and if needed to resize the width of the columns), because a good part of the readers won't read comments, only the article, an quickly. Also it would be very interesting to verify that a GV100 Nvlink bridge works with 2x2080ti, in particular because of 1) the absence of 2 slot size bridge among the consumer nvlink bridges, and 2) the 100G speed and 3) the cost difference.

Posted on 2019-01-26 15:49:44

I did update the chart, as well as some of the descriptive text in the article, based on the new info about the smaller Quadro RTX 5000 connectors and the Titan RTX bridges. I would be happy to test with the GV100's bridges, if we had any... but we don't, and they are so expensive (~$500!) that it isn't worth ordering them in. If you just need a 2-slot bridge, just the one designed for the Quadro RTX 6000 and it works (we have one and tested it here) with the GeForce RTX 2080 & 2080 Ti.

Posted on 2019-01-28 17:09:05
Heresy Ku

I had test Titan RTX NVLink bridge on 2 GeForce RTX 2080 Ti, it works.

Posted on 2019-02-27 09:08:52

Thank you for the heads-up! :)

Posted on 2019-02-27 21:57:11
Trevor Standley

Wondering if you can bridge a single 2080ti with a single RTX Titan.

Posted on 2019-04-09 00:55:26

You could physically put a bridge between them, but Windows won't allow you to enable SLI across different cards like that - so at least in Windows, you could not use any of the functionality of NVLink (since enabling SLI is what turns it on).

Posted on 2019-04-09 17:33:34
Trevor Standley

I don't care about SLI at all. I'm doing deep learning.
Why would I want to combine an RTX Titan with a 2080ti? Because the Titan has more memory. I can use that memory for running my two 4k displays in xorg and still have more than the 2080ti left over so my deep learning can use all the memory of that card when the two are used in a pair. The NVLink bridge really does speed up deep learning because a large portion of the time is spent communicating the parameters after each minibatch. But I've only ever tried it with 2 2080ti's. And when I do that I have the problem that Xorg is using several gigabytes of vram on the main GPU. Also, the way I am using Apex for fp16 training leads to an imbalance of vram usage. The main GPU gets hit much harder.

Posted on 2019-04-09 18:21:34

I can't speak to whether it would work on Linux, since that OS does not require putting the cards in SLI for NVLink to work. So... it might? I don't think we've tried mixing cards in NVLink on Linux, but that would be Dr Kinghorn's realm (I am not a Linux guy, myself).

Posted on 2019-04-09 18:23:34
Trevor Standley

Just tested it. It works with a 2080 ti and an RTX Titan. Shaves about 8% off the training time, and I get 24 GB on card 1 (which needs more to run my gui and for the way I train my models).

Posted on 2019-06-05 05:26:30

That is awesome! :)

Posted on 2019-06-05 16:03:36
Donald Kinghorn

Hi Trevor, I've done a bunch of testing for ML/AI workloads ... see https://www.pugetsystems.co...

However, I never tried mixing GPU's with NVLINK, but, I don't recommend it. It could(?) work but the differing memory sizes could cause trouble. My gut feeling is that it will not be a good thing even if it did work ... I did a search and didn't come up with anything definitive. I really think the memory mapping would cause issues.

If you have both of these (great!) cards then you might have good luck running different training runs on each card at the same time. I'm referring to hyper-parameter tuning and such ...

I'll keep your question in mind the next time I am doing testing ... best wishes --Don

Posted on 2019-04-09 19:20:15
Trevor Standley

Thanks. If it did work, I see no reason it wouldn't be a good thing. The two cards are almost the same speed, and the extra memory of the RTX card is exactly what I need. I already use them without a bridge, and it works great. Experiments I did with two 2080ti's in a different machine showed that NVLink really helps. Maybe I'll try moving the bridge to my local machine one day.

Posted on 2019-04-09 19:32:46

If you do give it a try, please let us know how it goes :)

Posted on 2019-04-09 20:25:35
Donald Kinghorn

Yes! The only way to really know is to try it. I hear you about the memory on RTX Titan having 24GB is nice. The next time I am doing appropriate testing I will try it.

If you do a test moving the bridge from your other machine please post your results back in a comment here or maybe better on
https://www.pugetsystems.co...
or
https://www.pugetsystems.co...

That would be really interesting ... and like I said when I get a chance to try it I will do the same thing. Thanks --Don

Posted on 2019-04-10 19:39:14
Trevor Standley

Just tested it. It works with a 2080 ti and a n RTX Titan.

Posted on 2019-06-05 05:25:05
Donald Kinghorn

Nice! Thanks for the feedback

Posted on 2019-06-05 16:26:53
Anh Nguyễn Trần Bảo

Did you have any Quadro RTX 8000s? If yes, then test them with NVLink bridges to make this complete. (This is because motherboards for Xeon W-3175X has 2 PCIe 3.0 x16 slots and 2 PCIe 3.0 x8 slots (mechanically x16) placed interleavely, all are dual-slots)

Posted on 2019-04-20 04:35:26

We have only one RTX 8000, so I can't test NVLink with it, but it is basically the same as the RTX 6000 just with twice the VRAM.

Posted on 2019-04-20 04:41:20
Anh Nguyễn Trần Bảo

And, NVIDIA does make Quadro RTX 8000 NVLink bridges. That means the result could be different.
https://www.nvidia.com/en-u...

Posted on 2019-04-20 11:05:14

The Quadro RTX 6000 and 8000 share the same bridge, though:

https://www.nvidia.com/en-u...

So given how widely inter-compatible most of the other bridges are, I cannot imagine the one specifically designed for the 6000 & 8000 working on the 6000 but not the 8000. And indeed, all the other bridges that worked on the 6000 in our testing should also function on the 8000 without issue :)

Posted on 2019-04-22 15:50:25

Ugh, it looks like NVIDIA changed their plans and *did* in fact make two separate sets of NVLink bridges for the RTX 6000 and RTX 8000. They are purely cosmetic differences: the RTX 8000 bridge is reflective, to match the styling on the heasink shrouds those cards come with.

Okay, rant time: It is ridiculous that NVIDIA is making 8 functionally identical parts when they could simply have 3. All that is needed is a 2, 3, and 4-slot NVLink bridge for the normal cards (GeForce RTX 2080, 2080 Ti, Titan RTX, RTX 6000, and RTX 8000) since they all share the same NVLink connector. Instead they have made two each for GeForce, Titan, RTX 6000, and RTX 8000 - with the only difference being styling, so that they match the look of the cards. I guess this is what happens when windowed chassis are so popular: you end up with form more important than function. The RTX 5000 legitimately needs its own bridges, since they made its connector smaller, but that too was unnecessary: the RTX 2080 also has a more limited NVLink connection, but it still uses the full-size connector so the same could have been done with the RTX 5000. Bah!

Posted on 2019-04-24 16:10:49
Paulo Gouveia

Hello guys
Im thinking to build a system with 4 x Quadro GV100 connected in pairs with 2 NVLINK bridges.
Any expected issue with this config?
Cheers

Posted on 2019-05-17 11:29:12

What OS? In Windows, at least last time I tried, the GP100 & GV100 cards required a separate GPU for video output when put into NVLink.

Posted on 2019-05-17 15:18:20