GeForce RTX 2080 Multi-GPU Scaling in OctaneRender and Redshift
Written on October 12, 2018 by William GeorgeIntroduction
We found previously that stacking multiple RTX 2080 video cards next to each other for multi-GPU rendering led to overheating and significant performance throttling, due to the dual-fan cooler NVIDIA has adopted as the standard on this generation of Founders Edition cards. Now that manufacturers like ASUS are putting out single-fan, blower-style cards we can repeat our testing to see if the throttling issues are resolved and find out how well these video cards scale when using 1, 2, 3, or even 4 of them for rendering in OctaneRender and Redshift.
Test Setup
When benchmarking we often run each test three times, make sure the results are pretty close, and then take the best one (highest score or lowest time, depending on the situation) as the final result. This helps ensure that something going on in the background doesn't throw things off and that each hardware configuration we test gets a fair shake. Because of our previous experience with the NVIDIA Founders Edition RTX 2080s throttling, though, we wanted to conduct longer tests - so we ran each combination of video cards and software for 30+ minutes straight, collecting far more data points and ensuring that we would see throttling if it was going to happen.
To keep the testing fair, and to be able to compare performance directly to our previous test, we used the same motherboard, CPU, and power supply. If you would like full details on the hardware configuration we ran these tests on, just click here to expand a detailed list.
| Testing Hardware | |
| Motherboard: | Asus WS C422 SAGE/10G |
| CPU: | Intel Xeon W-2175 2.5GHz (4.3GHz Turbo) 14 Core |
| RAM: | 8x Kingston DDR4-2666 32GB ECC Reg (256GB total) |
| GPU: | 1 - 4 x ASUS Turbo RTX 2080 8GB |
| Hard Drive: | Samsung 960 Pro 1TB M.2 PCI-E x4 NVMe SSD |
| OS: | Windows 10 Pro 64-bit |
| PSU: | EVGA SuperNova 1600W P2 |
| Software: | OctaneRender 3.08 Benchmark & Redshift 2.6.22 Demo (Age of Vultures scene) |
Benchmark Results & Analysis
The first question we wanted to answer is whether these single-fan, blower-style cards would throttle like the NVIDIA Founders Edition models did - so the first thing we tried was a full set of four GPUs. Running both OctaneRender and Redshift for over 30 minutes each, we looked at the difference between the highest and lowest results - and found there was only 1.4% variance in OctaneRender and 1.6% variance in Redshift. That is well within the margin of error when looking at this many test runs, and since we saw huge performance dips after just the second run with these programs before it is clear that the blower-style cooling solution fixes the overheating and throttling issues we had with the dual-fan Founders Edition cards.
With that question out of the way, we continued to run sets of three, two, and finally just one of these cards to see how well they scale. Here are charts showing those results for OctaneRender - both absolute scores as well as percent increase compared to a single card:
That looks great - near perfect scaling, with just some minor variance... exactly what we have come to expect from OctaneRender. Moving on, we have the same sort of result charts for Redshift as well:
Here, again, we see excellent results. Redshift has always shown a little more drop-off in performance when we've tested it with multiple GPUs, compared to Octane, but it still benefits greatly from having more than one video card to utilize. Because these blower-style cards are not throttling significantly under heavy load, there is no artificial penalty from using a full set of four cards to maximize performance.
ASUS Turbo vs NVIDIA Founders Edition Cards
There are some additional things to consider when looking at different video cards built with the same GPU. To illustrate some of those issues with these two particular GeForce RTX 2080 models...
Single Card Performance vs Multi Card Performance - If you look at just a single card of each type, the NVIDIA Founders Edition models are clocked slightly higher than the reference-speed ASUS Turbo cards. The specs NVIDIA lists on their website show a 5% clock speed difference (1800 vs 1710 MHz), and that matches almost exactly what we found when comparing single GPU performance:
However, when we ramp up to a full set of four graphics cards, the multi-GPU friendly cooler on the ASUS Turbo model maintains better performance than the dual-fan NVIDIA Founders Edition card - even though the NVIDIA version starts off with higher clock speeds:
Noise Levels - As individual cards, dual-fan models like the NVIDIA Founders Edition RTX 2080 will usually run quieter under load than a single-fan blower card. However, when the cards are stacked next to each other in a three or four GPU configuration, those dual fan designs have to ramp fan speeds higher to try and keep cool. I didn't have the equipment to measure noise levels, but the set of four Founders Edition cards sounded much louder to my ears under load than the ASUS Turbo models. Both sets of cards did ramp their fans up higher when crowded together than when used as single cards, though, which makes sense since airflow is more restricted and there is more heat surrounding the cards in that situation.
Which Type of GeForce RTX Card Should You Get for Multi-GPU Rendering?
If you just want a single video card, the NVIDIA Founders Edition GeForce RTX 2080 and 2080 Ti are great choices - and there are other brands that make similar dual-fan models. If you want to use multiple GPUs, however, single-fan cards that exhaust heat out of the system like the ASUS Turbo RTX 2080 are much better.
How Well Do GeForce RTX 2080 Cards Scale in OctaneRender and Redshift?
Assuming you use video cards with coolers that can keep them running well in multi-GPU configurations, the GeForce RTX video cards have excellent performance scaling in both OctaneRender and Redshift. GPU-based rendering has always benefited from multiple cards, in our experience, and the RTX series is no exception... but make sure your system's cooling is up to the task.
OctaneRender Workstations
Puget Systems offers a range of powerful and reliable systems that are tailor-made for your unique workflow.
Labs Consultation Service
Our Labs team is available to provide in-depth hardware recommendations based on your workflow.
Redshift Workstations
Puget Systems offers a range of powerful and reliable systems that are tailor-made for your unique workflow.
Labs Consultation Service
Our Labs team is available to provide in-depth hardware recommendations based on your workflow.
So you are saying there is a 25-27% performance difference between blower style card and dual fan card?! That is huge! And no one is really talking about this except you guys!
With the cards so close the airflow is really restricted, obviously. I'm wondering. On some motherboards there is three slots between the first and second x16 slots. This way you have room for a tripple cooler card, or more breathing room for a FE or blower card. Would you expect to see this large of a performance difference on a dual GPU config as with your 4x config?
Yes, if you are using these cards in a configuration where they are right next to each other the blower style are much, much better. As a single card, or in pairs where there is added space between them, the dual-fan cards are okay... though they still pump a lot of hot air back into the computer, which may raise temps on other components in the system. Back when I did the previous article, showing how badly four of the Founders Edition RTX 2080 cards throttled, I did a sanity check with just two of them spaced a couple slots apart - and they did not throttle, at least in an open-air testbed.
We haven't really done any work with triple-wide cards, mostly because you can't fit as many of them in a system. They are also very large and heavy, making them less than ideal for being shipped out to customers in fully assembled computers. I would expect that two of them in a system would be okay, provided there was some space between the cards for airflow / intake for the fans. With GPU based rendering, though, it is really nice to be able to cram as many video cards into a single system as possible (from a performance standpoint).
Makes sense. Really goes to show how important airflow in the case is, like you guys are experts on. True, The main negative with the triple slot cards that I see is that it makes it difficult to add other card like 10Gb/s/SFP+ network card or video output cards for NLEs.
https://www.redshift3d.com/...
Why the latest 2.6.23 version not used for benchmarks?
We often start these benchmarks days or weeks before publishing, and sometimes software versions get changed in the middle of our testing. In this case, and in another article that will be coming out Monday, we started with 2.6.22 a few weeks ago as it was the most recent version available when the new GeForce RTX 2080 and 2080 Ti cards got to our lab. Rather than go back and re-test a bunch of stuff when 2.6.23 came out, we stuck with 2.6.22... plus, I don't think there were any major changes between those versions (as far as I can tell from the 2.6.23 release notes: https://www.redshift3d.com/... ).
https://www.nvidia.com/en-u...
https://www.nvidia.com/en-u...
Nvidia Quadro RTX cards being blower fan designs makes a lot of sense now, since they'll be used in multi-GPU configurations.
Will Puget be testing Quadro RTX 6000 & RTX 5000 when they are available?
having an rtx VS quadro benchmark on red shift and octane would be EYE OPENING...
Certain companies get a little bit sensitive when mainstream and "professional" video cards are compared directly. However, if we have two articles with performance data on two separate lines of video cards but using the same underlying platform and same version of software it can make for an easy comparison for those who want that information ;)
Yes, I fully intend to test the new Quadro cards when we get samples. Probably just as individual cards, since we already have a good idea of how multi GPU scaling goes and they use blower fans which means we don't have to really worry about throttling.
Thanks! what about temps in full load? would 3 or 4 of these cards perform without temp issues during several hours in full load?... recomended PSU? thanks
Our open testbeds wouldn't be a great way to look at temperatures, since things will be a little warmer in the confines of an enclosed chassis, but we didn't see any significant throttling with these cards - so as long as they are in a chassis with sufficient airflow (intake, in particular - the cards themselves will exhaust most of their heat) you shouldn't have any trouble with heat. We ran them under constant load for about 1 hour per render engine / GPU combination, which is usually enough time to reach a steady temperature state, so I would expect these to do fine for even longer rendering sessions... but I haven't personally run them constantly overnight or anything like that.
As for the PSU, for the RTX 2080 cards you need an 8-pin and a 6-pin for each GPU. I would advise something rated for at least 1200-1300W for four of them (225W per GPU = 900W plus some for the rest of the system). Moving up to the 2080 Ti cards requires a pair of 8-pin connections for each card. That may be a little hard to find, and you would want a 1500-1600W power supply for four of those.
If I have two blowers and two open air 2080 GPUs, what's the best arrangement? I ask because https://www.pugetsystems.co... suggested that the middle GPUs suffer the most. If the middle GPUs are blowers and the top and bottom are open air, then is this less of an issue?
The bottom slot, where there is nothing else below it to block the fan intake, is definitely the best spot for a dual-fan card in a multi-GPU setup. This is assuming, I suppose, that your chassis has room below that card for it to breathe - if the card bottoms out on something that blocks the fans, it won't be any better.
A second spot, though? That is tough. Probably the top slot, where the fans will be blocked but at least there isn't another card above it to prevent heat from radiating out. It will still throttle up there, but not as badly as if it were in the middle.
> It will still throttle up there, but not as badly as if it were in the middle.
By your measure, is the throttling mostly from the accumulated heat in the system or its inability to get air? If there are blowers, then the accumulated heat should be less. Yet if the open air cards need their fans to have access to much air, then the performance for an OBBO setup should not be much better than an OOOO setup (O=open, B=blower).
I'm honestly not sure - I haven't tested a mixed blower + dual fan setup before, or tried to artificially create poor cooling conditions in order to measure their effect. I suspect, though, that it comes down to how hot the air being pulled in to "cool" the card is. If it is already very hot, it won't actually help pull more heat off the card's heatsink. So having a couple blower cards should help in that regard... hopefully. Good luck :)
Hello, would appreciate a lot if you could give your two cents on the RTX 2080 MSI Aero Cards for a 2/3/4 GPU setup :)
If I am looking at the correct card (this? https://www.msi.com/Graphic... then I would expect that to be a decent option for multi-GPU configurations. The pictures there show a blower-style fan, with plenty of venting at the back end of the card (where the monitor connectors are located) and little or no venting elsewhere (so that hot air is forced out the back, rather than escaping into the computer). The only unknown, since the pictures don't show a good perspective on it, is whether there is some space around the fan for intake when two (or more) cards are placed next to each other.
What about liquid cooling, stacked without spacing between? Could you test the Titan RTX like that, like with something similar to this: "EK-Vector RTX 2080 Ti RGB - Nickel + Plexi" https://www.ekwb.com/shop/ek-vector-rtx-2080-ti-rgb-nickel-plexi? ... as suggested as a compatible water block: https://www.ekwb.com/configurator/waterblock/3831109810477#DB_inline?height=260&width=530&inline_id=comp_table and https://www.ekwb.com/configurator/
Liquid-cooling is an option when you want to have GPUs stacked next to each other, but we don't do that sort of cooling so it isn't something I'll be testing. I would also note that if you are going to have 3 or 4 cards cooled in that way then you might need radiators in-between some of the cards, since the coolant will heat up as it passes through each waterblock. That can end up being a pretty complex and expensive sort of setup, which just makes it more of a bummer that there is no option for a more affordable, rear-exhausting Titan RTX air-cooled solution. There is always the Quadro RTX 6000, of course, but it is two and a half times the price :/
Thanks William. Yeah, the rad intermittency is something I've been contemplating ... whether one 240 in-line between each, or a 480 every two ... never done LC, so no ideas yet, still piddling around in my head. Thanks, all of Puget's reviews are super down to earth!
i find a video is Q&A OTOY CEO Jules urbach, in 38:30 Jules urbach say some thing about "Multi-GPU Performance about Octene",I think maybe you need it . https://www.bilibili.com/vi...
I just watched that clip - thank you for sending it to me! - and Jules says basically the same things I did in my conclusion of more recent articles on this topic (like this one: https://www.pugetsystems.co...
To summarize:
1 - With Octane, under normal circumstances, the actual rendering speed on the cards is not impacted by the PCI-E speed (generation, lanes, etc)
2 - The PCI-E speed will impact how fast data can be sent to and from the cards, but that should only be happening at the start and end of the render, so it shouldn't have a big effect on overall performance
3 - The one exception to these generalizations is if your scene data is bigger than the amount of GPU memory you have on the cards, in which case the rendering engine has to use "out of core" memory. That is a fancy term for using the main system memory to hold part of the scene data, and then sending it to the GPU as needed. That means a lot more data going back and forth over the PCI-E bus, during the rendering process, which could lead to situations where the PCI-E speed *does* impact performance much more.