Titan X Performance: PCI-E 3.0 x8 vs x16
CPUs and chipsets only have a certain number of PCI-E lanes (or "paths" for data transfer), but with the number of features and options that manufacturers are cramming onto modern motherboards, there are times when there are simply not enough PCI-E lanes to run everything at full speed. We see this most often with the mainstream series of CPUs (currently Haswell) and chipsets (Z87, H87, etc.) where if you want to use multiple video cards (or install any other type of PCI-E card in the second x16 slot), the video card(s) will only run at x8 speeds rather than the full x16 speed you would otherwise get.
Normally, if you want to run a pair of video cards at full x16 speeds, you need an enthusiast motherboard and CPU (such as an Ivy Bridge-E CPU on a X79 motherboard). The problem with this is that NVIDIA video cards currently do not officially support PCI-E Gen3. You can run an .exe that NVIDIA has provided to gain Gen3 capabilities, but NVIDIA does not offer any guarantee of stability if you use that utility.
Many hardware sites (such as TechPowerUp and AnandTech) have shown in the past that most video cards do not show any performance decrease by running in x8 mode and cannot utilize the larger bandwidth provided by the latest Gen3 specification. However, video cards are getting faster and faster so we felt it was worth revisiting to find out if the fastest video cards available still do not have any performance advantage running at PCI-E x16 Gen3 versus PCI-E x8 Gen2. In addition, multiple GPU setups have not been throughly tested and with the gaining popularity of 4k displays, we felt it was important to see if the PCI-E revision/speed would affect a dual GPU setup at the much more demanding 4k resolution.
To see how the chipset and CPU affects our results, we used two separate testing platforms consisting of the following hardware:
|Z87 Test System||X79 Test System|
|Motherboard:||ASUS Sabertooth Z87||ASUS P9X79 Deluxe|
|CPU:||Intel Core i7 4770K 3.5GHz Quad Core||Intel Core i7 4960X 3.6GHz Six Core|
|Used in both systems|
|PSU:||Seasonic X-1050 1050watt|
|RAM:||4x Kingston HyperX DDR3-1600 4GB Low Voltage|
|Video Card(s):||NVIDIA Geforce GTX Titan 6GB (driver 327.23)|
|Hard Drive:||Samsung 840 Pro 256GB|
|OS:||Windows 8.1 Pro 64-bit|
We will be using three different benchmarks at two different resolutions: 1080p and 4k. At 1080p we will only be using a single GTX Titan, but at 4k we will be using a pair of GTX Titans in SLI. The settings for the benchmarks were chosen to give us an average 40-50 FPS.
To test both PCI-E 2.0 and PCI-E 3.0, we simply changed the PCI-E revision setting in the BIOS to either Gen2 or Gen3.
Finally, to switch between x16 and x8 modes a piece of insulating material (actually the sticky part of a post-it note) was used to cover half of the contacts on the cards which forces it to run in x8 mode. Note that we did not have to do this on the Z87 motherboard when we used two GTX Titans in SLI as that CPU/chipset can only run multiple video cards at dual x8 speeds, not dual x16.
Starting with Unigine Heaven 4.0, lets first take a closer look at the Z87 test system. On that system, PCI-E 3.0 was very slightly faster than PCI-E 2.0, but oddly we saw higher scores in x8 mode than in x16 mode. The biggest variance was still only 1 FPS, however, which we would call within our margin of error.
On the X79 system, there is really nothing to discuss. The biggest variance was only .4 FPS which is well within our margin of error.
Our second benchmark - Hitman: Absolution - has some very interesting results. This time, the Z87 test system showed pretty much no performance difference across any of our PCI-E combinations. On the other hand, the X79 system gives us very mixed results. In x16 mode, PCI-E 2.0 is faster than PCI-E 3.0, but only by .6 FPS. But in x8 mode, PCI-E 3.0 is 1.2 FPS faster than PCI-E 2.0. 1.2 FPS is still not very much, but it is noteworthy.
For our final benchmark - Metro: Last Light - our results are again unremarkable. On the Z87 test system, the results were all essentially identical. The X79 system was slightly faster when running PCI-E 3.0 in both x8 and x16 mode, but only by .3-.7 FPS.
Running our benchmarks at 4k resolutions with two video cards should stress the PCI-E bus more than a single card at lower resolutions, but at least for Unigine Heaven there is no noteable difference in performance based on either the PCI-E revision or the number of PCI-E lanes.
Unlike Unigine Heaven 4.0, Hitman: Absolution does give us some interesting data to go over, at least for the X79 test system. On that system, at x16 speeds PCI-E 2.0 is actually 1.2 FPS faster than PCI-E 3.0. At x8 speeds, however, PCI-E 3.0 is actually much faster than PCI-E 2.0 by 1.5 FPS which is the biggest variance we saw in any of our tests.
Our final benchmark is actually the first one that gave us the results we originally expected. The Z87 system again shows no performance variance, but for the X79 system PCI-E 3.0 is either faster or the same as PCI-E 2.0 at both x8 and x16 speeds. At the same time, x16 outperforms x8 by as much as 1.5 FPS.
Our testing has pretty clearly shown that for gaming using either PCI-E 2.0 or PCI-E 3.0 will give you nearly identical performance. Oddly, in some benchmarks PCI-E 2.0 was actually faster than PCI-E 3.0. At the same time, x16 was not consistantly faster than x8. Again, x8 was actually faster than x16 in many cases. So unless you care about getting up to 1.5 FPS better performance, you might actually want to manually set your video cards to operate at x8 speeds - although we really would not recommend doing so.
This isn't to say that PCI-E 3.0 is not faster than PCI-E 2.0, or that x16 is the same as x8, but rather that current video cards and games are simply not able to utilize the additional bandwidth they provide. In fact, we recently showed that the performance of a Xeon Phi card is greatly reduced if you run it at x8 speeds in the blog post Performance of Xeon Phi on PCIe x8.
While we recommend using the latest PCI-E revision whenever possible, if your motherboard or video card only supports PCI-E 2.0 our results show that this really is not a problem. At the same time, if you want to install a sound card into your Z87 system but doing so would limit your video card to x8 speeds, that is also not a very big problem. At most you may see ~1.5 FPS drop in performance, but that change is so small that it is very unlikely to ever be noticeable.