Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/1058
Matt Bach (Senior Puget Labs Technician)

Why Do Hardware Reviewers Get Different Benchmark Results?

Written on October 7, 2017 by Matt Bach
Share:

Recently, "Tomasz Odwald" on Twitter noticed that a number of hardware reviews reported very different results in Cinebench R15 with the new Core i7 8700K CPU:

Speculation in the Twitter thread range from different RAM speeds to differences in room temperature, but in our experience "SirNoct_" is actually the closest with their answer of "It could be because of overclocking or some other type of hardware". If you look at a number of reviews for the Core i7 8700K and compare both the Cinebench multi-core score and the motherboard used, you will get the following:

Review Cinebench R15 Multi-Core Scores Motherboard Used
Overclock3D 1583 (MultiCore Enhancement on)
1429 (MultiCore Enhancement off)
Asus Z370-A Prime
Linus Tech Tips 1547 Asus Z370-A Prime
JayzTwoCents 1527 Asus ROG STRIZ Z370-E Gaming
ExtremeTech 1446 Asus Z370-A Prime
Paul's Hardware 1430 Gigabyte Aorus Z370 Ultra Gaming
Anandtech 1364 Gigabyte Z370 Gaming 7
BitWit 1284 Gigabyte Aorus Z370 Gaming 7

Sorting by Cinebench score makes the cause of the different results pretty obvious. Asus boards score high, Gigabyte scores low - except when Overclock3D diabled Multicore enhancement in the BIOS. In our experience, the issue is not that Gigabyte boards are inferior but rather that Asus is auto-overclocking without informing the user. This is something we first noticed on X99 Asus boards, but it carried over to X299 and is now apparently an issue on Z370. To be clear, we are not just relying on these reviews - we've seen the same behavior ourselves while qualifying motherboards from Gigabyte, Asus, and MSI for use in our product line.

The cause of this is easier to understand if we first explain how Turbo Boost 2.0 works on modern Intel CPUs. In short, Intel CPUs with Turbo Boost are supposed to run at specific speeds depending on the number of cores that are being loaded. Using the new Core i7 8700K as an example, you should see the following core frequencies:

1 Core 2 Cores 3 Cores 4 Cores 5 Cores 6 Cores
4.7GHz 4.6GHz 4.5GHz 4.4GHz 4.4GHz 4.3GHz

Factors like thermal and power limits can affect these numbers, but in most cases this is how fast a Core i7 8700K should run under different cores loads. The problem is that Asus in particular tends to take this as a very loose guideline rather than a hard specification. In fact, if you leave everything on "Auto" in the BIOS, Asus boards tend to run the CPU at the max single core Turbo frequency even when all the cores are loaded. So in an application like Cinebench that loads all the cores, an Asus board might be running the Core i7 8700K at 4.7Ghz on all cores instead of the 4.3GHz that it should be. This is by clearly overclocking since the BIOS will not only be upping the CPU frequency well beyond Intel's specifications, but will also be increasing the CPU voltage to match.

On a 6 core CPU like the Core i7 8700K, the difference between running at 4.7GHz versus 4.3GHz on all cores works out to be about a 10% difference assuming perfect scaling across multiple CPU cores. Most software is nowhere near perfect at using multiple cores, but Cinebench (and 3D rendering in general) does scale extremely well and it is no coincidence that the reviewers using Asus boards posted results that are almost exactly 10% faster than those using Gigabyte boards.

To be fair to Asus, this is an issue we have been seeing with increasing frequency on a number of motherboard brands. In our experience, Gigabyte tends to be pretty close to spec after launch, although we have had to work with them a number of times to get BIOS fixes on pre-launch BIOS versions. MSI also has some issues pre-launch, but they seem to favor underclocking rather than overclocking. EVGA we don't have a huge amount of experience with, but for the boards we carry we have found them to usually be pretty on target and don't underclock or overclock.

While we don't have proof, we believe that this kind of overclocking without informing the user may very well be a factor in the reported X299 VRM overheating issues. It also makes us very concerned about the recent report from ExtremeTech that

Intel Will No Longer Provide Per-Core Turbo Frequencies, Making Motherboard Tuning Impossible

If we can no longer be sure what speeds Intel CPUs should be running at under different loads, how can we be sure that motherboard manufacturer are not auto-overclocking our CPUs without our consent just to make their products look better than their competitors?

The good part is that at least on Asus boards, this "feature" can usually be easily fixed by either disabling "MultiCore Enhancement" or by changing the CPU Core Ratio setting in the BIOS from Auto to By Core Usage. If you are a hardware reviewer who happens to read this post, we highly recommend including in your testing a verification that the CPU is actually running at the speeds it is supposed to be. This is an issue we see all the time with Asus boards, but it does pop up on other brands as well - especially when using pre-launch or early BIOS versions.

Edit 10/8/2019: We forgot about "Multicore Enhancement" when we first published this article which is another BIOS setting that can also result in auto-overclock. Overclock3D actually has great explanation of this in their review and they call it out specifically in their charts as an overclocked result. The odd thing about this setting is that in our experience it is not always the culprit of auto-overclocking. It often is, but we also have found that we need to change the CPU Core Ratio as well before the CPU will run at stock speeds.

Here at Puget Systems, the method we use to verify that a CPU is running at the proper speeds is to use an application that can monitor the individual CPU core frequency (not just the maximum or average) such as CoreTemp and combine it with Prime95 to load the cores. Just limiting the thread count in Prime95 won't cut it since it can randomly bounce around the different cores, so we instead leave it on the maximum thread count and use Windows affinity to manually define which cores it is allowed to run on. This way, you can verify the all-core frequency as well as the single and other multi-core frequencies. This is especially useful for X299 CPUs to ensure that Turbo Boost 3.0 is working properly on the two strong cores which is an extremely common problem on X299 boards if Turbo Boost 3.0 is left in "Native" mode.

Really, this whole issue is caused by three factors:

  1. Motherboard manufactures (not just Asus) don't always follow Intel's specifications and often fail to let the user know that they are running the CPU outside of spec.
  2. Intel is publishing frequency specifications (even if not publicly), but not holding motherboard manufacturers responsible when they run their processors outside of spec without the end user's consent.
  3. Reviewers are not verifying that the hardware they are testing is running at the advertised speeds - although to be fair their job is to review hardware as it ships and shouldn't have to check for things like automatic overclocking.

The vast majority of hardware reviewers are likely not even aware of the possibility that they might be testing on auto-overclocked CPUs and to be fair, this hasn't been a major issue until fairly recently. However, this kind of testing is something we've been doing on all our workstations for years and we would love to see it become a regular part of hardware reviews as well.

Tags: CPU, Overclocking
SirNoct_

Oh wow I'm in a Pugest systems Article I'm famous but yeah I thought it would be something to do with the differences in the physical Hardware and or differences is overclocking. such as motherboards and different 8700k's

Posted on 2017-10-08 02:17:14
AliasNorth

good article, thanks.

automatic overclocking seems weird.

Posted on 2017-10-08 06:25:03
Gio Tech

to be honest this is a good strategy from Asus. however they should have informed the user of this feature. I guess it goes to show that Asus as a company is very confident with their motherboards. as for the first two results (overclock 3d and linus) it could be a case of the infamous "silicon lottery" or during the sandy bridge days we call them "golden chips". k processors that were highly overclockable. I do agree that reviewers should have been informed before conducting the tests.

Posted on 2017-10-08 10:01:18
Standing Order

It is worth noting here that OC3D conducted testing with ASUS Multi-Core enhancement turned on an off. It is misleading to just show their Multi-core enhanced results here.

Posted on 2017-10-08 19:42:13

You're right, I missed that as I was pulling together all the different review results. I updated the chart and added an edit to explain that Overclock3D is aware of this setting and that they specifically call out this BIOS setting as causing an overclocked result. Thanks for mentioning that, and good job Overclock3D for realizing that the default setting caused the CPU to overclock!

Posted on 2017-10-08 20:18:04
Danilo

I agree 100% with the article. I was discussing about this topic in a Brazilian forum on facebook (Teclab Extreme Pc's) and I got banned just because I pointed all of this issues. They think that the motherboard's manufacturers should disrespect intel's specifications. That this is going to be good for heavy users. In my opinion it will just make the user more confused when to decide which motherboard buy. The one with ehancements that provides more power (but that can kill your cpu aswell) or the one that respects all the specs from the cpu manufacturer.

Not to mention that the so called MCE is very unstable, increases the temperature and the power usage, making the system very unstable under load on some stress tests (like blender for example).

The press on the other side (sold or not) show to its public unrealistic results that may lead users to believe that an I7 8700k is fully capable of reach 1500/1600 stable on cinebench. Yes it does (with mce). But it also crashes in other tests. So it cannot be considered a consistent result to be published.

That happened with JayzTwoCents, Linus and other reviewers that runned the tests on Asus motherboards with the MCE enabled by default.

The thing is. If you have an I7 8700k, do not delid your CPU and do not activate MCE. Your warranty period is very valueable. That's what I think. Intel is kind turning the blind eye with its certifications in order to run away with its main responsibilities. Warranty is one of them. And intel is responsible for all motherboard's manufacturers that are not respecting its certification.

Posted on 2017-10-16 05:23:54