Hyper-Threading may be Killing your Parallel Performance

Hyper-Threading, hyperthreading, or just HT for short, has been around on Intel processors for over a decade and it still confuses people. I’m not going to do much to help with the confusion. I just want to point out an example from some testing I was doing recently with the ray-tracing application POV-ray that surprised me. Hyper-threading dramatically lowered the performance on a multi-core test system running Windows when running POV-ray in parallel.

POV-ray

POV-ray is a ray tracing program that has been running in parallel for many years and is well maintained and supported on Windows and Linux. It’s often used as a CPU benchmark because of its heavy computational demands and its parallel implementation. I was doing some benchmarking on our quad socket systems when I hit an unexpected anomaly with Hyper-Threading under Windows server 2008 R2. It was interesting enough that I thought I’d use it as an opportunity to talk about Hyper-Threading a bit.

Hyper-Threading … not so great (in my experience)

It will not come as a surprise to anyone who has worked with computers for HPC that Hyper-Threading is usually detrimental to performance (or at best does nothing). The HPC sys-admin practice for Intel based systems, that are going to be used for any serious computing or parallel workloads, is to disable Hyper-Threading in the BIOS by default. In the first few years of Hyper-Threading existence it was not unusual to see from 10-40% performance loss on a system if it was left enabled. However, in the last few years I haven’t really seen much system performance loss from leaving it on so I usually don’t bother with disabling it anymore (just in case there is something that might benefit from it). I know how many real cores are in my own systems and what performance I can get out of them. Just remember, for computation heavy workloads Hyper-Threading doesn’t add any real compute capability.

Hyper-Threading is really meant for “better” thread scheduling. I can see how that might be a good thing is some cases. Imagine a code that is structured so that it is spawning lots of parallel work threads from a pool of “work” in a “round-robin” type of process. If you can eliminate the overhead of the thread spawning process you may get a speedup. (?) I would also guess there are codes that are not well adapted for parallel execution that may get an “accidental” boost too.

“However, I have never personally seen ANY program benefit from Hyper-Threading. I’m guessing there are some codes out there that may benefit, but I’ve never experienced it first hand. If I find one I’ll let you know …”  I found one! cinebench, see comments –dbk

I don’t see much harm with leaving Hyper-Threading enabled these days but if you are running a demanding application it is absolutely worth seeing if disabling it in the BIOS helps with performance. I would most likely leave it enabled on a “consumer” processor like the core-i7 … but test! Having said that, my computer use is mostly on Linux and modern Linux kernels are great at process scheduling and don’t seem to have any trouble with Hyper-Threading. I don't personally use Windows in a demanding way very often and this is where the big (bad) “surprise” came in.

POV-ray bad scaling on Windows with hyperthreading enabled

The plot above shows a lot of information. It is a plot of “speedup” vs “number of CPU threads” for the standard benchmark scene calculation in POV-ray 3.7.0. run on our testbench quad Xeon system (4 x E5-4624L v2 10-core) There are 40 real cores. Jobs were run on from 1 to 40 cores under CentOS 6.5 Linux, and Windows Server 2008 R2 with Hyper-Threading on and off. This plot is showing the “parallel scaling” of POV-ray on this system. The straight diagonal line is “perfect” linear scaling. The measured performance points are fit against Amdahl’s Law to show the parallel scaling fall-off with increasing thread process count.

Major Findings

  • Hyper-Threading made no difference under Linux (results were the same with it on or off)
  • Windows 2008 R2 with Hyper-threading on showed very poor thread scaling. This was shockingly unexpected!
  • Disabling Hyper-Threading in the BIOS greatly improved the parallel performance with Windows 2008 R2.
  • Overall performance on Linux was better than Windows. (I’ll have another post up soon that will have the performance data and more details about the POV-ray testing )

Caveats

  • Remember this is just one particular application POV-ray 3.7. I had recently tested Zemax OpticStudio on this system with the same Windows 2008 R2 install and the scaling was fine with Hyper-Threading on.
  • Also note that this is an older Windows Server version, things may be different on Server 2012. (you have to run a Server version to get support to run on a quad socket system with Windows)

Hope you found this interesting!

Happy computing! –dbk