Home > Puget Systems Blog > Will your motherboard work with Intel Xeon Phi?
Will your motherboard work with Intel Xeon Phi?
Dr. Donald Kinghorn (HPC and Scientific Computing)

Will your motherboard work with Intel Xeon Phi?

Posted on August 6, 2013 by Dr. Donald Kinghorn

[ View All Blog Posts ]


             

24655 + 24656 = ???

Thinking about picking up a Xeon Phi coprocessor and taking a shot at many-core + SIMD parallel programming? Sounds like fun! Well, now what? There's a long list of things to consider but if you are thinking of just getting the card and trying to use it in an existing system you may be in for some disappointment if you expect it to work on any old motherboard.

First, Intel assumes you are plugging your new Xeon Phi into a dual socket Xeon E5 motherboard and they have a sensible reason for assuming that. Here's why. If you can't get your code to scale in parallel across the 8 to 16 cores of a dual E5 setup AND exploit the AVX vector instructions on the E5 then you aren't going to have much luck with the Phi. I can see the point but, if you are a developer and just want a more modest single CPU system, what then? Well, there are a couple of Sandy Bridge-E core i7, socket LGA 2011, X79 chipset boards that will work with Phi, (ASUS X79 boards note: "WS" version has support, others may not!). I'm looking at some newer "Haswell" boards now but I don't have any validated yet.

The biggest problem seems to be the requirement,

"BIOS support for memory mapped I/O address ranges above 4GB"

the Xeon Phi requires a motherboard and BIOS with large "Base Address Register" support!

What's that? When a system starts up one of the things the BIOS does is look for devices on the PCI bus. When it finds a device it queries for the "resources" being requested by the device. These resources can be a request for a memory mapped I/O (MMIO) address range. It then reserves the requested block of address and assigns a start address for the block to one of the devices base address registers. Those address are what the system uses to communicate with the device. The Xeon Phi requires that the BIOS provides a "large" i.e. 64-bit address for that.

Where to look for this in your BIOS

Start your system and hit the magic key that will drop you into the BIOS settings screens. (it may be the [Del] key or one of the F keys like F2 maybe ...) Look for *Advanced settings* and then look for  *PCI settings*. on an Intel board you may find something like; "Memory Mapped I/O above 4GB    [Enable]" on an ASUS board it will probably look like; "PCI 64bit Resource Handling Above 4G Decoding    [Enabled]" You might also see something referring to "MMIO above 4G" or maybe even "large BAR support". What you hope to find is *something* and an option to [Enable] it. It may or may not be enabled by default. If you don't find anything like this in your BIOS then all bets are off as to whether your Xeon Phi is going to work or not. Actually, my bet is that it wont work!

If you do have the right PCI 64bit resource support does that mean your board will work with Phi? NO! It's necessary but not sufficient.

Unfortunately I have tried boards that looked promising and had the right PCI settings but still did not fully assign resources correctly for Phi.

Here's a hint at why the Phi needs that large address space decoding.

Look at a few lines of output from  lspci -vv below for a Phi card,  

 

 

[kinghorn@phi1 ~]$ lspci -s 84:00 -vv 84:00.0 Co-processor: Intel Corporation Device 2250 (rev 11) 
         Subsystem: Intel Corporation Device 2500 
... 
         Region 0: Memory at 3c1c00000000 (64-bit, prefetchable) [size=8G] 
         Region 4: Memory at fbd00000 (64-bit, non-prefetchable) [size=128K] 
...

Take note of that [size=8G]!

Contrast that with the lspci output for an NVIDIA Titan card with 6GB
of memory,


[kinghorn@phi1 ~]$ lspci -s 03:00.0  -vv
03:00.0 VGA compatible controller: NVIDIA Corporation Device 1005 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation Device 1035
...
        Region 0: Memory at 5a000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at 50000000 (64-bit, prefetchable) [size=128M]
        Region 3: Memory at 58000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at a000 [size=128]
...

I think that [size=8G] in there from the Phi lspci output is a big hint about why Phi has the larger BAR requirement! What you see in the Titan lspci output is more what I'm used to seeing, a modest " aperture"="" to="" the="" device="" memory.

The bottom line is, unless your vendor has validated a specific board for use with the Xeon Phi then don't expect it to work ... I wish it wasn't so.

 


Tags: Intel, Xeon Phi, HPC


Share this blog post!

Rakib Hasan

Hi Dr. Kinghorn,
Thank you very much for the article.
I saw that you said ASUS X79 boards will work.
I am thinking about building a machine with ASUS P9X79LE board with an
Intel Core i7-3820 Sandy Bridge-E 3.6GHz 4-core processor. Do you think
it will work?
Also, before buying, I was checking the manual for the motherboard but couldn't find that option. If you have that board, could you point out the path for that option in BIOS?

Posted on 2013-10-05 03:28:53
Benjamin Lowry

Just updated my bios for my asus Z9PE-16 2/L and enabled the 4g decoding and it appears to have worked. Thanks for this guide, message me if anyone needs help with this!

Posted on 2014-01-24 00:00:52
Mickhael Young

hello there mate i was wondering i've just bought the asus z9pe d8 ws mother board, and had no idea how the xeon phi works on it, and plus i heard it needs to be optimize to get the best out of it, and so how is it to optimize the xeon phi for autodesk 3ds max?

Posted on 2014-04-14 16:16:47
Jeremy

Hi Dr. Kinghorn:
Thank you for a very insightful article. Nowadays, x79 mainboards are a rare find in my locale. The most common models are based on the z87 chipsets (socket 1150). Will motherboards built on this chipset work with Xeon Phi?

Posted on 2014-02-01 14:01:32
Gabriel Araujo

My Asus Maximus VI Extreme has (memory remap to enable or disable memory remap above 4G) Does it support Xeon Phi?

Posted on 2014-05-10 15:30:45
Donald Kinghorn

Hi Gabriel, I believe that is referring to being able to "use" system memory above 4GB. The support you need for Phi is on the PCI bus. ASUS uses the phrase "above 4G decoding" to describe this under PCI settings. It is actually referrign to large "Base Adress Register" support for on the PCI . It's worded badly in the BIOS manuals. ASUS boards that support Xeon Phi directly will say so explicitly in the advertising copy or in the manual explicitly.
Best regards --dbk

Posted on 2014-05-11 19:33:51
Gabriel Araujo

Thank you for your reply.

I researched more and found out that ASUS motherboard supports intel xeon phi. Is the board Z9PE-D8 WS, P8Z77 WS and P9X79 WS, support perfectly with Phi 3100 series and 5110P

Posted on 2014-05-12 12:32:09
andalib

hello i have macbook pro with thunderbolt port.is it possible to use intel xeon phi co-processor using external pcie chassis.please let me know

Posted on 2014-06-08 05:33:25
Donald Kinghorn

It could conceivably work but I think it's pretty unlikely. There are way too many things that could break/not work.
1) I don't know if thunderbolt supports 64-bit base address registers ( it might/should )

2) If 1 is OK then you still need a 3.x linux kernel for thunderbolt support
3) there is no Phi MPSS driver for 3.x kernels
4) you would have to run linux on your macbook (yuk!)

Posted on 2014-06-08 20:13:35
andalib

thanks Mr. Donald

thunderbolt supports 64-bit base address registers.but the thing is my macbook processor is core i7.i have seen in youtube intel guyz using xeon phi with intel xeon e5/e7 processor.i am just curious about xeon phi. if it works in my macbook then i can virtualize so many OS simultaneously

Posted on 2014-06-09 06:00:59
Donald Kinghorn

Two things:
It's not going to work with mac OS X OR thunderbolt at this point in time (I talk with Intel about thinderbolt latter this week ..., right now this kind of usage does not work with thunderbolt)
Second there is no way to use the card for virtualization the way your are thinking (i.e. you can't run virtual machines ON the Phi. ...)

Wait until the end of the year and look at it again, things may be different.
Best wishes --dbk

Posted on 2014-06-09 18:23:49
See a problem on this page? Let us know.