|
|
+ |
|
= | ??? |
Thinking about picking up a Xeon Phi coprocessor and taking a shot at many-core + SIMD parallel programming? Sounds like fun! Well, now what? There's a long list of things to consider but if you are thinking of just getting the card and trying to use it in an existing system you may be in for some disappointment if you expect it to work on any old motherboard.
First, Intel assumes you are plugging your new Xeon Phi into a dual socket Xeon E5 motherboard and they have a sensible reason for assuming that. Here's why. If you can't get your code to scale in parallel across the 8 to 16 cores of a dual E5 setup AND exploit the AVX vector instructions on the E5 then you aren't going to have much luck with the Phi. I can see the point but, if you are a developer and just want a more modest single CPU system, what then? Well, there are a couple of Sandy Bridge-E core i7, socket LGA 2011, X79 chipset boards that will work with Phi, (ASUS X79 boards note: "WS" version has support, others may not!). I'm looking at some newer "Haswell" boards now but I don't have any validated yet.
The biggest problem seems to be the requirement,
"BIOS support for memory mapped I/O address ranges above 4GB"
the Xeon Phi requires a motherboard and BIOS with large "Base Address Register" support!
What's that? When a system starts up one of the things the BIOS does is look for devices on the PCI bus. When it finds a device it queries for the "resources" being requested by the device. These resources can be a request for a memory mapped I/O (MMIO) address range. It then reserves the requested block of address and assigns a start address for the block to one of the devices base address registers. Those address are what the system uses to communicate with the device. The Xeon Phi requires that the BIOS provides a "large" i.e. 64-bit address for that.
Where to look for this in your BIOS
Start your system and hit the magic key that will drop you into the BIOS settings screens. (it may be the [Del] key or one of the F keys like F2 maybe …) Look for *Advanced settings* and then look for *PCI settings*. on an Intel board you may find something like; "Memory Mapped I/O above 4GB [Enable]" on an ASUS board it will probably look like; "PCI 64bit Resource Handling Above 4G Decoding [Enabled]" You might also see something referring to "MMIO above 4G" or maybe even "large BAR support". What you hope to find is *something* and an option to [Enable] it. It may or may not be enabled by default. If you don't find anything like this in your BIOS then all bets are off as to whether your Xeon Phi is going to work or not. Actually, my bet is that it wont work!
If you do have the right PCI 64bit resource support does that mean your board will work with Phi? NO! It's necessary but not sufficient.
Unfortunately I have tried boards that looked promising and had the right PCI settings but still did not fully assign resources correctly for Phi.
Here's a hint at why the Phi needs that large address space decoding.
Look at a few lines of output from lspci -vv below for a Phi card,
[kinghorn@phi1 ~]$ lspci -s 84:00 -vv 84:00.0 Co-processor: Intel Corporation Device 2250 (rev 11)
Subsystem: Intel Corporation Device 2500
...
Region 0: Memory at 3c1c00000000 (64-bit, prefetchable) [size=8G]
Region 4: Memory at fbd00000 (64-bit, non-prefetchable) [size=128K]
...
Take note of that [size=8G]!
Contrast that with the lspci output for an NVIDIA Titan card with 6GB
of memory,
[kinghorn@phi1 ~]$ lspci -s 03:00.0 -vv
03:00.0 VGA compatible controller: NVIDIA Corporation Device 1005 (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 1035
...
Region 0: Memory at 5a000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 50000000 (64-bit, prefetchable) [size=128M]
Region 3: Memory at 58000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at a000 [size=128]
...
I think that [size=8G] in there from the Phi lspci output is a big hint about why Phi has the larger BAR requirement! What you see in the Titan lspci output is more what I'm used to seeing, a modest " aperture"="" to="" the="" device="" memory.
The bottom line is, unless your vendor has validated a specific board for use with the Xeon Phi then don't expect it to work … I wish it wasn't so.