We have 39 nodes on the cluster each with two Xeon-Phi cards. A Xeon-Phi card runs a cut-down version of Linux, and works in practice as an individual node, albeit with a very constrained set of software. Below we will call
- a host computer with Xeon-Phi cards installed for a "Xeon-Phi node" or "host"
- a Xeon-Phi card for a "Xeon-Phi card", "Xeon-Phi" or "card"
A Xeon-Phi card has 60 cores, each with 4 hyper threads giving a total of 240 logical cores. It is equipped with 8GB of memory. Therefore the memory per core is extremely constrained at 8 GB / 240 = 34 MB / logical core.
A Xeon-Phi core is essentially an old Pentium II core, which has got a 512 bit wide vector unit attached. Key to good performance on the Xeon-Phi's is good vectorisation in the code.
If you just want to run a serial program, you can do it by
- starting an interactive job in the Xeon-Phi queue (astro_phi)
- Compiling your program for Xeon-Phi's on the host node (see below)
- logging in to a phi card from the host.
The full hostname of each card is: nodeXXX-mic[0,1], but the shorthand name is mi0 and mic1. Therefore "ssh mic0" will get you on to the card. 4) Executing the program on the node
Running interactively on a Xeon-Phi node can be done just like on any other node on the cluster, but it is important that you specify that you need Xeon-Phi resources
srun -p astro_phi --gres=mic:2 --nodes=1 --cpus-per-task=40 --ntasks-per-core=2 --pty bash
this will start an interactive bash shell on one of the Xeon-Phi nodes
Compiling programs for running on Xeon-Phi
The micro-architecture of Xeon-Phis is slightly different from standard x86. Therefore all programs have to be recompiled before they can run on the card. Code for the Xeon-Phi card has to be compiled with the intel fortran compilers.
We recommend you to use as new a version of the compiler as possible (as of this writing, version 16.0.0), given that significant updates to Xeon-Phi compilation is happening with the latest compiler updates.
To compile for Xeon-Phi you have to use the
-mmic compiler switch
For example the program:
PROGRAM hello print *, 'Hello World' END
can be compiled with
ifort -mmic helloworld.f90 -o hw.mic
Using MPI with the Xeon Phis
This section will be expanded. Running a program in MPI mode on the Xeon-Phi cards is non-trivial.
The automatic linking of the SLURM library has to be disabled, because that library does not exist on the Xeon-Phi has to be disabled. This can be done by using the ".org" compilers; e.g.
mpiifort.org, mpiicc.org, mpiicpc.org
Furthermore, a number of environment variables, related to letting SLURM control the startup and distribution of MPI ranks has to be turned off. This can be accomplished by loading the phi module after you have loaded the compiler and intel-MPI module:
module load intel/16.0.0 module load intelmpi/126.96.36.199 module load phi
Finally, instead of using
srun, the SLURM job launcher, one has to use
mpiexec.hydra for launching the job.
Because the host file is not generated automatically, this currently has to be done by hand. F.ex. to use the four cards on
node851, one could have a host file with the lines
node850-mic0 node850-mic1 node851-mic0 node851-mic1
This can be generated automatically using a small SLURM utility that generates that outputs the host list inside a job
scontrol show hostnames | xargs -I H echo H-mic0 > tmp; scontrol show hostnames | xargs -I H echo H-mic1 >> tmp; sort tmp > hosts; rm tmp
Best practices and tips