Running on Xeon-Phi nodes

Getting started

We have 39 nodes on the cluster each with two Xeon-Phi cards. A Xeon-Phi card runs a cut-down version of Linux, and works in practice as an individual node, albeit with a very constrained set of software. Below we will call

  • a host computer with Xeon-Phi cards installed for a "Xeon-Phi node" or "host"
  • a Xeon-Phi card for a "Xeon-Phi card", "Xeon-Phi" or "card"

A Xeon-Phi card has 60 cores, each with 4 hyper threads giving a total of 240 logical cores. It is equipped with 8GB of memory. Therefore the memory per core is extremely constrained at 8 GB / 240 = 34 MB / logical core.

A Xeon-Phi core is essentially an old Pentium II core, which has got a 512 bit wide vector unit attached. Key to good performance on the Xeon-Phi's is good vectorisation in the code.

If you just want to run a serial program, you can do it by

  1. starting an interactive job in the Xeon-Phi queue (astro_phi)
  2. Compiling your program for Xeon-Phi's on the host node (see below)
  3. logging in to a phi card from the host.

The full hostname of each card is: nodeXXX-mic[0,1], but the shorthand name is mi0 and mic1. Therefore "ssh mic0" will get you on to the card. 4) Executing the program on the node

 

Running interactively

Running interactively on a Xeon-Phi node can be done just like on any other node on the cluster, but it is important that you specify that you need Xeon-Phi resources

srun -p astro_phi --gres=mic:2 --nodes=1 --cpus-per-task=40 --ntasks-per-core=2 --pty bash

this will start an interactive bash shell on one of the Xeon-Phi nodes

 

 Compiling programs for running on Xeon-Phi

The micro-architecture of Xeon-Phis is slightly different from standard x86. Therefore all programs have to be recompiled before they can run on the card. Code for the Xeon-Phi card has to be compiled with the intel fortran compilers.

We recommend you to use as new a version of the compiler as possible (as of this writing, version 16.0.0), given that significant updates to Xeon-Phi compilation is happening with the latest compiler updates.

To compile for Xeon-Phi you have to use the -mmic compiler switch

For example the program:

PROGRAM hello
print *, 'Hello World'
END

can be compiled with

ifort -mmic helloworld.f90 -o hw.mic

 

Using MPI with the Xeon Phis

This section will be expanded. Running a program in MPI mode on the Xeon-Phi cards is non-trivial.

The automatic linking of the SLURM library has to be disabled, because that library does not exist on the Xeon-Phi has to be disabled. This can be done by using the ".org" compilers; e.g.

mpiifort.org, mpiicc.org, mpiicpc.org

Furthermore, a number of environment variables, related to letting SLURM control the startup and distribution of MPI ranks has to be turned off. This can be accomplished by loading the phi module after you have loaded the compiler and intel-MPI module:

module load intel/16.0.0
module load intelmpi/5.1.1.109
module load phi

Finally, instead of using srun, the SLURM job launcher, one has to use mpiexec.hydra for launching the job.

Because the host file is not generated automatically, this currently has to be done by hand. F.ex. to use the four cards on node850 and node851, one could have a host file with the lines

node850-mic0
node850-mic1
node851-mic0
node851-mic1

This can be generated automatically using a small SLURM utility that generates that outputs the host list inside a job

scontrol show hostnames | xargs -I H echo H-mic0 > tmp; scontrol show hostnames | xargs -I H echo H-mic1 >> tmp; sort tmp > hosts; rm tmp

Best practices and tips