Hardware

Cluster Hardware

 Queue Nodes CPUs RAM Accelerators (if any)  Notes

 astro_long

astro_short

astro_devel

139  2x 10-core Xeon E5-2680v2 @ 2.8GHz 64 GB - 3.2 GB / core. DDR3-1866 MHz 

100 nodes: none

39 nodes: 2x Xeon Phi 5110P

- L2: 256KB/core, L3: 25MB total, newest supported extension: AVX.

- Xeon Phi cards have 60 cores at ~1 GHz, 8GB RAM, 30MB L2 cache.

 astro_gpu

 30 2x 4-core Xeon E5640  @ 2.67GHz 72 GB - 9 GB / core. DDR3-1333 MHz 

27 nodes: 4x Tesla C2050

3 nodes: 4x Tesla C1060

- L2: 256KB/core, L3: 12MB total, newest supported extension: SSE4.2.

- Tesla C1060: 4GB RAM, 240 thread procs., CUDA compute capability v1.3.

- Tesla C2050: 3072MB RAM, 448 thread procs., CUDA compute capability v2.0 (Fermi).

 astro_fe  4  4x 8-core Xeon E5-4650  @ 2.70GHz  768 GB - 24 GB / core  Quadro K2000 (but they are not set up for GPU computing)

- L2: 256KB/core, L3: 20MB total, newest supported extension: AVX.

- provides queue access to frontend/analysis nodes (astro06 - astro09).

Notes

  • See here for information on the different queues.
  • Individual queues are interconnected with inifiniband network.
  • The 20 core astro nodes have FDR (Fourteen Data Rate) with a 2:1 blocking factor and 24 nodes per switch (6 uplink switches, 2 core switches).
  • The astro_gpu nodes have QDR (Quad Data Rate) infiniband.
  • The astro_gpu nodes are based on a Supermicro SuperServer SYS-7046GT-TRF-TC4 node with dual quad-core Westmere CPUs.
  • The astro_long|short|devel nodes are based on a Dell shoe-box design with a dual 10-core ivy-bridge CPU. The plain nodes are Dell C6220II, while the nodes with Xeon-Phi cards are Dell C8220x.

 

Analysis Hardware

Name CPUs RAM  GPUs Local Scratch Space (if any)  Notes
 astro01.hpc.ku.dk  8 cores: 2x Xeon X5550 @ 2.67GHz  72 GB - 9 GB / core  4x Tesla C1060    

- Identical to a GPU node.

- L2: 256KB/core, L3: 8MB, newest supported extension: SSE4.2.

astro04  48 cores: 8x Xeon X5680 @ 3.33GHz  247 GB - 5.14 GB / core    /sc1: 66TB, /sc2: 77TB  

- Using ScaleMP software across four 2-socket nodes.

- L2: 256KB/core, L3: 12MB, newest supported extension: SSE4.2.

- Not accessible from outside. Only via astro06!

 astro06.hpc.ku.dk  32 cores: 4x Xeon E5-4650 @ 2.70GHz 768 GB - 24 GB / core
 
 
Quadro K2000  /scratch: 2.7TB RAID0 SSD - GPU for remote HW rendering through VirtualGL.

 astro07.hpc.ku.dk

 astro08.hpc.ku.dk

 astro09.hpc.ku.dk

32 cores: 4x Xeon E5-4650 @ 2.70GHz    768 GB - 24 GB / core Quadro K2000     - GPU for remote HW rendering through VirtualGL.

Notes

  • astro04 is a virtual NUMA (non-uniform memory access) server composed of 4 nodes each having 2x6 cores, 72 GB memory, 24 disk slots, 2 raid controllers, and a dual QDR infiniband connection. The four nodes are uplinked to a dedicated QDR switch and are running the virtual machine software ScaleMP, which effectively present the four nodes as one machine. ScaleMP uses some of the system memory for caching purposes, decreasing the available memory from 288 GB to 247 GB. 
  • The latency between CPU cores on different nodes is very different relative to core on the same node (0-11, 12-23, 24-35, 36-47). It is therefore an advantage to launch programs with, for example, taskset:

% taskset -c 0-11 idl

will launch an IDL session restricted to running on core on the first node.

  • The main advantage of astro04 is that it has fast, low latency, local storage /sc1 and /sc2, and a 10gbit connection to the storage server.

Storage

  • The home directory (/users/astro) is a GPFS system. We have a shared 16TB quota.
  • Archive (/users/astro/archive) is a symbolic link to the two ZFS filesystems exported from astro05 as NFS volumes. Each archive system can be found under /users/astro/archive0 and /users/astro/archive1, and each user will either have a directory under archive0 or archive1, but should always refer to it as /users/astro/archive/<username>.
  • archive0 has 215TB spread over 90 3TB disks. It uses RAIDZ2 for redundancy in stripes of 16+2 disks.
  • archive1 has 172TB spread over 37 6TB disks. It uses RAIDZ2 for redundancy in stripes of 16+2 disks with 1 hot spare.
  • The storage server, astro05, is serving both the two archive filesystems, and a similar sized filesystem for DARK. It has an intel Xeon E5-1650, 6-core CPU, 3.2GHz, 128GB of memory for efficient caching, and a 10 gbit/s uplink to the core switch.

Network

  • External connection: The local HPC center is a Tier-1 CERN node and has a dual 10 gbit/s connection. In practice we have reach 100 MB/s ot-of-house by tranferring several large files in parallel
  • Internal connections:
    • astro04 and astro05 are connected with a 10 gbit/s uplink.
    • astro06 - astro09 are connected with a dual bonded 1 gbit/s.
    • astro01 and all cluster nodes have a 1 gbit/s uplink.