UK HPC New Cluster

Hardware Information

* Information given here is replicated from here


The University of Kentucky has awarded a contract to Dell Inc for a new supercomputer cluster to replace our Lipscomb High Performance Computing Cluster, also known as the DLX. The University put out an RFP on March 29 and received five responses. After a thorough assessment by an evaluation committee of faculty and IT administrators, Dell's proposal was chosen as the best fit for our needs, within our budget, and the most flexible proposal overall.

The new cluster will be installed in McVey Hall when it arrives in July and must be fully operational before the end of August. Due to the limited power in McVey Hall, the Lipscomb cluster must be powered down before the full new cluster can be powered up. This will mean an intense effort by Dell technicians, our HPC team, Data Center Operations, and the Center for Computational Sciences to minimize the disruption for our researchers. As soon as possible, a Conversion Schedule will be posted.

The current(always tentative) migration schedule looks like this:
-  Week 5, Oct 08-12:   Phase I complete. UK sysadms installing SW and testing.Dell bringing up compute racks one at a time.
-   Week 6, Oct 15-19:   [10/15] Power down most of old cluster, move XDH, power up new cluster
      DDN starts copy of home / share
      UK sysadms continue installing SW and testing
      Dell begins validation testing
-   Week 7, Oct 15-19:   Sometime this week we let our users on.


The new facility will be an important step in helping UK researchers to keep pace in the design of new materials for magnetic applications, understanding of biological chemical processes, investigations of the origins of the universe and subatomic particles, development of new types of batteries, and improvements in automobile manufacture and design of improved pharmaceuticals.

The new cluster will feature:

  • 256 basic compute nodes, each with dual Intel E5-2670 2.6 GHz 8 core (2x8) processors and 64GB
  • 8 "fat" compute nodes, each with quad Intel E5-4640 2.4 GHz 8 core (4x8) processors and 512GB
  • 24 GPU enabled nodes, each with dual Intel E5-2670 2.6 GHz 8 core (2x8) processors, 64GB, and two NVIDIA M2075 GPUs
  • 800 TB of DDN storage for home and scratch space
  • FDR InfiniBand interconnect fabric

The new supercomputer will have a theoretical maximum of 140 teraflops. A teraflop is a trillion floating point operations (calculations) per second. We're hoping for 80% efficiency on a sustained test, which would have been good enough for #160 on the June 2012 Top 500 supercomputers list. The DLX runs at about 40 teraflops in a sustained test and debuted at #259 on the November 2012 list.
For technical help with HPC, please email your questions to help-hpc@uky.edu as usual.



UK HPC - Dell Cluster

Hardware Information

* Information given here is replicated from http://hpc.uky.edu/Hardware/


Hardware information for the 140 Teraflop Dell cluster (dlx.uky.edu).

Basic Nodes

  • 256 Nodes (4096 cores), ~95 Teraflops
  • Dell C6220 Server, 4 nodes per 2U chassis
  • Dual Intel E5-2670 8 Core (Sandy Bridge) @ 2.6 GHz
  • 2 sockets/node x 8 cores/socket = 16 cores/node
  • 64 GB/node of 1600 MHz RAM
  • 500 GB local (internal) SATA disk
  • Linux OS (RHEL)

Hi-Mem (Fat) Nodes

  • 8 Nodes (256 cores), ~4.9 Teraflops
  • Dell R820, one node per 2U
  • Quad Intel E5-4640 8 core (Sandy Bridge) @ 2.4 GHz
  • 4 sockets/node x 8 cores/socket = 32 cores/node
  • 512 GB/node of 1600 Mhz RAM
  • 4 x 1 TB local (internal) NLSAS disk
  • Linux OS (RHEL)

GPU Nodes

  • 24 Nodes (384 CPU cores, 48 GPUs), 33.6 Teraflops (8.9 CPU and 24.7 GPU)
  • Dell C6220 Server, 4 nodes per 2U chassis
  • Dual Intel E5-2670 8 Core (Sandy Bridge) @ 2.6 GHz
  • 2 sockets/node x 8 cores/socket = 16 CPU cores/node
  • 64 GB/node of 1600 MHz RAM
  • 500 GB local (internal) SATA disk
  • Dell C410x PCIe Expansion System, 8 cards (16 max) per 3U chassis
  • 8 NVIDIA M2075 GPU Cards, configured 2 per node
  • Linux OS (RHEL)

GPU 'Legacy' Nodes

  • Bought August 2011 for testing and experimentation.
  • 4 Nodes (48 CPU cores, 16 GPUs)
  • Dell C6100 Server, 4 nodes per 2U chassis
  • Dual Intel Xeon X5650 (Westmere) @ 2.66 GHz.
  • 2 sockets/node x 6 cores/socket = 12 CPU cores/node
  • 32 GB/node
  • 250 GB local (internal) SAS disk
  • Dell C410x PCIe Expansion System, 16 cards per 3U chassis
  • 16 NVIDIA M2070 GPU Cards, configured 4 per node
  • Linux OS (RHEL)

Login Nodes

  • 2 Nodes (32 cores)
  • Dell R720, one node per 2U
  • Dual Intel E5-2670 8 Core (Sandy Bridge) @ 2.6 GHz
  • 2 sockets/node x 8 cores/socket = 16 cores/node
  • 128 GB/node of 1600 Mhz RAM
  • 500 GB local (internal) SATA disk
  • Linux OS (RHEL)

Admin Nodes

  • 2 Nodes (32 cores)
  • Dell R720, one node per 2U
  • Dual Intel E5-2670 8 Core (Sandy Bridge) @ 2.6 GHz
  • 2 sockets/node x 8 cores/socket = 16 cores/node
  • 32 GB/node of 1600 Mhz RAM
  • 500 GB local (internal) SATA disk
  • Linux OS (RHEL)

Interconnect Fabric

  • Mellanox Fourteen Data Rate (FDR) Infiniband
  • 2:1 over-subscription, 14.0625 Gbit/s

Global cluster filesystem

  • DDN GridScaler SFA12K storage appliance with the IBM GPFS
  • 580 2TB 7,200 RPM 6Gb/s SAS drives (data)
  • 20 600GB 15K RPM 6Gb/s SAS drives (metadata)
  • 1160 TB raw with about 928 TB usable
  • Read: 25 GB/s throughput and 780,000 IO/S
  • Write: 22 GB/s throughput and 690,000 IO/S

Other Information

  • Fills most of 11 equipment racks.
  • Uses about 140 KW when loaded
  • Dedicated TSM node for fast backups and access to near-line storage

Software Information

Dell HPC Cluster related software information can be accessed at http://hpc.uky.edu/documentation