30Aug 2017

D.A.V.I.D.E. Supercomputer, Developed by E4 Computer Engineering


D.A.V.I.D.E. (Development for an Added Value Infrastructure Designed in Europe) is an Energy Aware Petaflops Class High Performance Cluster based on Power Architecture and coupled with NVIDIA Tesla Pascal GPUs with NVLink. The innovative design of D.A.V.I.D.E. has been developed by E4 Computer Engineering for PRACE*, which has as its ultimate goal to produce a leading edge HPC cluster showing higher performance, reduced power consumption and ease of use. 
D.A.V.I.D.E. is based on OpenPOWER platform and is among the harbingers of a new generation of HPC systems which deliver high performances while being environmentally conscious. It has been built using best-in-class components plus custom hardware and an innovative middleware system software.


D.A.V.I.D.E. represents the third generation of the European Prace Pre-Commercial Procurement (PCP) project to develop a Whole-System Design for Energy Efficient HPC. PRACE initiated the selection process in 2012 to obtain R&D services which should result in future PRACE HPC systems to become more energy efficient. The strict selection process saw several phases, in which three pre-selected European companies were asked to design HPC solutions that would not only be able to outstandingly perform on the most complex algorithms used by scientists and researchers, but would also drastically reduce power consumption. PRACE’s selection process is highly competitive and, during Phase II, the three companies produced early prototypes. All of them were invited to bid for Phase III, during which the pilot for the three promising solutions will be built and deployed at the hosting centres. E4 Computer Engineering stood out since the early steps of the selection and, thanks to the expertise acquired through years of collaboration with leading European research centers, has proposed D.A.V.I.D.E., a solution that was ranked first place for performance capabilities.


D.A.V.I.D.E. is composed by 45 nodes connected with an efficient Infiniband EDR 100 Gb/s networking, with a total peak performance of 990 TFlops and an estimated power consumption of less than 2kW per node. Each node is a 2 OU OCP form factor and hosts two IBM POWER8 Processors with NVIDIA NVLink and four Tesla P100 data center GPUs, with the intra-node communication layout optimized for best performance.

  • Derived from the IBM® POWER8 System S822LC (codename Minsky). 
  • 2 OU 21” Open Rack Enclosure with integrated piping & power distribution. 
  • Power8-based node in OCP form-factor, with leading edge features specifically engineered for HPC workloads. 
  • Two IBM POWER8 with NVlink and four NVIDIA Tesla P100 HSXM2. 
  • Differently from Minsky, DAVIDE uses direct liquid cooling for CPUs and GPUs. 
  • Each compute node has a peak performance of 22 TFLOPS and a power consumption of less than 2kW. 

  • Direct hot‐water cooling (about 27 °C) for the CPUs and GPUs. 
  • Extremely flexible and requiring minor modifications of the infrastructure. 
  • Each rack has an independent liquid‐liquid or liquid/air heat exchanger unit with redundant pumps. 
  • The system has internal pumps on the GPUs. Each Rack has its CDU. 
  • The compute nodes are connected to the heat exchanger through pipes and a side bar for water distribution. 

  • The system is coupled with four NVIDIA Tesla P100 HSMX2 per node with NVLINK interconnect, to deliver performance for the most demanding compute applications, providing: 
   5.3 TFLOPS of double precision floating point (FP64) performance
   10.6 TFLOPS of single precision (FP32) performance
   21.2 TFLOPS of half‐precision (FP16) performance 
  • A single link supports up to 40 GB/s of Bidirectional Bandwidth. The NVLink implementation in NVIDIA Tesla P100 supports up to four links, enabling ganged configurations with aggregate maximum bidirectional bandwidth of 160 GB/sec. 


A key feature of D.A.V.I.D.E. is an innovative technology for measuring, monitoring and capping the power consumption of the node and of the whole system, through the collection of data from the relevant components (processors, memory, GPUs, fans) to further improve energy efficiency. The technology has been developed in collaboration with the University of Bologna. 

  • Off-the-shelf components 
  • High speed and accurate per-node power sensing synchronized among the nodes 
  • Data accessible out-of-band and without processor intervention 
  • Out-of-Band and synchronized fine grain performance sensing 
  • Dedicated data-collection subsystem running on management nodes 
  • Predictive Power Aware job scheduler and power manager 


D.A.V.I.D.E. is designed to perform on the most complex algorithms used by scientists and researchers and to manage the most complex and compelling workloads with highly parallelized code and demanding memory bandwidth requirements, such as weather forecasting, QCD, machine learning, deep learning, computational fluid dynamics and genomic sequencing. 


The whole system was fully assembled in air-cooled configuration in April 2017 at E4’s facility in order to perform baseline performance, power and energy benchmarks, The system ranked #299 in TOP500 and #14 in GREEN500 in the June 2017 list. The system is currently (September 2017)  being converted to liquid cooling and will be delivered to CINECA in the Sept/Oct timeframe. 
The system is currently available to a select number of users for porting applications and profiling energy consumption at E4 premises. It will be then installed at CINECA (PRACE Italian hosting member), a non-profit consortium of 70 Italian universities, the National Institute of Oceanography and Experimental Geophysics (OGS), the National Research Council (CNR), and the Ministry of Education, Universities and Research (MIUR) and will be available to the scientific community to improve discovery and research. 



*What’s PRACE PCP and what is its goal?
Pre-Commercial Procurement (“PCP”) is a relatively new model of procurement that is gaining usage in many European Union Member States. PCP stands out as an effective tool to tackle discrepancies between how EU Member States and other countries benefit from their basic research expenditure. It has the following key elements:
• It is for R&D services only
• Risk-benefit sharing between the public purchaser and the R&D service providers, with sharing of Intellectual Property Rights;
• The PCP R&D work has to be performed at market prices.
PCP is a phased model that aims at conducting R&D up to the development of a limited volume of first products/services in the form of a test series. The target can typically be a solution to a major technical challenge. Discover more at http://www.prace-ri.eu/.   

549.8 Kb davide_paper_ieee.pdf

Follow us

E4 Computer Engineering SpA

  Via Martiri della Libertà, 66
42019 Scandiano (RE) – Italy
+39 0522 991811
+39 0522 991803