GPGPU Research Projects

Overview

GPGPU (General Purpose computing on Graphics Processing Units) has become increasingly popular in the HPC community in recent years, where the GPU is now a viable component of new-generation compute platforms in addition to its traditional role in visualisation work. While it is not the only path towards widespread availability of peta-scale computing, it is currently the most promising one.

With the above in mind, ICHEC is pursuing the following goals:

• To port codes that are of major interests to the Irish scientific community to take advantage of GPGPUs.
• To build a strong pool of expertise in the field that empowers ICHEC to advise and support users who wish to evaluate GPU computing themselves.
• To build partnerships - with manufacturers, developers and users communities - that enable access to new hardware, codes and tools for the Irish community.

Currently, the technology and software ecosystems are still maturing and some work is needed before GPGPU is ready for widespread, general-purpose production computation. However, the latest version of NVIDIA's GPU architecture, code-named Fermi, looks extremely promising. It has the potential to change the HPC landscape significantly by providing features that are inherent in true HPC hardware, e.g. proper IEEE 754 norm support and ECC memory correction. There is already anticipation that some significant performance gains can be achieved on hybrid platforms using FERMI, due to its new features, compared to former architectures.

ICHEC has therefore invested considerable effort to ensure that emerging technologies in GPGPUs can be effectively exploited when they become widely available (likely to happen over the next 12 months). The expertise we have gained on both older and current hardware platforms will be beneficial in this regard, as well as for projects which will require access to large installations with GPUs, e.g. those in PRACE.

ICHEC is recognised as official CUDA Research Center In recognition of the different GPGPU research activities conducted by ICHEC over the past year, we are proud to have been designated by NVIDIA as an official CUDA Research Center. The CUDA Research Center program is designed to identify institutions that are doing "world-changing research by leveraging CUDA and NVIDIA GPUs". The main benefits for us are direct access to hardware, software and expertise from NVIDIA, which will be invaluable for our future research in the field.

Projects

A Java Library for the Generation and Scheduling of PTX Assembly

ICHEC computational scientists Christos Kartsaklis and Gilles Civario have developed a tool called JASM, introduced at the prestigious NVIDIA GPU Technology Conference 2009 where Christos presented a talk which discussed ongoing progress regarding the development of a Java-based library for rapid kernel prototyping in NVIDIA PTX and PTX instruction scheduling. It is aimed at developers seeking total control of emitted PTX, highly parametric emission of, and tuneable instruction reordering. It is primarily used for code development at ICHEC but is also expected that the NVIDIA GPU community will also find it beneficial.

Live recordings of Christos' presentation at the conference can be downloaded from NVIDIA's web site as a FLV file or a MP4 file. Also available are the PowerPoint presentation used and the full paper that was submitted to the conference. The JASM tool is available for downloading here.

GPGPU Optimisation of the Weather Research and Forecasting (WRF) model

As part of a collaboration with the Irish Climate Analysis and Research Units (ICARUS) at NUI Maynooth, Nicola McDonnell is involved in the implementation of the physics kernel of WRF on the Stoney system where a number of NVIDIA Tesla cards have been installed. Previous work by Michalakes et al. at NCAR (National Center for Atmospheric Research) has utilised NVIDIA systems to achieve a 100-fold speed-up in the physics kernel of WRF, and a 30% speed-up of the overall model.

Quantum Espresso GPU porting

ICHEC is collaborating with the CNR DEMOCRITOS Group in a feasibility study of porting Quantum Espresso to GPGPU architectures. Two parts of this integrated suite of codes - for electronic-structure calculations and materials modeling at the nanoscale level - are being examined in parallel by two different groups.

ICHEC computational scientist Ivan Girotto is working on the PWscf part of the code which performs electronic and ionic structure calculations; a group from Washington University is looking at another part of the code which deals with basic Car-Parrinello simulations. In addition, two development approaches are also being evaluated in parallel: HMPP and CUDA.

DL_POLY CUDA porting

DL_POLY is a well known molecular dynamics simulation code developed by STFC Daresbury Laboratory in the UK. As part of a collaborative research programme with Daresbury Labs, ICHEC has been involved in porting DL_POLY version 3.10 to CUDA. Much of the work had been carried out by Christos Kartsaklis in close collaboration with Dr. Ilian Todorov and Prof. Bill Smith from Daresbury.

The code has been successfully ported and it runs efficiently with a speed-up of around 4x so far. Furthermore, the code is now parallelised with a mix of MPI, OpenMP and CUDA, allowing an efficient usage of both the CPUs and the GPUs of a HPC cluster.

This work has been described in an accepted paper entitled "DL_POLY 3: Hybrid CUDA/OpenMP porting of the non-bonded force-field for two-body systems" and was presented by Christos Kartsaklis at the 240th American Chemical Society National Meeting in Boston (22nd-26th August, 2010).

ICHEC computational scientist Gilles Civario presented a summary of the work on this port at the prestigious NVIDIA GPU Technology Conference 2010. This presentation can be downloaded from NVIDIA's web site as a FLV file or a MP4 file.

The work has also been presented by Ruairi Nestor at the GPUs and Accelerators in HPC Workshop, held at Daresbury Laboratory in September 2010.

The CUDA-enabled port of DL_POLY has been released as part of the official distribution of DL_POLY_4, which is available for download from Daresbury Laboratory.

Summer Scholarships projects

In 2010, the ICHEC Summer Scholarships were awarded to two students to carry out two GPGPU-related projects:

• Implementation of the multiple polynomial sieve factoring algorithm on the GPU architecture
• Development of a General Matrix Multiply (GEMM) library for hybrid CPU-GPU architectures

These projects were defined and implemented in close collaboration with the end users and/or the developers of the codes: Prof. Mike Scott in the first case and Dr. Massimiliano Fatica from NVIDIA in the second. Under the supervision of ICHEC computational scientists, along with interaction with end users/developers, we expect the two projects to be successful in producing formal publications and we will release the final code to the community at the end of the projects.

Other activities

As part of his GPGPU activity, Gilles Civario has delivered a number of presentations that may be of interest:

• GPGPU: hype or revolution?: presentation made during the last IAHPC conference in UCD.
• Code acceleration with HMPP: brief presentation of HMPP, a new programming technique for hardware accelerators.