HDF5

Description and Overview

Hierarchical Data Format version 5 (HDF5) is a set of file formats, libraries, and tools for storing and managing large scientific datasets. Originally developed at the National Center for Supercomputing Applications, it is currently supported by the non-profit HDF Group.

HDF5 is different product from previous versions of software named HDF, representing a complete redesign of the format and library.  It also includes improved support for parallel I/O. The HDF5 file format is not compatible with HDF 4.x versions. You can use the 'h5toh4' and 'h4toh5' converters that are available on all NERSC machines. 

If you are not familiar with parallel I/O, please refer to the tutorial below. 

Scientific I/O Tutorial

Using HDF5 On Hopper

There are HDF5 libraries provided by Cray and by NERSC. Use the command "module avail hdf5" to see the available versions. The default HDF5 library for use with the default PGI compiler suite is provided by Cray.

Cray serial HDF5 with PGI Compiler

% module load hdf5 
% ftn  ...   (for Fortran code)
% cc  ...    (for C code)
% CC  ...    (for C++ code)

Cray serial HDF5 with GNU Compiler

     % module swap PrgEnv-pgi PrgEnv-gnu
     % module load hdf5 
     % ftn  ...  (for Fortran code)
     % cc  ...   (for C code)
     % CC  ...   (for C++ code)

NERSC serial HDF5 library with PGI Compiler

     % module load hdf5/1.8.7
     % h5cc  ...    (for C code)
     % h5fc  ...    (for Fortran code)
     % h5c++  ...   (for C++ code)

NERSC serial HDF5 with Intel Compiler

     % module swap PrgEnv-pgi PrgEnv-intel
     % module load hdf5/1.8.7 
     % h5cc  ...    (for C code)
     % h5fc  ...    (for Fortran code)
     % h5c++  ...   (for C++ code)

Cray parallel HDF5 with PGI Compiler

     % module load hdf5-parallel 
     % ftn  ...    (for Fortran code)
     % cc  ...    (for C code)

Cray parallel HDF5 with Cray Compiler

     % module swap PrgEnv-pgi PrgEnv-cray 
     % module load hdf5-parallel 
     % ftn  ...  (for Fortran code)
     % cc  ...   (for C code)

Using HDF5 On Euclid/Carver

Serial HDF5

    % module load hdf5 
    % h5fc  ...  
    % h5cc  ... 
    or  
    % mpif90  ...  $HDF5 
    % mpicc  ... $HDF5

Parallel HDF5

   % module load hdf5_parallel
   % h5pfc  ... 
   % h5pcc  ... 
   or
   % mpif90  ... $HDF5
   % mpicc  ... $HDF5