NERSCPowering Scientific Discovery Since 1974

HDF5

Description and Overview

Hierarchical Data Format version 5 (HDF5) is a set of file formats, libraries, and tools for storing and managing large scientific datasets. Originally developed at the National Center for Supercomputing Applications, it is currently supported by the non-profit HDF Group.

HDF5 is different product from previous versions of software named HDF, representing a complete redesign of the format and library.  It also includes improved support for parallel I/O. The HDF5 file format is not compatible with HDF 4.x versions. You can use the 'h5toh4' and 'h4toh5' converters that are available on all NERSC machines. 

If you are not familiar with parallel I/O, please refer to the tutorial on scientific I/O.

Using HDF5 On Cray Systems

There are HDF5 libraries provided by Cray . Use the command "module avail cray-hdf5"  to see the available Cray versions. 

Cray serial HDF5 on Edison and Cori

% module load cray-hdf5
% ftn ... (for Fortran code)
% cc ... (for C code)
% CC ... (for C++ code)

Cray parallel HDF5 on Edison and Cori

% module load cray-hdf5-parallel 
% cc ... (for C code)
% ftn ... (for Fortran code)

 

 For questions about HDF5 on any NERSC systems, please send email to consult@nersc.gov. Additional information is available at The HDF Group.

Availability at NERSC

  • Cray's versions:
    • Edison: 1.10.1.1 (default), 1.10.0.3
    • Cori: 1.10.1.1 (default), 1.10.0.3
  • NERSC's versions:
    • Edison: 1.10.1 (default), 1.9.234
    • Cori: 1.10.1(default),  1.9.234, rados

Have you encountered any problem in using HDF5? Check our Q&A

h5wordle

Q&A

  1. HDF5 1.8.16 read HDF5 1.10 file
  2. HDF5 1.10 code read HDF5 1.8.16 file
  3. HDF5 file locking
  4. HDF5 1.10 with Darshan
  5. Cray netCDF with HDF5
  6. HDF5 1.10.0.3 and HDF5 1.10.1.1
  7. Where are those old versions
  • Can existing HDF5 1.8.16 code read the file generated by HDF5 1.10?

No. Unless when you use HDF5 1.10 to create the files, you set the file format version to be compatible and avoid using latest 1.10 features, e.g., SWMR. How to do that? 

With H5py: 

f = h5py.File(‘name.hdf5’, libver=‘earliest’)’  # create the file with most compatible version, performance benefit from HDF5 1.10 loses         

With HDF5: 

fapl = H5Pcreate (H5P_FILE_ACCESS);
status = H5Pset_libver_bounds (fapl, H5F_LIBVER_EARLIEST, H5F_LIBVER_EARLIEST); // setting the lower and upper bound to be earliest. 
  • Can HDF5 1.10 code read the existing files generated by HDF5 1.8.16?

Yes

  • HDF5 1.10.0 file locking issue (Fixed with 1.10.1) but need to set one env variable)

HDF5 1.10.x which has the SWMR feature, and enables flock to allow multiple applications opening the same file. The flock, however, is not enabled by certain filesystem at NERSC, e.g., /project, burst buffer, so the fix is to disable the HDF5's flock by setting the environmental variable HDF5_USE_FILE_LOCKING to be FALSE. The flock is enabled on Lustre file system, i.e., SCRATCH, so you won't see any flock error there. Note that Cray's default version (1.10.1.1) is ok, but you can also compile your code with NERSC-built 1.10.1. 

export HDF5_USE_FILE_LOCKING=FALSE         
  • HDF5 1.10 conflicts with Darshan, Resolved with Darshan 3.1.4 (Users don't need to do anything now)

When using the latest version of HDF5 on Cori/Edison, Darshan should be unloaded. The HDF5 1.10.x has changed the size of hid_t from 32 bit integer to 64 bit integer, while Darshan's HDF5 wrapper still uses 32 bit.

Darshan 3.1.4 has been installed, and has disabled the HDF5 wrappers, so Darshan will not crash applications, will not track HDF5 1.10 functions calls either. IO calls at MPI layer and POSIX layer are still being tracked by Darshan, which is useful.  

  • Default cray-netcdf-hdf5parallel is not compatible with default HDF5

Default cray-netcdf-hdf5parallel is built with hdf5 1.10.0.3, but the default hdf5 has been upgraded to 1.10.1.1 to fix a bug. You can either explicitly load 1.10.0.3 to work with cray-netcdf-hdf5parallel or you can try disabling version check: 

export HDF5_DISABLE_VERSION_CHECK=1         
  • Are there any difference in file format between 1.10.0.3 and 1.10.1.1?

No

  • Can I load old versions that I used to work with?

Yes, (but not recommended)

module load pe_archive
module load cray-hdf5-parallel/1.8.16 (example)

 

New Features in HDF5 1.10

  • Concurrent Access to an HDF5 File: Single-Writer / Multiple-Reader (SWMR)

  • Virtual Dataset (VDS)

  • Scalable Chunk Indexing

  • Persistent Free File Space Tracking

  • Collective Metadata I/O Feature for Improving Parallel HDF5 Performance

    more information at HDF5 website

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
HDF datatran2 libraries/ I/O 1.8.13 hdf5/1.8.13 2015-09-30
HDF pdsf_sl6 libraries/ I/O 1.8.13 hdf5/1.8.13 2014-08-14 2014-08-14
hdf5 cori applications/ I/O 1.10.1 hdf5/1.10.1 2018-05-31 2018-05-31
hdf5 cori applications/ I/O 1.10.2 hdf5/1.10.2 2018-07-09
hdf5 genepool pe_libraries/ general 1.8.10-patch1 hdf5/1.8.10-patch1 2013-03-28 2013-04-08
hdf5 genepool pe_libraries/ general 1.8.11 hdf5/1.8.11 2013-06-11
hdf5 genepool pe_libraries/ general 1.8.12 hdf5/1.8.12 2014-01-14
hdf5 genepool pe_libraries/ general 1.8.13 hdf5/1.8.13 2014-09-24
hdf5 genepool pe_libraries/ general 1.8.15-patch1 hdf5/1.8.15-patch1 2015-11-09
hdf5 genepool pe_libraries/ general 1.8.4-patch1 hdf5/1.8.4-patch1 2013-04-17
HDF5 genepool applications/ bioinformatics 1.8.7 hdf5/1.8.7 2012-04-04 2012-04-04
hdf5 genepool pe_libraries/ general 1.8.9 hdf5/1.8.9 2012-07-19 2012-07-19
hdf5 genepool libraries/ general 4.10.2 rpm/4.10.2 2013-01-24 2013-01-24
hdf5 genepool_sl6 pe_libraries/ general 1.8.10-patch1 hdf5/1.8.10-patch1 2014-12-12 2014-12-12
hdf5 genepool_sl6 pe_libraries/ general 1.8.12 hdf5/1.8.12 2014-12-12
hdf5 genepool_sl6 pe_libraries/ general 1.8.13 hdf5/1.8.13 2014-12-12
hdf5 genepool_sl72 pe_libraries/ general 1.8.15-patch1 hdf5/1.8.15-patch1 2017-01-27
hdf5 phoebe pe_libraries/ general 1.8.10-patch1 hdf5/1.8.10-patch1 2013-03-29 2013-06-26
hdf5 phoebe pe_libraries/ general 1.8.11 hdf5/1.8.11 2013-06-18
hdf5 phoebe pe_libraries/ general 1.8.4-patch1 hdf5/1.8.4-patch1 2013-05-15
hdf5 phoebe pe_libraries/ general 1.8.9 hdf5/1.8.9 2012-07-26 2012-07-26
hdf5 phoebe libraries/ general 4.10.2 rpm/4.10.2 2013-05-15 2013-06-26