The NERSC Burst Buffer is based on Cray DataWarp that uses flash or SSD (solid-state drive) technology to significantly increase the I/O performance on Cori for all file sizes and all access patterns.
The Burst Buffer is available for all to use - and is easy to use! This page is intended as a quick source of information for NERSC users interested in getting started with the Burst Buffer. Burst Buffer Basics What is a Burst Buffer? The Burst Buffer is a layer of SSD storage in Cori that sits within the high-speed network, between the on-node storage and the Lustre and GPFS file systems. It offers high-performance I/O on a per-job or short-term basis. Can I use the Burst Buffer? Yes!… Read More »
NERSC selected a number of HPC research projects to participate in the center’s new Burst Buffer Early User Program, where they were able to test and run their codes using the new Burst Buffer feature on the center’s newest supercomputer, Cori. The following are the applications selected to receive early access to the Burst Buffer, and in some cases active NERSC support in tailoring their code to take advantage of the Burst Buffer features. Please see this news article for more details on… Read More »
There are a number of known issues to be aware of when using the Burst Buffer on Cori. This page will be updated as problems are discovered, and as they are fixed. General Issues Do not use a decimal point when you specify the burst buffer capacity - slurm does not parse this correctly and will allocate you one grain of space instead of the full request. This is easy to work around - request 3500GB instead of 3.5TB, etc. Data is at risk in a Persistent Reservation if an SSD fails - there… Read More »
Hints and tips on how to optimize your Burst Buffer performance Note: this only applies to the Cori Burst Buffer and should not be taken as general Burst Buffer advice (i.e. your mileage will vary on other systems). This page will be updated as the DataWarp software is updated and performance continues to improve. For larger files, ensure your Burst Buffer allocation will be striped over multiple nodes Currently, the Burst Buffer granularity is 82GiB in the wlm_pool, and 20.14GiB in the… Read More »
The Burst Buffer Early User program has resulted in several publications, posters and talks. For reference, we list them here. "Accelerating Science with the NERSC Burst Buffer", W. Bhimji, Cray User Group (CUG) 2016 (Best Paper award) "Accelerating Science with the NERSC Burst Buffer", D. Bard, Salishan Conference on High-Speed Computing 2016 "Data Elevator: Low-contention Data Movement in Hierarchical Storage System", Bin Dong, Suren Byna, Kesheng Wu, Prabhat, Hans Johansen, Jeffrey N. Read More »