The Superfacility concept is a framework for integrating experimental and observational instruments with computational and data facilities. Data produced by light sources, microscopes, telescopes and other devices can stream in real-time to large computing facilities where it can be analyzed, archived, curated, combined with simulation data and served to the science user community via powerful computing, storage and networking systems. Connected with high-speed programmable networking, this superfacility model is more than the sum of its parts, allowing for discoveries across data sets, institutions and domains and making data from one-of-a-kind facilities and experiments broadly accessible.
The NERSC Superfacility project is designed to identify the technical and policy challenges in this concept for an HPC center. It coordinates and manages the work to address these challenges, in partnership with target science teams. It is designed to ensure that the solutions developed are widely useful (rather than one-off engagements) and that will scale to multiple user groups, and will be scalable for NERSC staff to support.
Services in Development
Data Management and Sharing
We are working to develop and deploy tools that can be used to handle the large volumes of data generated by superfacility partners.
- Globus is our tool of choice for large data transfers. We have several optimized data transfer nodes that can access every file system at NERSC.
- We are working to offer a new interface into HPSS that eliminates much of the difficulty of bundling and uploading files.
- A command line tool to do parallel transfers between file systems at NERSC (including HPSS) is under development.
- Batch system integration of data movement is being explored
- The NERSC Data Dashboard lets you see where your data is on the Project file system.
- A PI Dashboard is under development to allow PIs to address common issues (like permission drift) for the data they control
- Spin, a service platform for deploying science gateways, has been successfully deployed
- Globus Sharing has been enabled for data on the Project file system
Recent papers related to the Superfacility Model
- "Enabling a SuperFacility with Software Defined Networking", Richard Shane Canon, Tina Declerck, Brent Draney, Jason Lee, David Paul, David Skinner, CUG 2017
Recent articles about the Superfacility Model
- "Superfacility Framework Advances Photosynthesis Research" (2019)
- "The Superfacility Concept", feature episode of NERSC podcast (2019)
- "Superfacility – How new workflows in the DOE Office of Science are changing storage system requirements" (2016)
- "ESnet Paves Way for HPC "Superfacility" Real-Time Beamline Experiments" (2015)
Recent talks about the Superfacility model
- "The NERSC Superfacility Project: A Technical Overview", GPUs for Science (Cory Snavely)
- "Supercomputing and the Scientist: how HPC and analytics are transforming Experimental Science", Keynote at DataTech, Edinburgh, UK (Debbie Bard)
- "A Superfacility Model for Data-Intensive Science", Juliacon 2017 (Kathy Yelick)