Data & Networking
We recognize the importance of being able to manage your files and data. Scientific datasets are growing very rapidly and there is an increasing need for large scale data storage and high performance data transfers. As science becomes more distributed organizing and sharing data becomes very important.
This section covers several topics on how to successfully manage your data at NERSC.
Connecting to NERSC
All NERSC computers (except HPSS) are reached by use of programs that implement the Secure Shell (SSH) communication and encryption protocol, version 2, or by Grid tools that use trusted certificates. Read More »
NERSC File Systems
NERSC has a number of different file systems where users can store data and run applications. Some of the file systems are local to a particular system while others are "global" and can be accessed by multiple systems at NERSC. Read More »
Transferring Data
Learn about the best way and most efficient tools to move data in, out and around the center. Read More »
HPSS Data Archive
HPSS, the High Performance Storage System, is NERSC data archive and backup system. NERSC has two HPSS systems, one for users (archive.nersc.gov) and one for system backups. You can access HPSS from any system that supports HSI, HTAR or ftp/pftp, or by using gridFTP clients. Read More »
Optimizing I/O performance on the Lustre file system
The Lustre file system is mounted on both the Franklin and Hopper systems. This page describes how get the best performance out of the Lustre file system. Read More »
I/O Formats
I/O continues to be one of the main bottlenecks for scientific applications. This page describes some strategies for writing and reading data efficiently. Read More »
Sharing Data
Inside NERSC - Sharing Your Own Data Between NERSC Systems Use Global Home for source code and other smaller files. For large data, use Global Scratch. Accessing the Data on Local-scratch of Hopper and Franklin on other NERSC Systems Local-scratch on Hopper or Franklin is not accessible directly by other NERSC Systems (E.g. Euclid), so you need to move the data to Global Home or Global Scratch. Inside NERSC - Sharing Data with Other NERSC Users Use the Project Directory to share data with other… Read More »
Network Performance
In this ESnet Network Performance Knowledge Base you will find information on how to make your wide area data transfers go faster. The goal of this site is to help you maximize your wide-area network bulk data transfer performance by tuning the TCP settings for your end hosts and by using file transfer tools that are designed to maximize network throughput. Read More »
