NERSCPowering Scientific Discovery Since 1974

Edison /scratch1 and /scratch2 file systems upgraded to Grid Raid during the file system maintenance on 8/29/2016-9/7/2016

September 7, 2016

The two Lustre file systems on Edison, /scratch1 and /scratch2, have been upgraded to Grid Raid, and as of noon on September 7, 2016, they are made available to users as clean file systems. While this upgrade improves I/O performance in general, a couple of changes should be relevant to all scratch users on Edison.

  1. Your $SCRATCH directory will be recreated. You will be assigned to one of the /scratch1 and /scratch2 file systems randomly in a Round-Robin fashion. You may see your $SCRATCH directory no longer points to the same file system as the one you used to have before the upgrade. If you used the absolute paths (such as /scratch1/scratchdirs/<Your username>)  in your job scripts in the past, please replace them with your current $SCRATCH. 
  2. The upgrade has combined the original 4 OSTs into a big OST with 4 times capacity, so that the number of OSTs in each file system has been reduced to 1/4 of the original numbers. So Edison scratch1 and scratch2 file systems now have 24 OSTs each. We have reduced the default stripe count to 1 (from 2) on /scratch1 and /scratch2 to reflect this change. You are encouraged to do your own tests to determine the optimal stripe count for your I/O workload. 
  3. While the performance with file per process is largely improved, you may see a lower performance with MPIIO due to the reduced number of OSTs. We expect the MPIIO (shared file I/O) will be improved when the Lustre lock ahead feature available in the future release of Lustre 1.8. Please contact consult at nersc.gov if you need MPIIO tuning advice before then.