NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
07/11/18 23:45 PDT07/13/18 16:20 PDTCoriSystem in degraded mode. The system is currently degraded while Cray engineers complete hardware work.
07/11/18 7:00 PDT07/11/18 23:45 PDTCoriScheduled maintenance. The maintenance has been further extended due to ongoing multiple blower issues
07/11/18 7:00 PDT07/11/18 19:00 PDTEdisonSystem in degraded mode. cscratch1 will not be available on Edison due to Cori schedule maintenance.
07/11/18 7:00 PDT07/11/18 19:00 PDTData Transfer NodesSystem in degraded mode. cscratch1 will not be available on DTN systems due to Cori schedule maintenance.
07/08/18 17:20 PDT07/09/18 5:15 PDTScience Database ServicesSystem in degraded mode. Service was slow or unresponsive for several periods ranging from 10 minutes to 2 hours during this time window due to a failing backup process.
07/07/18 11:30 PDT07/07/18 17:40 PDTCoriSystem in degraded mode. Scratch filesystem issues have been repaired. Engineers have returned full system functionality to Cori as of 17:40 PDT.
07/07/18 11:30 PDT07/07/18 17:40 PDTEdisonSystem in degraded mode. Scratch filesystem issues have been repaired. Engineers have returned full system functionality to Edison as of 17:40 PDT.
07/06/18 19:12 PDT07/06/18 20:00 PDTEdisonUnavailable. Edison was temporarily unavailable due to previous incident. System functionality has now been restored.
07/06/18 11:25 PDT07/06/18 15:22 PDTEdisonLogins available, batch jobs not running. Engineers are actively working to repair a hardware issue. Logins and job submission is currently available, however new jobs are not running.
07/05/18 16:15 PDT07/05/18 19:30 PDTMongoDB03Unavailable.
06/27/18 7:10 PDT06/27/18 17:52 PDTEdisonScheduled maintenance.
06/26/18 8:59 PDT07/05/18 8:59 PDTData Transfer NodesScheduled maintenance. Dtn03 is down for maintenance
06/23/18 3:58 PDT06/23/18 6:56 PDTCoriSystem in degraded mode. The system is degraded, engineers are investigating.
06/22/18 0:44 PDT06/22/18 2:00 PDTEdisonSystem in degraded mode. The system is degraded, engineers are investigating. Logins and job submission are available. Jobs requesting /cscratch1 will not run.
06/20/18 11:00 PDT06/20/18 12:00 PDTNIMScheduled maintenance.
06/17/18 10:05 PDT06/18/18 0:05 PDTHPSS UserUnavailable. HPSS system 'Archive'/'User' is currently unavailable. The system has been rebooted, after a necessary configuration modification. Pending completion of startup tasks after the reboot process, the system is expected to be back online by 06-18 at 00:30.
06/13/18 7:00 PDT06/15/18 10:15 PDTCoriScheduled maintenance. Cori scheduled maintenance has been extended due to hardware issues.
06/09/18 13:35 PDT06/09/18 17:17 PDTCoriSystem in degraded mode.
06/09/18 9:24 PDT06/09/18 12:05 PDTCoriSystem in degraded mode.
06/08/18 19:12 PDT06/08/18 20:48 PDTCoriSystem in degraded mode. Cori is currently degraded. Engineers are actively working to resolve this issue.
06/04/18 13:10 PDT06/04/18 15:37 PDTCoriSystem in degraded mode. Cori is currently degraded due to problems with the cscratch1 filesystem. Engineers are actively working to resolve this issue. Logins are available, and job submission is available, however new jobs are not running.
06/04/18 13:10 PDT06/04/18 15:55 PDTEdisonSystem in degraded mode. Edison is currently degraded due to problems with the cscratch1 filesystem. Engineers are actively working to resolve this issue. Logins are available, and job submission is available, however new jobs are not running.
05/31/18 10:50 PDT05/31/18 11:11 PDTCoriSystem in degraded mode. cscratch1 is currently unavailable.
05/30/18 7:00 PDT05/30/18 22:38 PDTEdisonScheduled maintenance.
05/12/18 11:50 PDT05/12/18 14:15 PDTCoriSystem in degraded mode. Engineers are continuing to address system issues .
05/12/18 3:19 PDT05/12/18 9:50 PDTCoriSystem in degraded mode. Nersc engineers and Cray personnel are troubleshooting network issues on Cori. System is in a degraded state, with no new jobs starting. Work is still in progress, but updates will be made available as new information arises.
05/11/18 15:54 PDT05/11/18 17:17 PDTCoriSystem in degraded mode.
05/11/18 3:34 PDT05/11/18 5:55 PDTCoriSystem in degraded mode. Batch scheduler unavailable, no new jobs starting.
05/08/18 21:52 PDT05/08/18 22:53 PDTEdisonSystem in degraded mode.
05/08/18 18:00 PDT05/08/18 18:54 PDTHPSS BackupScheduled maintenance.
05/08/18 14:00 PDT05/08/18 14:18 PDTNetworkScheduled maintenance. DNS servers at OSF will be upgraded.
05/08/18 13:00 PDT05/08/18 19:25 PDTHPSS UserScheduled maintenance.
05/08/18 11:00 PDT05/08/18 12:00 PDTNX ServicesScheduled maintenance. NX Certificate update.
05/08/18 9:00 PDT05/08/18 15:28 PDTScience GatewaysScheduled maintenance.
05/08/18 9:00 PDT05/08/18 10:37 PDTJGI Web and Database servicesScheduled maintenance. Web and database servers (gpweb*, gnweb*, and gpdb*) will be down for system software updates.
05/08/18 9:00 PDT05/08/18 11:48 PDTProjectAScheduled maintenance.
05/08/18 9:00 PDT05/08/18 14:10 PDTMyProxy, NIM, SpinScheduled maintenance. Services will be down briefly (5-15 minutes) within the maintenance window for system software updates.
05/08/18 9:00 PDT05/08/18 10:25 PDTScienceDBsScheduled maintenance. MySQL and Postgres Services on nerscdb03 and nerscdb04 will be down briefly (5-15 minutes) within the maintenance window for system software updates
05/08/18 7:00 PDT05/09/18 13:12 PDTCoriScheduled maintenance.
05/08/18 7:00 PDT05/08/18 13:00 PDTHPSS UserSystem in degraded mode.
05/08/18 7:00 PDT05/08/18 18:00 PDTHPSS BackupSystem in degraded mode.
04/29/18 12:30 PDT04/29/18 13:30 PDTCoriSystem in degraded mode. Slurm scheduling was unavailable. Running jobs were not impacted.
04/26/18 21:30 PDT04/26/18 23:42 PDTCoriUnavailable. Scheduling is paused. NERSC engineers are working to resolve the matter. ETA for back to normal is 00:30.
04/26/18 15:01 PDT04/26/18 17:32 PDTCoriUnavailable.
04/25/18 9:00 PDT04/25/18 11:25 PDTHPSS BackupScheduled maintenance.
04/23/18 23:43 PDT04/24/18 5:44 PDTCoriUnavailable. Cori is currently down and engineers are currently investigating. An update will be provided at 03:00 PST. **03:00 PST UPDATE** NERSC engineers are still working to resolve this outage. An update will be provided again at 06:00 PST. **06:00 PST UPDATE** The system is now available as of 05:44 PST.
04/23/18 10:30 PDT04/23/18 11:31 PDTNIMScheduled maintenance.
04/23/18 8:00 PDT04/23/18 13:30 PDTProjectAScheduled maintenance.
04/22/18 11:08 PDT04/22/18 12:00 PDTHPSS UserSystem in degraded mode. New user connections may fail or hang. Engineers are aware of the problem and working on this issue.
04/11/18 9:00 PDT04/11/18 15:30 PDTHPSS UserScheduled maintenance.
04/11/18 7:00 PDT04/11/18 15:03 PDTEdisonScheduled maintenance. Update: please note updated end time. Edison will be unavailable due to software upgrades.
04/01/18 10:06 PDT-EdisonJobs using the /global/project filesystem may be submitted, but will not start at this time due to a hardware issue.
04/01/18 10:06 PDT-CoriJobs using the /global/project filesystem may be submitted, but will not start at this time due to a hardware issue.
04/01/18 8:59 PDT04/01/18 10:30 PDTProjectSystem in degraded mode. /project has a hardware issue and is in degraded mode. NERSC engineers are working to resolve the matter. Another update will be at Noon.
03/28/18 10:50 PDT03/28/18 16:12 PDTNIMScheduled maintenance. NIM will be unavailable due to db upgrades.
03/27/18 10:00 PDT03/27/18 13:00 PDTScience GatewaysScheduled maintenance. NEWT will be unavailable for 2-3 hours while being migrated to new hardware.
03/22/18 7:00 PDT03/22/18 14:00 PDTCoriScheduled maintenance. Cori will be unavailable while NERSC engineers perform system software maintenance.
03/21/18 15:11 PDT03/21/18 18:08 PDTGenepoolSystem in degraded mode. Denovo queues have been paused, due to ongoing problems with filesystem stability. NERSC system administrators continue to investigate.
03/21/18 9:00 PDT03/21/18 13:09 PDTHPSS UserScheduled maintenance. HPSS systems will be unavailable while NERSC engineers perform a scheduled maintenance.
03/21/18 9:00 PDT03/21/18 13:09 PDTHPSS BackupScheduled maintenance. HPSS systems will be unavailable while NERSC engineers perform a scheduled maintenance.
03/21/18 7:00 PDT03/21/18 12:10 PDTEdisonScheduled maintenance. Planned maintenance event for Edison system upgrade. Edison will be unavailable during this window of time.
03/17/18 2:30 PDT03/17/18 9:42 PDTCoriSystem in degraded mode. Cori is currently in a degraded mode, with issues pertaining to the cscratch1 file system. Engineers are working to resolve the matter. Updates will be added as work progresses.

Show Older Outages: