NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
04/18/19 14:00 PDT04/18/19 14:13 PDTJupyterScheduled maintenance.
04/17/19 11:30 PDT04/17/19 13:44 PDTHPSS Archive (User)Unavailable.
04/17/19 7:00 PDT04/17/19 16:41 PDTEdisonScheduled maintenance.
04/12/19 8:00 PDT04/12/19 10:30 PDTCoriUnavailable. Slurm job scheduler debugging in underway.
04/10/19 12:49 PDT04/10/19 13:22 PDTGlobal HomesUnavailable. Engineers are working on the issue. Half of users are affected.
04/10/19 12:43 PDT04/10/19 14:40 PDTGlobusUnavailable. Users are not able to activate NERSC endpoints (NERSC DTN, NERSC Cori, NERSC Edison, and NERSC HPSS). Existing transfers will not be effected. Engineers are investigating the problem.
04/10/19 10:00 PDT04/10/19 14:39 PDTexvivoUnavailable. scheduling on exvivo and cgpu has been paused. resources were concurrently impacted with Edison
04/10/19 10:00 PDT04/10/19 10:30 PDTScience DatabasesScheduled maintenance. Services on nerscdb03 and nerscdb04 will be down briefly (5-15 minutes) within the maintenance window for upgrades to system software.
04/10/19 10:00 PDT04/10/19 14:39 PDTcgpuUnavailable. scheduling on exvivo and cgpu has been paused. resources were concurrently impacted with Edison
04/10/19 9:30 PDT04/10/19 15:20 PDTNX ServicesScheduled maintenance. Transitioning to a new hardware. All open sessions will be terminated.
04/10/19 9:00 PDT04/10/19 9:29 PDTMyProxy, NIMScheduled maintenance. Scheduled maintenance. Services will be down briefly (5-15 minutes) within the window for upgrades to system software.
04/10/19 9:00 PDT04/10/19 15:25 PDTSpinScheduled maintenance. Services will be down briefly (1-2 min) within the window for upgrades to Docker and system software.
04/10/19 8:15 PDT04/10/19 14:19 PDTEdisonUnavailable. Edison currently not available - Batch system is down, Scratch file systems unavailable. Engineers are investigating the problem.
04/10/19 7:00 PDT04/10/19 20:45 PDTCoriScheduled maintenance.
04/09/19 9:00 PDT04/09/19 12:04 PDTNetworkScheduled maintenance. NERSC will update their core routers. Users will have intermittent connectivity during the maintenance.
04/06/19 7:45 PDT04/06/19 12:00 PDTCoriSystem in degraded mode. Transient batch system issues. Jobs still running. Job submissions may fail. Engineers investigating, next update at 14:00
04/05/19 8:48 PDT04/05/19 10:00 PDTCoriUnavailable. Troubleshooting filesystem issues.
04/03/19 11:06 PDT04/03/19 13:30 PDTHPSS Archive (User)System in degraded mode. Legacy Data at OSF will be unavailable.
04/02/19 20:30 PDT04/02/19 22:00 PDTCoriSystem in degraded mode. Cori was in a degraded mode due to a large number of nodes down and filesystem issues.
04/02/19 20:30 PDT04/02/19 21:50 PDTGlobal CommonUnavailable. Due to a system software issue, the NERSC global file systems /global/homes and /global/common were not available on several NERSC systems from 20:30 to 21:50
04/02/19 20:30 PDT04/02/19 21:50 PDTGlobal HomesUnavailable. Due to a system software issue, the NERSC global file systems /global/homes and /global/common were not available on several NERSC systems from 20:30 to 21:50
03/25/19 6:30 PDT03/25/19 9:09 PDTMFASystem in degraded mode. MFA One-Time-Password service unavailable: this may be affecting access to HPC resources. Engineers are working to resolve the problem. Please try to login as you normally would and contact us if there is an issue.
03/20/19 10:00 PDT03/20/19 10:26 PDTNX ServicesScheduled maintenance.
03/13/19 10:00 PDT03/13/19 15:17 PDTProjectBScheduled maintenance.
03/13/19 9:33 PDT03/13/19 12:20 PDTSpinScheduled maintenance. Services will be down briefly (1-2 min) within the window for a software upgrade.
03/13/19 9:10 PDT03/13/19 15:15 PDTJGI Web Servers and StorageScheduled maintenance. Services will be unavailable for GPFS client software upgrades and Webfs storage firmware and software upgrades.
03/13/19 9:00 PDT03/13/19 11:40 PDTHPSS Archive (User)Scheduled maintenance.
03/13/19 7:00 PDT03/13/19 20:50 PDTCoriScheduled maintenance.
03/07/19 13:42 PST03/07/19 14:03 PSTProjectUnavailable. Engineers are investigating.
03/06/19 12:00 PST03/06/19 14:25 PSTCoriUnavailable.
03/06/19 12:00 PST03/06/19 13:28 PSTEdisonUnavailable.
03/06/19 9:00 PST03/06/19 13:00 PSTHPSS Archive (User)Scheduled maintenance.
03/06/19 9:00 PST03/06/19 10:40 PSTHPSS Regent (Backup)Scheduled maintenance.
03/05/19 9:00 PST03/05/19 14:12 PSTSeqFSScheduled maintenance. The scope of the maintenance requires a complete outage of seqFS.
03/02/19 23:10 PST03/03/19 1:00 PSTProjectBSystem in degraded mode.
03/02/19 14:45 PST03/02/19 16:05 PSTProjectBSystem in degraded mode.
02/27/19 7:00 PST02/27/19 17:30 PSTEdisonScheduled maintenance.
02/27/19 4:52 PST02/27/19 6:00 PSTHPSS Archive (User)Unavailable. Archive is unavailable due to an unscheduled maintenance.
02/20/19 9:00 PST02/20/19 13:09 PSTHPSS Archive (User)Scheduled maintenance.
02/12/19 10:37 PST02/12/19 11:06 PSTMyProxyScheduled maintenance. Service will be down briefly (10-15 min) within the window for kernel and system software updates and reboots.
02/12/19 10:37 PST02/12/19 12:07 PSTScience DatabasesScheduled maintenance. Services will be down briefly (10-15 min) within the window for kernel and system software updates and reboots.
02/12/19 9:00 PST02/12/19 9:10 PSTMyNERSC, NIMScheduled maintenance. NIM and MyNERSC services will be down briefly (10-15 min) within the window for kernel and system software updates and reboots.
02/12/19 8:30 PST02/12/19 12:25 PSTSpinScheduled maintenance. Services will be down briefly (1-2 min) within the window for urgent kernel and security updates and system reboots.
02/12/19 8:30 PST02/12/19 12:35 PSTJGI Web and Database serversScheduled maintenance. Servers will be down briefly (10-15 min) within the window for kernel and system software updates and reboots. The webfs storage system will be unavailable from approximately 9:30-11:30 PST for a firmware and software upgrade.
02/09/19 18:42 PST02/10/19 0:46 PSTCoriSystem in degraded mode. Access to all network filesystems may be slower than usual. Engineers are on site, we hope to have the issue resolved by 03:00 PST.
02/07/19 16:00 PST02/07/19 20:00 PSTMulti-Factor AuthenticationScheduled maintenance. Scheduled maintenance. Replacement of hardware components. No outage is expected.
02/07/19 8:00 PST02/07/19 17:00 PSTHPSS Archive (User)Scheduled maintenance.
02/06/19 7:00 PST02/06/19 19:15 PSTCoriScheduled maintenance. Maintenance Completed at 19:15 PST.
02/06/19 7:00 PST02/06/19 11:10 PSTData Transfer NodesScheduled maintenance.
02/05/19 6:00 PST02/05/19 7:05 PSTNetworkSystem in degraded mode. Users may notice intermittent sluggish performance accessing NERSC assets. Engineers are aware and investigating a DNS issue.
02/02/19 16:05 PST02/03/19 23:20 PSTCoriSystem in degraded mode.
02/01/19 12:51 PST02/01/19 13:37 PSTCoriUnavailable.
01/28/19 7:04 PST-HPSS Archive (User)Users may see delays or transient I/O errors while the HPSS archive transitions to CRT. If you get an I/O error, please retry your retrieval after several (5 - 6) hours. This work should be completed by 02/01.
01/26/19 11:30 PST01/26/19 13:00 PSTMulti-Factor AuthenticationScheduled maintenance. Scheduled maintenance. Configuration changes related to system redundancy will be made. No outage is expected.
01/25/19 7:21 PST01/25/19 9:57 PSTHPSS Archive (User)Unavailable. Engineers are actively investigating.
01/25/19 7:04 PST-HPSS Archive (User)Users may see delays or transient I/O errors while the HPSS archive transitions to CRT. If you get an I/O error, please retry your retrieval after several (5 - 6) hours. This work should be completed by 02/01.
01/24/19 13:10 PST01/24/19 14:14 PSTCoriUnavailable. Cori is currently unavailable, our engineers are troubleshooting the issue.
01/24/19 6:56 PST-HPSS Archive (User)Users may see delays or transient I/O errors while the HPSS archive transitions to CRT. If you get an I/O error, please retry your retrieval after several (5 - 6) hours. This work should be completed by 02/01.
01/22/19 7:22 PST-HPSS Archive (User)Users may see delays or transient I/O errors while the HPSS archive transitions to CRT. If you get an I/O error, please retry your retrieval after several (5 - 6) hours. This work should be completed by 02/01.
01/21/19 10:39 PST01/21/19 15:05 PSTHPSS Archive (User)Unavailable. The system is unavailable due to site power issue
01/21/19 10:39 PST01/21/19 14:09 PSTHPSS Regent (Backup)Unavailable. The system is unavailable due to site power issue
01/21/19 9:35 PST01/21/19 18:13 PSTCoriUnavailable. The system is unavailable due to a center wide power outage. Power supply is stable now, NERSC engineers are working to bring the system back up.
01/21/19 9:35 PST01/21/19 17:59 PSTEdisonUnavailable. The system is unavailable due to a center wide power outage. Power supply is stable now, NERSC engineers are working to bring the system back up.
01/21/19 9:35 PST01/21/19 19:11 PSTGenepoolUnavailable. The system is unavailable due to a center wide power outage. Power supply is stable now, NERSC engineers are working to bring the system back up.

Show Older Outages: