NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
12/12/18 10:00 PST12/12/18 12:50 PSTNetworkScheduled maintenance. Service interruptions are expected during an upcoming networking maintenance. Large data transfer speeds may be affected.
12/12/18 9:17 PST12/12/18 9:40 PSTNX ServicesUnavailable.
12/12/18 9:00 PST12/12/18 12:40 PSTHPSS Archive (User)Scheduled maintenance.
12/11/18 19:30 PST12/11/18 22:43 PSTCoriSystem in degraded mode. Batch scheduling software not responding to all events. Engineer is currently investigating.
12/06/18 18:00 PST12/06/18 18:30 PSTMulti-Factor AuthenticationScheduled maintenance. Network maintenance on the MFA service is complete. Logins for users opted in to MFA were unexpectedly impacted from approximately 18:05-18:30PT; the exact cause is under investigation.
12/06/18 8:00 PST12/06/18 8:55 PSTHPSS Archive (User)Unavailable.
12/05/18 7:00 PST12/05/18 19:52 PSTEdisonScheduled maintenance.
12/04/18 9:00 PST12/04/18 13:25 PSTMongoDBScheduled maintenance. mongodb01-04 will be upgraded from version 3.4 to version 3.6 and will be unavailable for some of this period
11/28/18 9:00 PST11/28/18 10:56 PSTHPSS Archive SystemScheduled maintenance.
11/27/18 9:00 PST11/27/18 17:00 PSTMongoDBScheduled maintenance. mongodb01-04 will have featureCompatibilityVersion set to 3.4. No outage expected but services "at-risk"
11/16/18 16:00 PST11/21/18 17:36 PSTHPSS UserUnavailable. NERSC HPSS Systems are being shut down due to poor air quality in the Bay Area due to the Camp Fire. Poor air quality can negatively impact the expected lifetime of data on tape.
11/16/18 16:00 PST11/21/18 17:36 PSTHPSS BackupUnavailable. NERSC HPSS Systems are being shut down due to poor air quality in the Bay Area due to the Camp Fire. Poor air quality can negatively impact the expected lifetime of data on tape.
11/13/18 8:00 PST11/13/18 9:00 PSTHPSS UserUnavailable.
11/07/18 15:00 PST11/07/18 16:00 PSTHPSS UserUnavailable. The Archive Storage system will be temporarily unavailable while engineers perform emergency maintenance.
11/07/18 9:00 PST11/07/18 13:00 PSTHPSS BackupScheduled maintenance.
11/02/18 8:53 PDT11/02/18 9:20 PDTCoriUnavailable. Logins were available, but no new jobs were starting.
10/31/18 11:00 PDT10/31/18 12:00 PDTNIMScheduled maintenance. NIM will be unavailable while NERSC engineers perform planned maintenance.
10/30/18 9:34 PDT-EdisonPlease note that interactive jobs will be disrupted between 10:00 and 12:00 today, October 30.
10/30/18 9:34 PDT-CoriPlease note that interactive jobs will be disrupted between 10:00 and 12:00 today, October 30.
10/27/18 13:00 PDT10/29/18 18:00 PDTEdisonSystem in degraded mode. Edison compute nodes are being rebooted to resolve a software issue in a rolling fashion. Login nodes and job submission are available.
10/27/18 11:00 PDT10/29/18 18:00 PDTCoriSystem in degraded mode. Cori compute nodes are being rebooted to resolve a software issue in a rolling fashion. Login nodes and job submission are available.
10/24/18 9:00 PDT10/24/18 12:18 PDTHPSS UserScheduled maintenance. Scheduled maintenance for the Archive (User) System
10/17/18 20:27 PDT10/18/18 8:24 PDTNX ServerUnavailable. NX / NoMachine is unavailable
10/17/18 10:00 PDT10/17/18 13:00 PDTScience DatabasesScheduled maintenance. Services on nerscdb03 and nerscdb04 will be down briefly (5-15 minutes) within the maintenance window for system software updates.
10/17/18 10:00 PDT10/17/18 11:00 PDTSpinScheduled maintenance. Science Gateways and services on Spin that use global file systems will be down while a storage subsystem is reconfigured.
10/17/18 10:00 PDT10/17/18 11:00 PDTScience GatewaysScheduled maintenance. Science Gateways and services on Spin that use global file systems will be down while a storage subsystem is reconfigured.
10/17/18 9:00 PDT10/17/18 10:45 PDTJGI Web and Database ServersScheduled maintenance. Servers will be down briefly (5-15 minutes) within the maintenance window for system software updates and reboots.
10/17/18 7:00 PDT10/17/18 23:55 PDTCoriScheduled maintenance. Cori monthly maintenance. The login nodes and Cori scratch filesystem (cscratch1) will remain available for part of the maintenance window, but no jobs will run. The batch system will be unavailable.
10/10/18 10:23 PDT10/11/18 10:41 PDTHPSS BackupUnavailable. System requires restart to clear mount queue.
10/10/18 2:00 PDT10/10/18 11:04 PDTHPSS UserSystem in degraded mode. HPSS systems are degraded, users may experience delays in retrieving data.
10/09/18 9:30 PDT10/09/18 12:40 PDTMyProxyScheduled maintenance. NERSC quarterly maintenance. Services will be down briefly (5-15 minutes) within the maintenance window for system software updates.
10/09/18 9:30 PDT10/09/18 9:35 PDTNIMScheduled maintenance. NERSC quarterly maintenance. Services will be down briefly (5-15 minutes) within the maintenance window for system software updates.
10/09/18 9:00 PDT10/09/18 16:15 PDTSpinScheduled maintenance. NERSC quarterly maintenance. Services may be briefly interrupted (1-2 minutes) within the maintenance window for container restarts.
10/09/18 8:00 PDT10/09/18 11:09 PDTData Transfer NodesScheduled maintenance.
10/09/18 8:00 PDT10/09/18 12:00 PDTGlobus EndpointsScheduled maintenance. Globus data transfers and NERSC endpoint activation (DTN, HPSS, Edison, Cori, PDSF, DTN-JGI, and shared NERSC endpoints) will not work.
10/02/18 17:42 PDT10/03/18 13:10 PDTHPSS UserUnavailable. System is currently unavailable. Engineers are investigating the issue, but no estimated uptime is available yet.
10/02/18 17:42 PDT10/03/18 13:09 PDTHPSS BackupUnavailable. System is currently unavailable. Engineers are investigating the issue, but no estimated uptime is available yet.
09/26/18 7:00 PDT09/27/18 2:29 PDTEdisonScheduled maintenance. Edison Monthly Maintenance
09/25/18 9:00 PDT09/25/18 19:35 PDTScience DatabasesScheduled maintenance. Postgres databases on nerscdb03 will be upgrade to v10 and unavailable during this period
09/24/18 8:41 PDT09/24/18 9:39 PDTEdisonSystem in degraded mode. Engineers checking into Lustre scratch issues
09/20/18 16:50 PDT10/17/18 7:00 PDTCoriSystem in degraded mode. Degraded state extended. DataWarp/BurstBuffer is currently unavailable. Bug identified, patch has been released, awaiting patch installation during planned Cori maintenance on 10/17.
09/19/18 9:00 PDT09/19/18 13:52 PDTHPSS UserScheduled maintenance.
09/19/18 7:00 PDT09/20/18 0:27 PDTCoriScheduled maintenance.
09/18/18 15:36 PDT09/18/18 17:30 PDTCoriDedicated runs. Large reservation of Cori Haswell nodes. All nodes for JGI and realtime queues, and limited nodes for debug queue, will remain available during this reservation. Interactive nodes will not be available. Note: this is a re-run of the dedicated runs which were interrupted this morning.
09/18/18 9:56 PDT09/18/18 15:36 PDTCoriUnavailable. Cori is currently unavailable, engineers are investigating the issue. Next update will be provided as more information becomes available.
09/18/18 9:00 PDT09/18/18 9:56 PDTCoriDedicated runs. Large reservation of Cori Haswell nodes. All nodes for JGI and realtime queues, and limited nodes for debug queue, will remain available during this reservation. Interactive nodes will not be available. Note: this reserved time was interrupted by an unplanned outage, and has been replaced by a subsequent reservation starting at 15:36 PDT.
09/17/18 6:11 PDT09/17/18 8:15 PDTCoriSystem in degraded mode. Existing jobs continue to run, new jobs are unable to start . Engineers are working to correct a network issue.
09/01/18 8:28 PDT09/01/18 10:00 PDTCoriSystem in degraded mode. cscratch1 perfomance is degraded, engineers investigating.
08/25/18 20:32 PDT08/26/18 3:04 PDTCoriSystem in degraded mode.
08/24/18 0:43 PDT08/24/18 4:28 PDTCoriSystem in degraded mode. The system is degraded, engineers are investigating. Logins are available, however jobs cannot be submitted.
08/21/18 10:46 PDT08/21/18 16:08 PDTPDSFSystem in degraded mode.
08/17/18 8:00 PDT08/21/18 7:12 PDTNERSC CenterScheduled maintenance. The NERSC facility will be conducting power maintenance. All services will be unavailable for the duration of the maintenance window.
08/17/18 8:00 PDT08/21/18 7:12 PDTCoriScheduled maintenance.
08/17/18 8:00 PDT08/17/18 19:26 PDTEdisonScheduled maintenance.
08/17/18 8:00 PDT08/21/18 16:49 PDTGenepoolScheduled maintenance.
08/17/18 8:00 PDT08/20/18 16:13 PDTPDSFScheduled maintenance.
08/17/18 8:00 PDT08/20/18 11:25 PDTDNAScheduled maintenance.
08/17/18 8:00 PDT08/20/18 11:25 PDTGlobal CommonScheduled maintenance.
08/17/18 8:00 PDT08/20/18 11:25 PDTGlobal HomesScheduled maintenance.
08/17/18 8:00 PDT08/20/18 11:25 PDTProjectScheduled maintenance.
08/17/18 8:00 PDT08/20/18 11:25 PDTProjectAScheduled maintenance.
08/17/18 8:00 PDT08/20/18 11:25 PDTProjectBScheduled maintenance.

Show Older Outages: