NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
03/24/17 10:00 PDT03/24/17 12:30 PDTEdisonScheduled maintenance. Edison is unavailable. SLURM is being upgraded.
03/23/17 18:00 PDT03/23/17 21:00 PDTCoriUnavailable. The system is down for a short time while engineers resolve remaining issues from scheduled maintenance.
03/23/17 5:49 PDT03/23/17 6:13 PDTEdisonSystem in degraded mode. The Edison scheduler was down, and new jobs could not be submitted. Running jobs were not affected. Engineers are currently investigating.
03/22/17 11:00 PDT03/22/17 11:30 PDTNIMScheduled maintenance. Engineers are performing a planned maintenance. NIM will be unavailable.
03/21/17 7:00 PDT03/23/17 18:00 PDTCoriScheduled maintenance.
03/19/17 12:00 PDT03/19/17 21:30 PDTCoriSystem in degraded mode. KNL jobs were unable to be submitted, and new KNL jobs would not start.
03/16/17 16:48 PDT03/21/17 7:00 PDTCoriSystem in degraded mode. Cori is in degraded mode at this time. Only knl,quad,cache and haswell jobs will run. This will be fixed after the upgrade tomorrow. Logins are still available.
03/16/17 14:17 PDT03/16/17 14:36 PDTGenepoolSystem in degraded mode. Genepool Web Servers are degraded. Engineers are working to resolve the issue ASAP.
03/16/17 7:07 PDT03/16/17 7:15 PDTCoriSystem in degraded mode. SLURM was briefly unavailable, no jobs could be submitted and some running jobs may have been affected.
03/15/17 9:02 PDT03/15/17 12:59 PDTHPSS UserScheduled maintenance. Archive (User) Scheduled Maintenance. The system is down.
03/15/17 8:06 PDT03/15/17 9:55 PDTScience Database ServicesUnavailable. mysql and postgres services hosted on scidb1 were unavailable. Engineers have resolved the issue.
03/12/17 8:00 PDT03/12/17 16:05 PDTCoriUnavailable. Jobs are running. New jobs will not start. Engineers are working to solve the issue.
03/10/17 14:16 PST03/10/17 15:47 PSTEdisonSystem in degraded mode. Jobs requiring /scratch3 will not run. Engineers are actively working on the situation.
03/07/17 19:15 PST03/08/17 9:30 PSTCoriSystem in degraded mode. Datawarp is unavailable. Engineers are investigating the issue
03/07/17 13:56 PST03/07/17 15:00 PSTNX ServicesUnavailable.
03/07/17 7:30 PST03/07/17 9:30 PSTNIMScheduled maintenance. NIM will be unavailable.
03/03/17 11:50 PST03/03/17 15:40 PSTScience Gateway ServicesSystem in degraded mode. Engineers are currently investigating a possible network issue with the Science Gateway nodes. Portal sites may be affected.
03/01/17 9:00 PST03/01/17 10:23 PSTData Transfer NodesScheduled maintenance. DTN cluster will be down for maintenance.
03/01/17 9:00 PST03/01/17 12:53 PSTHPSS BackupScheduled maintenance.
03/01/17 9:00 PST03/01/17 9:19 PSTMatgenScheduled maintenance. Logins will be unavailable. Running jobs should not be affected.
03/01/17 6:00 PST03/03/17 19:34 PSTCoriScheduled maintenance. Cori was down for cabinet additions and HSN (high-speed network) maintenance. Logins will not be available.
02/28/17 6:00 PST03/01/17 6:00 PSTCoriScheduled maintenance. Cori will be degraded due to cabinet additions. Datawarp nodes will be reduced during this time.
02/28/17 1:35 PST02/28/17 4:25 PSTCoriSystem in degraded mode. /globla/cscratch1 filesystem was in degrade mode, preventing new jobs to run.
02/25/17 11:10 PST02/25/17 17:18 PSTEdisonUnavailable. Engineers are actively working to resolve a rectifier issue with Edison row 2. Further updates will be provided as soon as possible. Logins are available however jobs are impacted.
02/24/17 16:15 PST02/24/17 16:38 PSTCoriLogins available, batch jobs not running. Engineers were working on an issue with the batch system.
02/24/17 13:04 PST-CoriThe BurstBuffer DW alternate pool 'sm_pool' is temporarily unavailable until March 1, 2017.
02/23/17 8:00 PST02/23/17 15:50 PSTEdisonScheduled maintenance. Engineers will be performing hardware maintenance. Login nodes will be available.
02/21/17 21:15 PST02/21/17 22:15 PSTCoriSystem in degraded mode. The majority of the system's compute nodes are currently unavailable. Engineers are investigating the issue
02/21/17 8:00 PST02/21/17 21:15 PSTCoriScheduled maintenance. Cori will be unavailable while updates are applied. Logins will be available, however no jobs will run.
02/21/17 0:05 PST02/21/17 9:24 PSTEdisonSystem in degraded mode. Edison is running in a degraded mode w/o /global/cscratch1. Engineers are investigating the issue.
02/18/17 12:22 PST02/18/17 13:07 PSTCoriSystem in degraded mode. Cori "cscratch1" is currently unavailable due to issues with its metadata servers. Engineers are actively working on this issue, and expect to have the filesystem up shortly.
02/17/17 22:37 PST02/17/17 22:45 PSTCoriSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/17/17 22:37 PST02/17/17 22:45 PSTEdisonSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/16/17 23:22 PST02/16/17 23:28 PSTCoriSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/16/17 16:40 PST02/16/17 17:02 PSTCoriSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/14/17 21:00 PST02/15/17 9:00 PSTGenepoolSystem in degraded mode. Jobs submissions were failing intermittently. All issues have been resolved.
02/14/17 7:00 PST02/14/17 7:55 PSTNIMScheduled maintenance. NIM will be unavailable for a planned update.
02/13/17 10:15 PST02/13/17 11:30 PSTCoriUnavailable. SLURM paused for Cray maintenance.
02/13/17 10:00 PST02/13/17 10:12 PSTCoriSystem in degraded mode. System is in degraded mode while engineers perform hardware maintenance. Jobs are running. Logins are available.
02/11/17 9:00 PST02/13/17 12:18 PSTProjectScheduled maintenance. Global /project is unavailable on all systems.
02/11/17 9:00 PST02/13/17 12:45 PSTData Transfer NodesSystem in degraded mode. /project is currently unavailable. See 'Project' listing for further details.
02/10/17 15:57 PST02/10/17 17:09 PSTEdisonSystem in degraded mode. cscratch1 unavailable, engineers investigating. Update by 18:00
02/10/17 15:57 PST02/10/17 17:09 PSTCoriUnavailable. cscratch1 unavailable, engineers investigating. Update by 18:00
02/10/17 15:57 PST02/10/17 17:09 PSTData Transfer NodesSystem in degraded mode. cscratch1 was unavailable, engineers investigating.
02/10/17 12:38 PST-GenepoolJobs requiring /project will not run. See 'Project' listing for further details.
02/10/17 10:00 PST02/10/17 10:48 PSTGenepoolScheduled maintenance. Genepool is down due to emergency maintenance. All queues are stopped and jobs will be purged. Logins will remain available.
02/09/17 19:01 PST02/13/17 17:15 PSTCoriSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See 'Project' listing for further details.
02/09/17 16:08 PST02/09/17 19:20 PSTEdisonSystem in degraded mode. In addition, there is an issue with /cscratch. It will be unavailable. Engineers are working to resolve this issue.
02/09/17 15:49 PST02/09/17 19:01 PSTCoriUnavailable. Cori is unavailable while engineers work to resolve a filesystem issue.
02/08/17 9:00 PST02/08/17 11:50 PSTHPSS UserScheduled maintenance.
02/07/17 22:38 PST02/13/17 12:47 PSTPDSFSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See 'Project' listing for further details.
02/07/17 22:38 PST02/10/17 11:00 PSTGenepoolSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See Project for further details.
02/07/17 22:38 PST02/13/17 13:24 PSTEdisonSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See 'Project' listing for further details.
02/07/17 22:38 PST02/09/17 15:49 PSTCoriSystem in degraded mode. Global /project is available read-only for users who need data access until Friday Noon PST. Logins are available. Jobs not requiring /project will run. See Project MOTD for details.
02/07/17 22:38 PST02/13/17 15:01 PSTScience Gateway ServicesUnavailable. Many services are unavailable or have impacted performance due to the down state of /project.
02/06/17 13:05 PST02/07/17 3:30 PSTGlobal HomesUnavailable. The system will be unavailable due to a center wide outage.
02/06/17 13:05 PST02/11/17 9:00 PSTProjectScheduled maintenance. /project is currently mounted read-only on DTN and Edison login nodes until approx. 09:00 AM Feb. 11th. Afterwards, /project will be taken offline until 2/15 for further maintenance.
02/06/17 12:24 PST02/07/17 10:36 PSTNX ServicesUnavailable. The system will be unavailable for a planned maintenance.
02/06/17 8:17 PST02/06/17 12:43 PSTNetworkUnavailable. The system will be unavailable due to a center wide outage.
02/06/17 7:14 PST02/06/17 16:00 PSTJGI ServicesScheduled maintenance. Various server and web service maintenance.
02/06/17 6:30 PST02/07/17 11:28 PSTPDSFScheduled maintenance. The system will be unavailable due to a center wide outage.
02/06/17 6:30 PST02/07/17 22:38 PSTCoriScheduled maintenance. The system will be unavailable due to a center wide outage.

Show Older Outages: