NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
02/25/17 11:10 PST02/25/17 17:18 PSTEdisonUnavailable. Engineers are actively working to resolve a rectifier issue with Edison row 2. Further updates will be provided as soon as possible. Logins are available however jobs are impacted.
02/24/17 16:15 PST02/24/17 16:38 PSTCoriLogins available, batch jobs not running. Engineers were working on an issue with the batch system.
02/24/17 13:04 PST-CoriThe BurstBuffer DW alternate pool 'sm_pool' is temporarily unavailable until March 1, 2017.
02/23/17 8:00 PST02/23/17 15:50 PSTEdisonScheduled maintenance. Engineers will be performing hardware maintenance. Login nodes will be available.
02/21/17 21:15 PST02/21/17 22:15 PSTCoriSystem in degraded mode. The majority of the system's compute nodes are currently unavailable. Engineers are investigating the issue
02/21/17 8:00 PST02/21/17 21:15 PSTCoriScheduled maintenance. Cori will be unavailable while updates are applied. Logins will be available, however no jobs will run.
02/21/17 0:05 PST02/21/17 9:24 PSTEdisonSystem in degraded mode. Edison is running in a degraded mode w/o /global/cscratch1. Engineers are investigating the issue.
02/18/17 12:22 PST02/18/17 13:07 PSTCoriSystem in degraded mode. Cori "cscratch1" is currently unavailable due to issues with its metadata servers. Engineers are actively working on this issue, and expect to have the filesystem up shortly.
02/17/17 22:37 PST02/17/17 22:45 PSTCoriSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/17/17 22:37 PST02/17/17 22:45 PSTEdisonSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/16/17 23:22 PST02/16/17 23:28 PSTCoriSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/16/17 16:40 PST02/16/17 17:02 PSTCoriSystem in degraded mode. /cscratch1 was briefly unavailable due to filesystem issues.
02/14/17 21:00 PST02/15/17 9:00 PSTGenepoolSystem in degraded mode. Jobs submissions were failing intermittently. All issues have been resolved.
02/14/17 7:00 PST02/14/17 7:55 PSTNIMScheduled maintenance. NIM will be unavailable for a planned update.
02/13/17 10:00 PST02/13/17 10:12 PSTCoriSystem in degraded mode. System is in degraded mode while engineers perform hardware maintenance. Jobs are running. Logins are available.
02/11/17 9:00 PST02/13/17 12:18 PSTProjectScheduled maintenance. Global /project is unavailable on all systems.
02/11/17 9:00 PST02/13/17 12:45 PSTData Transfer NodesSystem in degraded mode. /project is currently unavailable. See 'Project' listing for further details.
02/10/17 15:57 PST02/10/17 17:09 PSTEdisonSystem in degraded mode. cscratch1 unavailable, engineers investigating. Update by 18:00
02/10/17 15:57 PST02/10/17 17:09 PSTCoriUnavailable. cscratch1 unavailable, engineers investigating. Update by 18:00
02/10/17 15:57 PST02/10/17 17:09 PSTData Transfer NodesSystem in degraded mode. cscratch1 was unavailable, engineers investigating.
02/10/17 12:38 PST-GenepoolJobs requiring /project will not run. See 'Project' listing for further details.
02/10/17 10:00 PST02/10/17 10:48 PSTGenepoolScheduled maintenance. Genepool is down due to emergency maintenance. All queues are stopped and jobs will be purged. Logins will remain available.
02/09/17 19:01 PST02/13/17 17:15 PSTCoriSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See 'Project' listing for further details.
02/09/17 16:08 PST02/09/17 19:20 PSTEdisonSystem in degraded mode. In addition, there is an issue with /cscratch. It will be unavailable. Engineers are working to resolve this issue.
02/09/17 15:49 PST02/09/17 19:01 PSTCoriUnavailable. Cori is unavailable while engineers work to resolve a filesystem issue.
02/08/17 9:00 PST02/08/17 11:50 PSTHPSS UserScheduled maintenance.
02/07/17 22:38 PST02/13/17 12:47 PSTPDSFSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See 'Project' listing for further details.
02/07/17 22:38 PST02/10/17 11:00 PSTGenepoolSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See Project for further details.
02/07/17 22:38 PST02/13/17 13:24 PSTEdisonSystem in degraded mode. Logins are available. Jobs not requiring /project will run. See 'Project' listing for further details.
02/07/17 22:38 PST02/09/17 15:49 PSTCoriSystem in degraded mode. Global /project is available read-only for users who need data access until Friday Noon PST. Logins are available. Jobs not requiring /project will run. See Project MOTD for details.
02/07/17 22:38 PST02/13/17 15:01 PSTScience Gateway ServicesUnavailable. Many services are unavailable or have impacted performance due to the down state of /project.
02/06/17 13:05 PST02/07/17 3:30 PSTGlobal HomesUnavailable. The system will be unavailable due to a center wide outage.
02/06/17 13:05 PST02/11/17 9:00 PSTProjectScheduled maintenance. /project is currently mounted read-only on DTN and Edison login nodes until approx. 09:00 AM Feb. 11th. Afterwards, /project will be taken offline until 2/15 for further maintenance.
02/06/17 12:24 PST02/07/17 10:36 PSTNX ServicesUnavailable. The system will be unavailable for a planned maintenance.
02/06/17 8:17 PST02/06/17 12:43 PSTNetworkUnavailable. The system will be unavailable due to a center wide outage.
02/06/17 7:14 PST02/06/17 16:00 PSTJGI ServicesScheduled maintenance. Various server and web service maintenance.
02/06/17 6:30 PST02/07/17 11:28 PSTPDSFScheduled maintenance. The system will be unavailable due to a center wide outage.
02/06/17 6:00 PST02/07/17 22:28 PSTNERSC CenterScheduled maintenance. NERSC will have a center wide outage beginning Monday, Feb 6, 2017 at 06:00 am PST through Tuesday, Feb 7, 2017 21:00 pm PST. The work includes facility electrical work, an fsck on /home and /project and network maintenance.
02/06/17 6:00 PST02/07/17 8:50 PSTGenepoolScheduled maintenance. The system will be unavailable due to a center wide outage.
02/02/17 7:02 PST02/02/17 9:45 PSTCoriUnavailable. No new jobs are currently starting; running jobs should continue; global filesystems may be unavailable.
01/29/17 18:45 PST01/30/17 2:07 PSTEdisonUnavailable. Edison is down due to a hardware issue. Engineers are investigating the issue.
01/25/17 18:00 PST01/25/17 20:24 PSTNIMUnavailable. NIM is experiencing some interment problems. Engineers are currently working on it.
01/25/17 8:00 PST01/25/17 9:30 PSTNIMScheduled maintenance.
01/19/17 8:08 PST01/19/17 16:27 PSTEdisonScheduled maintenance. /scratch2 will be upgraded and not accessible during this time.
01/18/17 12:39 PST01/18/17 14:00 PSTCoriUnavailable. High Speed Network issues generating slurm, gpfs, and datawarp inaccessibility
01/17/17 20:00 PST01/18/17 19:55 PSTEdisonSystem in degraded mode. /scratch3 is currently unavailable. Engineers are investigating the issue.
01/17/17 12:06 PST01/17/17 12:40 PSTGlobal HomesUnavailable. Logins are unavailable during this time.
01/17/17 12:06 PST01/17/17 12:40 PSTGlobal CommonUnavailable. Engineers are working to resolve this issue.
01/17/17 12:06 PST01/17/17 13:07 PSTCoriUnavailable. System unavailable due to a filesystem issue. Engineers are working to resolve the problem ASAP.
01/17/17 12:06 PST01/17/17 14:00 PSTGenepoolUnavailable. System unavailable due to a filesystem issue. Engineers are working to resolve the problem ASAP.
01/17/17 12:06 PST01/17/17 13:10 PSTPDSFUnavailable. System unavailable due to a filesystem issue. Engineers are working to resolve the problem ASAP.
01/17/17 12:01 PST01/17/17 13:07 PSTNERSC CenterUnavailable. Center wide outage due to a problem with the filesystems. Engineers are working to bring systems back up ASAP.
01/17/17 10:00 PST01/17/17 10:45 PSTMongoDB ServicesScheduled maintenance. Services on mongodb01 and mongodb02 will be unavailable or at-risk for upgrade of MongoDB version to 3.2. Services on mongodb03 and mongodb04 are available.
01/17/17 8:00 PST01/17/17 20:00 PSTEdisonScheduled maintenance. Edison is currently unavailable while engineers perform scheduled maintenance.
01/17/17 8:00 PST01/17/17 8:42 PSTNIMScheduled maintenance. The NIM interface will be inaccessible during this time.
01/14/17 13:40 PST01/14/17 22:30 PSTCoriUnavailable. Cori is down due to HSN issues. Engineers are actively working to resolve this issue as soon as possible.
01/13/17 16:45 PST01/13/17 17:15 PSTProjectB/projectb unavailable. /projectb is experiencing some intermittent problems. Engineers are currently investigating the issue.
01/13/17 11:00 PST01/13/17 11:41 PSTEdisonSystem in degraded mode. /scratch3 filesystem is having intermittent issues. Engineers are actively working on the problem.
01/12/17 8:00 PST01/12/17 18:14 PSTEdisonScheduled maintenance. /scratch1 will be upgraded and not accessible during this time.
01/11/17 15:38 PST01/11/17 16:57 PSTCoriSystem in degraded mode. No KNL jobs can be submitted nor started. Existing running jobs OK. Engineers investigating. Haswell job submission, start, and running are working properly at this time.
01/11/17 9:03 PST01/11/17 15:15 PSTNX ServicesScheduled maintenance. NX unavailable for maintenance and upgrades. All connected sessions will be terminated.
01/10/17 14:45 PST01/10/17 14:48 PSTCoriSystem in degraded mode. Slurm partitions are currently down. Engineers are currently working on the system.

Show Older Outages: