NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
06/23/17 12:00 PDT06/23/17 20:00 PDTCoriDedicated runs. A large scale dedicated science run will occupy the majority of Cori's KNL nodes. Haswell nodes will remain available.
06/23/17 10:30 PDT06/23/17 11:54 PDTCoriScheduled maintenance. Cori KNL nodes will be undergoing a scheduled maintenance for node performance improvements. Haswell nodes will remain available.
06/22/17 16:04 PDT06/22/17 16:31 PDTNX ServicesUnavailable. The NX server is unavailable. Engineers are investigating the issue.
06/21/17 9:00 PDT06/21/17 15:00 PDTHPSS UserScheduled maintenance. The end time of this Archive maintenance has been extended
06/15/17 14:30 PDT06/15/17 14:52 PDTCoriSystem in degraded mode. partitions have been downed in order to do maintenance work.
06/15/17 9:02 PDT06/15/17 9:07 PDTCoriSystem in degraded mode. System was unavailable, partitions were downed for hardware work.
06/13/17 7:00 PDT06/13/17 15:07 PDTEdisonScheduled maintenance. Hardware maintenance. Logins will be available.
06/12/17 23:50 PDT06/13/17 2:00 PDTCoriSystem in degraded mode. KNL mode changes are disabled.
06/11/17 7:43 PDT06/11/17 22:45 PDTCoriUnavailable. Cori was unavailable because of a failed coil board, resulting in a downed cabinet.
06/07/17 16:13 PDT06/07/17 19:10 PDTEdisonSystem in degraded mode.
06/07/17 11:40 PDT06/07/17 15:35 PDTEdisonUnavailable. System is unavailable due to hardware failure. Engineers are currently working on it.
05/31/17 19:08 PDT05/31/17 20:18 PDTCoriDedicated runs. Cori is performing dedicated runs for testing following planned maintenance.
05/31/17 8:00 PDT05/31/17 19:08 PDTCoriScheduled maintenance.
05/30/17 13:30 PDT05/30/17 13:41 PDTCoriSystem in degraded mode. SLURM partitions were paused for warm swapping compute nodes back into the system. The work has been completed.
05/25/17 7:00 PDT05/25/17 16:08 PDTEdisonScheduled maintenance.
05/24/17 18:12 PDT05/24/17 18:23 PDTEdisonUnavailable. Batch submission and query is unavailable. New job starts are unavailable and running jobs will continue to run
05/24/17 15:25 PDT05/24/17 15:45 PDTEdisonSystem in degraded mode. Engineers are investigating an issue with the batch system (slurm). Jobs cannot currently be submitted.
05/20/17 13:00 PDT05/20/17 16:20 PDTstaffdb01Unavailable. Staff database "staffdb01" is currently unavailable due to hard disk failure.
05/16/17 22:00 PDT05/17/17 2:33 PDTCoriDedicated runs. Logins and job submission available. Jobs will remained queued until the dedicated test is complete.
05/16/17 12:00 PDT05/16/17 17:00 PDTEdisonScheduled maintenance. /cscratch1 is unavailable until the Cori maintenance is complete.
05/16/17 12:00 PDT05/16/17 17:00 PDTData Transfer NodesScheduled maintenance. /cscratch1 is unavailable until the Cori maintenance is complete.
05/16/17 12:00 PDT05/16/17 17:00 PDTGenepoolScheduled maintenance. /cscratch1 is unavailable until the Cori maintenance is complete
05/16/17 12:00 PDT05/16/17 17:00 PDTJGI ServicesScheduled maintenance. /cscratch1 is unavailable until the Cori maintenance is complete.
05/16/17 11:15 PDT05/16/17 11:20 PDTNERSC WebsiteScheduled maintenance. Quarterly Maintenance - www.nersc.gov, my.nersc.gov, crd.lbl.gov and cs.lbl.gov will be undergoing scheduled maintenance
05/16/17 11:00 PDT05/16/17 11:20 PDTLicense servers: dmv, dmv1, dmv2Scheduled maintenance. Quarterly Maintenance
05/16/17 6:00 PDT05/16/17 22:20 PDTCoriScheduled maintenance. Quarterly Maintenance - logins will be disabled and system will be offline.
05/16/17 6:00 PDT05/16/17 16:00 PDTEdisonScheduled maintenance. Quarterly Maintenance - logins will be disabled and system will be offline while /home2 is unavailable.
05/16/17 6:00 PDT05/17/17 1:00 PDTNERSC CenterScheduled maintenance. NERSC will have a center wide outage beginning Tuesday May 16, 2017 from 06:00 PST until 17:00 PST. The work will include system patches, an fsck on /home2, DTN hardware work, and SGN patches.
05/16/17 6:00 PDT05/16/17 12:20 PDTJGI ServicesScheduled maintenance. Quarterly Maintenance - logins will be disabled and system will be offline while /home2 is unavailable.
05/16/17 6:00 PDT05/16/17 13:30 PDTData Transfer NodesScheduled maintenance. DTN will be offline to perform cabling changes and patches.
05/16/17 6:00 PDT05/16/17 14:00 PDTScience Gateway ServicesScheduled maintenance. Quarterly Maintenance
05/16/17 6:00 PDT05/16/17 7:11 PDTNX ServicesScheduled maintenance. Quarterly Maintenance
05/16/17 5:00 PDT05/16/17 13:20 PDTPDSFScheduled maintenance. Quarterly Maintenance - logins will be disabled and system will be offline.
05/16/17 5:00 PDT05/16/17 17:00 PDTGenepoolScheduled maintenance. Quarterly Maintenance - logins will be disabled and system will be offline
05/04/17 17:22 PDT05/04/17 22:15 PDTCoriSystem in degraded mode. Cori is experiencing a possible hardware issue that is under investigation. Logins and job submission are available but new jobs will not start
05/04/17 8:00 PDT05/04/17 16:30 PDTCoriScheduled maintenance. Cori will be undergoing planned maintenance during this time period.
04/25/17 15:48 PDT-CoriBurstBuffer users should read: www.nersc.gov/users/computational-systems/cori/burst-buffer/known-issues
04/25/17 14:54 PDT04/25/17 15:05 PDTCoriUnavailable. The batch system was unavailable. Logins were available.
04/25/17 1:00 PDT04/25/17 14:29 PDTEdisonUnavailable. Edison is currently unavailable and engineers are currently investigating the issue.
04/24/17 11:00 PDT04/24/17 17:25 PDTEdisonUnavailable. Edison is currently unavailable. Engineers are actively working on the situation.
04/21/17 16:53 PDT04/25/17 15:45 PDTCoriSystem in degraded mode. BurstBuffer users should read: www.nersc.gov/users/computational-systems/cori/burst-buffer/known-issues
04/20/17 21:15 PDT04/21/17 7:35 PDTCoriDedicated runs. Cori will be undergoing dedicated test runs follow maintenance. Logins and job submission will be available.
04/19/17 9:00 PDT04/19/17 12:30 PDTHPSS BackupScheduled maintenance. Regent will be unavailable while engineers work on the system.
04/19/17 9:00 PDT04/19/17 14:00 PDTHPSS UserScheduled maintenance. Archive will be unavailable while engineers work on the system.
04/19/17 7:00 PDT04/20/17 21:15 PDTCoriScheduled maintenance. Cori will be down for hardware upgrades.
04/18/17 6:00 PDT04/19/17 7:00 PDTCoriScheduled maintenance. A subset of KNL resources will be unavailable to prepare for the hardware upgrade on 04/19. No other resources will be affected.
04/15/17 9:00 PDT04/15/17 12:35 PDTCoriDedicated runs. System is scheduled for KNL full-scale runs. Haswell nodes will remain available. Job submission will still be available.
04/14/17 9:00 PDT04/14/17 21:00 PDTCoriDedicated runs. System is scheduled for KNL full-scale runs. Haswell nodes will remain available. Job submission will still be available.
04/13/17 19:07 PDT04/14/17 5:04 PDTCoriUnavailable. scratch filesystems unavailable. No new jobs starting. Engineers are investigating.
04/13/17 19:07 PDT04/14/17 3:58 PDTEdisonUnavailable. scratch filesystems unavailable. No new jobs starting. Engineers are investigating.
04/13/17 9:00 PDT04/13/17 22:00 PDTCoriDedicated runs. System is scheduled for KNL full-scale runs. Haswell nodes will remain available. Job submission will still be available.
04/12/17 5:50 PDT04/12/17 12:00 PDTNERSC WebsiteSystem in degraded mode. www.nersc.gov is currently degraded. Engineers are working to resolve the issue.
04/10/17 13:40 PDT04/10/17 14:00 PDTNERSC WebsiteUnavailable. www.nersc.gov, my.nersc.gov, cs.lbl.gov, and crd.lbl.gov are currently unavailable. Engineers are actively working to resolve this issue.
04/10/17 11:00 PDT04/10/17 13:00 PDTNERSC WebsiteScheduled maintenance. Maintenance will be performed on: www.nersc.gov, my.nersc.gov, crd.lbl.gov, cs.lbl.gov.
04/10/17 8:43 PDT04/10/17 16:13 PDTEdisonUnavailable. Edison is currently down due to failure of a node cabinet. Engineers are actively working to resolve the issue.
04/10/17 8:00 PDT04/10/17 18:00 PDTCoriDedicated runs. System is scheduled for KNL full-scale runs. Haswell nodes will remain available. Job submission will still be available.
04/07/17 15:45 PDT04/07/17 21:45 PDTCoriDedicated runs. System is scheduled for KNL full-scale runs. Haswell nodes will remain available. Job submission will still be available.
04/07/17 10:00 PDT04/07/17 10:45 PDTNX ServicesScheduled maintenance. NX license is being swapped. User sessions may be interrupted.
04/07/17 8:00 PDT04/07/17 15:42 PDTCoriScheduled maintenance. Engineers will be performing system maintenance to improve stability and reliability of KNL compute nodes.
04/05/17 11:11 PDT04/05/17 11:27 PDTCoriSystem in degraded mode. Job submission and slurm commands were unavailable during this event. Engineers are working to determine the root cause.
04/05/17 9:00 PDT04/05/17 10:45 PDTHPSS BackupScheduled maintenance. Regent will be unavailable while engineers work on the system.

Show Older Outages: