NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
12/15/17 9:00 PST12/15/17 13:00 PSTHPSS BackupScheduled maintenance. The system will be undergoing scheduled maintenance at this time.
11/15/17 1:00 PST11/15/17 4:15 PSTCoriSystem in degraded mode. The system was in degraded mode during the specified times. The system is now back up.
11/14/17 7:00 PST11/14/17 7:50 PSTNIMScheduled maintenance.
11/09/17 9:00 PST11/09/17 12:00 PSTCoriScheduled maintenance. System will be in Degraded Mode. Datawarp will be unavailable 09:00 to 13:00 for a critical software update. Jobs using Datawarp (BurstBuffer) will not run or will be cancelled.
11/08/17 9:00 PST11/08/17 12:25 PSTHPSS UserScheduled maintenance.
11/02/17 22:23 PDT11/03/17 2:00 PDTCoriSystem in degraded mode. Cori was in a degraded mode. Jobs are running and logins are available.
11/02/17 4:09 PDT11/02/17 7:30 PDTHPSS BackupUnavailable.
10/27/17 22:40 PDT10/27/17 23:54 PDTHPSS UserUnavailable. System will be restarted due to migration issues on Archive.
10/26/17 17:31 PDT10/26/17 18:01 PDTCoriSystem in degraded mode. Cori was degraded during this time due to a failure of the batch scheduler. Existing jobs were unaffected, but new jobs did not start during this timeframe.
10/25/17 10:00 PDT10/25/17 14:00 PDTGenepool WebsitesScheduled maintenance. Services will be unavailable during a storage system software and firmware upgrade.
10/24/17 17:11 PDT10/24/17 18:00 PDTEdisonLogins available, batch jobs not running. Edison was down briefly due to failed service nodes.
10/23/17 9:00 PDT10/24/17 9:00 PDTCoriDedicated runs. Large scale science runs. During this time users can login and submit jobs as usual. Small jobs may run, but larger jobs that require more nodes than available may not run until the reservation is complete.
10/18/17 1:55 PDT10/18/17 8:00 PDTHPSS UserSystem in degraded mode. Archive is in a degraded state. NERSC engineers are working with the vendor to resolve the issue.
10/17/17 9:00 PDT10/17/17 17:00 PDTPostgres Databases on scidb1/2Unavailable. Postgres databases on scidb1/2 aka nerscdb01/02 will be migrating to a new host and unavailable during this period. After this date these dbs should be accessed only from the new nerscdb03.nersc.gov host. Users of these services should have received an email with this info, please contact us with any issues. MySQL services are unaffected but will be migrated at a later date
10/15/17 13:35 PDT10/15/17 14:46 PDTCoriSystem in degraded mode. System filesystem issue, no new jobs will start.
10/06/17 15:00 PDT10/09/17 22:43 PDTNERSC CenterScheduled maintenance. NERSC quarterly maintenance. This will include power and cooling across the Center.
10/06/17 15:00 PDT10/10/17 20:23 PDTCoriScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:38 PDTDNAScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 11:43 PDTData Transfer NodesScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 22:43 PDTEdisonScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/10/17 16:19 PDTGenepoolScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:38 PDTGlobal CommonScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:38 PDTGlobal HomesScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 15:30 PDTHPSS BackupScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 15:30 PDTHPSS UserScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/10/17 14:25 PDTPDSFScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:38 PDTProjectScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:38 PDTProjectAScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:38 PDTProjectBScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/10/17 11:23 PDTNIM; MyNERSC; NERSC, CS, & CRD web sitesScheduled maintenance. NERSC center-wide shutdown and electrical upgrades. A basic informational website will be available.
10/06/17 15:00 PDT10/09/17 11:10 PDTMongoDBScheduled maintenance. Scheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 13:20 PDTMatgenScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 12:50 PDT Science Gateways and DatabasesScheduled maintenance. Scheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 10:57 PDTNX ServicesScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 11:03 PDTJGI Oracle DatabaseScheduled maintenance.
10/06/17 15:00 PDT10/09/17 12:05 PDTGenepool Websites and DatabasesScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/08/17 18:30 PDTMyProxyScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/06/17 15:00 PDT10/09/17 12:58 PDTSpinScheduled maintenance. NERSC center-wide shutdown and electrical upgrades.
10/05/17 7:30 PDT10/05/17 8:50 PDTNX ServicesUnavailable. NX is unavailable. Engineers are investigating the problem
10/04/17 13:00 PDT10/04/17 15:00 PDTSpinScheduled maintenance. No impact on services is expected, but staff will be unable to create or reconfigure services during the maintenance.
09/22/17 14:29 PDT-CoriA large portion of KNL nodes will be reserved for science runs on Monday 9/25/17 at 11:00 to Tuesday 9/26/17 at 11:00. Login nodes, job submissions, and Haswell nodes will remain available. Knl_interactive, knl, and knl_regularx partitions, however, will have limited availability.
09/19/17 15:15 PDT09/19/17 18:30 PDTHPSS BackupUnavailable. HPSS is currently unavailable due to electrical issues on site. Updates will be provided as they become available.
09/19/17 15:15 PDT09/19/17 18:30 PDTHPSS UserUnavailable. HPSS is currently unavailable due to electrical issues on site. Updates will be provided as they become available.
09/19/17 11:00 PDT09/20/17 11:00 PDTCoriDedicated runs. Large scale reservations for science runs. All KNL nodes are reserved. Login nodes, job submission, and Haswell nodes will remain available.
09/13/17 9:10 PDT-HPSS BackupHPSS will remain available during the NERSC scheduled maintenance 10/06/17 13:00 - 10/10/17 17:00 PDT
09/13/17 9:10 PDT-HPSS UserHPSS will remain available during the NERSC scheduled maintenance 10/06/17 13:00 - 10/10/17 17:00 PDT
09/13/17 8:00 PDT09/13/17 13:37 PDTHPSS UserScheduled maintenance.
09/12/17 14:21 PDT09/12/17 14:32 PDTCoriSystem in degraded mode. Logins are available, batch jobs are running. New jobs will not start due to hardware work.
09/06/17 18:42 PDT09/07/17 6:00 PDTCoriDedicated runs. All KNL nodes are reserved for large scale science runs. KNL job submission is available, and Haswell nodes are open.
09/06/17 8:00 PDT09/06/17 18:42 PDTCoriScheduled maintenance.
09/05/17 15:40 PDT09/05/17 18:09 PDTCoriSystem in degraded mode. Due to extreme weather conditions in Berkeley the KNL partitions are down. New KNL jobs won't start.
09/05/17 13:31 PDT09/05/17 15:40 PDTCoriLogins available, batch jobs not running. Due to extreme weather conditions in Berkeley the batch scheduling partitions are down.
09/05/17 13:31 PDT09/05/17 15:40 PDTEdisonLogins available, batch jobs not running. Due to extreme weather conditions in Berkeley the batch scheduling partitions are down.
09/04/17 17:01 PDT09/04/17 20:40 PDTEdisonUnavailable.
09/04/17 13:42 PDT09/04/17 14:05 PDTCoriSystem in degraded mode. Slurm commands and job submission was unavailable during this time due to software issues, which have been corrected.
09/02/17 14:25 PDT09/02/17 18:35 PDTEdisonUnavailable. All queues have been paused due to high temperature conditions in Berkeley.
09/02/17 11:30 PDT09/02/17 19:00 PDTCoriSystem in degraded mode. All queues have been paused due to high temperature conditions in Berkeley.
09/02/17 11:30 PDT09/02/17 14:25 PDTEdisonSystem in degraded mode. All queues have been paused due to high temperature conditions in Berkeley.
09/01/17 18:58 PDT-EdisonDue to extreme temperatures in the Bay Area, the scheduler will be paused during the hot parts of the day this weekend. Queues will resume as the temperature permits.
09/01/17 18:57 PDT-CoriDue to extreme temperatures in the Bay Area, the scheduler will be paused during the hot parts of the day this weekend. Queues will resume as the temperature permits.
09/01/17 15:30 PDT09/01/17 19:30 PDTGenepoolLogins available, batch jobs not running. Due to extreme temperatures in the Bay Area, the scheduler will be paused during the hot parts of the day this weekend. Queues will resume as the temperature permits.
09/01/17 15:30 PDT09/01/17 19:30 PDTPDSFLogins available, batch jobs not running. Due to extreme temperatures in the Bay Area, the scheduler will be paused during the hot parts of the day this weekend. Queues will resume as the temperature permits.
09/01/17 14:43 PDT09/01/17 18:29 PDTCoriLogins available, batch jobs not running. Due to extreme temperatures in the Bay Area, the scheduler will be paused during the hot parts of the day this weekend. Queues will resume as the temperature permits.
09/01/17 14:43 PDT09/01/17 18:34 PDTEdisonLogins available, batch jobs not running. Due to extreme temperatures in the Bay Area, the scheduler will be paused during the hot parts of the day this weekend. Queues will resume as the temperature permits.
08/31/17 8:23 PDT08/31/17 8:30 PDTNetworkUnavailable. Brief network interruption on circuits between ESnet, NERSC and LBL.

Show Older Outages: