NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
01/17/18 9:00 PST01/17/18 11:15 PSTHPSS BackupScheduled maintenance.
01/16/18 14:24 PST-Edisonxfer queue is currently unavailable. Engineers are working to resolve the issue.
01/16/18 14:20 PST-Corixfer and bigmem queues are currently unavailable. Engineers are working to resolve the issue.
01/11/18 10:45 PST01/11/18 11:12 PSTEdisonLogins available, batch jobs not running. Edison was unavailable 10:45-11:12 due to an issue with Slurm.
01/10/18 21:26 PST01/10/18 22:38 PSTCoriUnavailable. Cori's batch scheduler was unavailable during this time
01/10/18 6:24 PST01/10/18 9:25 PSTjupyter-dev.nersc.govUnavailable. Jupyter-dev.nersc.gov was unavailable 6:24 to 9:25. The issue has been resolved.
01/10/18 5:00 PST01/10/18 6:42 PSTCoriUnavailable. Cori was unavailabe from 05:00--06:42 PST due to a wedging issue. The system has been brought back up by engineering and is currently fully functional.
01/09/18 23:46 PST-Corixfer and bigmem queues are unavailable.
01/09/18 22:15 PST-EdisonProduction delayed due to slurm accounting database issues. Next update on 11:00 PST
01/09/18 22:05 PST-CoriMaintenance extended to 01/09/18 23:00 PST
01/09/18 9:00 PST01/09/18 11:30 PSTNX ServicesScheduled maintenance. NX unavailable for software upgrades and maintenance.
01/09/18 8:20 PST01/09/18 18:00 PSTHPSS BackupScheduled maintenance. Please note that this maintenance will fall on a Tuesday (01/09), to coincide with other scheduled system maintenances. Scheduled maintenance of the HPSS Backup system. The system will be unavailable during this window of time. There will be another update of the status at 18:00.
01/09/18 8:00 PST01/09/18 18:27 PSTGenepoolScheduled maintenance. Genepool will be patched for security improvements during times listed.
01/09/18 8:00 PST01/09/18 12:33 PSTGenepool Web and Database ServicesScheduled maintenance. Individual services will be briefly unavailable (5-15 minutes) while security updates are applied.
01/09/18 8:00 PST01/09/18 8:40 PSTNIMScheduled maintenance. Service will be briefly unavailable (5-15 minutes) while security updates are applied.
01/09/18 8:00 PST01/09/18 8:36 PSTMyProxyScheduled maintenance. Service will be briefly unavailable (5-15 minutes) while security updates are applied.
01/09/18 8:00 PST01/09/18 12:30 PSTScience Gateway ServicesScheduled maintenance. Individual services will be briefly unavailable (15-45 minutes) while security updates are applied.
01/09/18 8:00 PST01/09/18 17:55 PSTSpinScheduled maintenance. Individual services will be briefly unavailable (1-2 minutes) at various times during the maintenance window while security updates are applied.
01/09/18 8:00 PST01/09/18 18:00 PSTMendelScheduled maintenance. Genepool and Denovo will be patched for security improvements during times listed.
01/09/18 6:00 PST01/09/18 23:00 PSTCoriScheduled maintenance. Cori will be unavailable, while engineers perform system maintenance for the specified duration.
01/09/18 6:00 PST01/10/18 10:14 PSTEdisonScheduled maintenance. Edison was be unavailable, while engineers performed system maintenance for the specified duration. Production was delayed due to slurm accounting database issues. Edison has returned to service.
01/03/18 9:00 PST01/03/18 15:00 PSTHPSS BackupScheduled maintenance. Scheduled maintenance of the HPSS Backup system. The system will be unavailable during this window of time.
01/03/18 9:00 PST01/03/18 14:10 PSTHPSS UserScheduled maintenance.
12/27/17 9:00 PST12/27/17 16:00 PSTHPSS UserScheduled maintenance. The system will be unavailable due to scheduled maintenance.
12/25/17 4:37 PST12/25/17 12:50 PSTHPSS BackupSystem in degraded mode. Data writes may fail. NERSC engineers are working on the issue.
12/19/17 22:56 PST-PDSFJobs specifying /project filesystem licenses will currently not run.
12/19/17 14:09 PST12/19/17 14:33 PSTEdisonUnavailable. Edison was unavailable due to /csratch1 filesystem issues.
12/19/17 13:13 PST12/19/17 14:33 PSTCoriUnavailable. Cori was unavailable due to /csratch1 filesystem issues.
12/19/17 13:13 PST12/19/17 14:09 PSTEdisonSystem in degraded mode. Edison /csratch1 filesystem is currently degraded. Engineers are currently investigating the issue and will provide updates.
12/19/17 11:15 PST12/19/17 11:29 PSTCoriSystem in degraded mode. Cori Slurm partitions were down for duration of time listed due to warm-swapping. The system is back up and fully functional.
12/19/17 10:40 PST12/19/17 10:56 PSTCoriSystem in degraded mode. Cori Slurm partitions were down for duration of time listed due to warm-swapping. The system is back up and fully functional.
12/19/17 9:47 PST-EdisonJobs specifying /project filesystem licenses will currently not run.
12/19/17 9:47 PST-CoriJobs specifying /project filesystem licenses will currently not run.
12/19/17 8:00 PST12/20/17 10:10 PSTProject/project unavailable. A number of project directory quotas have been set too low, engineers are investigating the issue
12/15/17 9:00 PST12/15/17 13:00 PSTHPSS BackupScheduled maintenance. The system will be undergoing scheduled maintenance at this time.
12/14/17 11:45 PST12/14/17 12:45 PSTScience Gateway ServicesSystem in degraded mode. Science gateway nodes were briefly degraded during the times listed. System availability is currently restored.
12/14/17 6:00 PST12/14/17 18:52 PSTCoriScheduled maintenance.
12/12/17 6:45 PST12/12/17 7:30 PSTCoriUnavailable. Cori was unavailable for a brief period of time due to software issues.
12/12/17 6:00 PST12/12/17 20:36 PSTEdisonScheduled maintenance. Logins, mainframe and filesystems will be unavailable.
12/10/17 20:40 PST12/10/17 22:32 PSTCoriSystem in degraded mode.
12/08/17 13:52 PST12/08/17 14:23 PSTEdisonSystem in degraded mode. Edison was is in a degraded state due to a job queue problem.
12/08/17 10:11 PST12/08/17 10:34 PSTCoriSystem in degraded mode. Partitions were down due to hardware maintenance.
12/07/17 3:37 PST12/07/17 5:47 PSTCoriUnavailable. The system is back up as of 05:47 PST.
12/06/17 9:00 PST12/06/17 15:42 PSTHPSS BackupScheduled maintenance.
12/06/17 9:00 PST12/06/17 13:40 PSTHPSS UserScheduled maintenance.
12/05/17 15:10 PST12/05/17 17:00 PSTJupyterdevUnavailable. Jupyterdev is currently unavailable while engineers reboot service node.
12/04/17 9:06 PST12/04/17 12:05 PSTPDSFLogins available, batch jobs not running. SLURM was unavailable. Users could log in, but no jobs could be submitted. Running jobs may have been affected.
12/04/17 8:45 PST-Cori12/4/17 9:00 AM-12/5/17 5:00 AM PDT Dedicated runs. Large scale science runs. During this time users can login and submit jobs as usual. Small jobs may run, but larger jobs that require more nodes than available may not run until the reservation is complete.
12/01/17 21:24 PST12/02/17 2:24 PSTEdisonSystem in degraded mode. Cori is in a degraded state due to filesystem issues. Existing jobs are running, and new jobs can be submitted, but will not start. Engineers are working to bring the systems back up.
12/01/17 21:18 PST12/02/17 2:24 PSTCoriSystem in degraded mode. Cori is in a degraded state due to filesystem issues. Existing jobs are running, and new jobs can be submitted, but will not start. Engineers are working to bring the systems back up.
11/30/17 11:31 PST11/30/17 12:42 PSTCoriSystem in degraded mode.
11/29/17 9:00 PST11/29/17 17:00 PSTMySQL Databases on scidb1Scheduled maintenance. MySQL databases on scidb1 aka nerscdb01 will be migrating to a new host and unavailable during this period. After this date these dbs should be accessed only from the new nerscdb04.nersc.gov host. Users of these services should have received an email with this info, please contact us with any issues.
11/22/17 13:25 PST11/22/17 19:55 PSTCoriSystem in degraded mode. /cscratch is currently unavailable. Engineers are investigating the issue.
11/22/17 12:45 PST11/22/17 19:59 PSTEdisonUnavailable. /scratch3 is currently unavailable. Engineers are investigating the issue.
11/22/17 9:00 PST11/22/17 11:55 PSTHPSS UserScheduled maintenance. HPSS archive will be unavailable during this scheduled maintenance.
11/20/17 17:50 PST11/21/17 10:20 PSTScience Database, MySQL ServerUnavailable.
11/15/17 1:00 PST11/15/17 4:15 PSTCoriSystem in degraded mode. The system was in degraded mode during the specified times. The system is now back up.
11/14/17 7:00 PST11/14/17 7:50 PSTNIMScheduled maintenance.
11/09/17 9:00 PST11/09/17 12:00 PSTCoriScheduled maintenance. System will be in Degraded Mode. Datawarp will be unavailable 09:00 to 13:00 for a critical software update. Jobs using Datawarp (BurstBuffer) will not run or will be cancelled.
11/08/17 9:00 PST11/08/17 12:25 PSTHPSS UserScheduled maintenance.
11/02/17 22:23 PDT11/03/17 2:00 PDTCoriSystem in degraded mode. Cori was in a degraded mode. Jobs are running and logins are available.
11/02/17 4:09 PDT11/02/17 7:30 PDTHPSS BackupUnavailable.
10/27/17 22:40 PDT10/27/17 23:54 PDTHPSS UserUnavailable. System will be restarted due to migration issues on Archive.
10/26/17 17:31 PDT10/26/17 18:01 PDTCoriSystem in degraded mode. Cori was degraded during this time due to a failure of the batch scheduler. Existing jobs were unaffected, but new jobs did not start during this timeframe.
10/25/17 10:00 PDT10/25/17 14:00 PDTGenepool WebsitesScheduled maintenance. Services will be unavailable during a storage system software and firmware upgrade.
10/24/17 17:11 PDT10/24/17 18:00 PDTEdisonLogins available, batch jobs not running. Edison was down briefly due to failed service nodes.
10/23/17 9:00 PDT10/24/17 9:00 PDTCoriDedicated runs. Large scale science runs. During this time users can login and submit jobs as usual. Small jobs may run, but larger jobs that require more nodes than available may not run until the reservation is complete.
10/18/17 1:55 PDT10/18/17 8:00 PDTHPSS UserSystem in degraded mode. Archive is in a degraded state. NERSC engineers are working with the vendor to resolve the issue.
10/17/17 9:00 PDT10/17/17 17:00 PDTPostgres Databases on scidb1/2Unavailable. Postgres databases on scidb1/2 aka nerscdb01/02 will be migrating to a new host and unavailable during this period. After this date these dbs should be accessed only from the new nerscdb03.nersc.gov host. Users of these services should have received an email with this info, please contact us with any issues. MySQL services are unaffected but will be migrated at a later date

Show Older Outages: