NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
12/09/16 16:49 PST12/09/16 22:10 PSTScience Gateway ServicesUnavailable. SGN01 unavailable - Portals on portals.nersc.gov are currently down. Investigation ongoing. This includes qcd.nersc.gov, sgn01, spot.nersc.gov, unwise.nersc.gov, cxidb.org
12/09/16 9:40 PST12/09/16 10:32 PSTCoriUnavailable. Engineers are actively investigating filesystem problems which are causing logins and various operations to hang.
12/09/16 9:40 PST12/09/16 10:32 PSTEdisonSystem in degraded mode. The "cscratch1" filesystem is currently unavailable.
12/08/16 7:40 PST12/08/16 16:03 PSTCoriUnavailable. Cori is unavailable while engineers are working to resolve an issue.
12/05/16 14:00 PST12/05/16 14:38 PSTNX ServicesScheduled maintenance. The NX server will be restarted to implement needed configuration changes. All connected sessions will be terminated.
12/02/16 17:32 PST12/02/16 19:22 PSTCoriSystem in degraded mode. High speed network services disrupted. No new jobs will start.
11/30/16 19:26 PST11/30/16 21:18 PSTCoriSystem in degraded mode. Cori is experiencing network failures. Engineers are investigating the issue. Job scheduling has been paused, however new jobs can be submitted.
11/28/16 9:00 PST11/28/16 16:54 PSTMongoDB ServicesScheduled maintenance. Services on mongodb0* will be unavailable or at-risk for an upgrade of the MongoDB version
11/27/16 18:42 PST11/27/16 22:07 PSTCoriSystem in degraded mode. /cscratch1 is currently unavailable. Engineers are investigating the issue.
11/27/16 18:42 PST11/27/16 22:07 PSTEdisonSystem in degraded mode. /cscratch1 is currently unavailable. Engineers are investigating the issue.
11/22/16 4:25 PST11/22/16 9:53 PSTHPSS BackupSystem in degraded mode.
11/21/16 15:50 PST11/21/16 17:28 PSTPDSFSystem in degraded mode. PDSF jobs have been paused. Network issues causing /global filesystem to not be available. Engineers are currently investigating the issue.
11/21/16 15:39 PST11/21/16 17:36 PSTGenepoolUnavailable. Genepool jobs have been paused. We are investigating.
11/21/16 15:39 PST11/21/16 17:15 PSTCoriSystem in degraded mode. Network issue causing /global filesystem issues. Slurm partitions are currently marked down. Engineers are investigating.
11/21/16 15:39 PST11/21/16 17:25 PSTEdisonSystem in degraded mode. Network issues causing /global filesystem to be unavailable. Slurm partitions currently marked down. Engineers is currently working on the issue.
11/21/16 15:30 PST11/21/16 16:40 PSTNetworkUnavailable. Network issue causing /global filesystems to be unavailable on the systems. Engineers are aware and are working on the issue.
11/21/16 15:30 PST11/21/16 17:27 PSTScience GatewaysSystem in degraded mode. Network issue causing /global filesystems to be unavailable on the systems. Engineers are aware and are working on the issue.
11/21/16 10:32 PST11/21/16 14:05 PSTEdisonSystem in degraded mode. Several compute nodes are missing /global/cscratch1. Nodes will be drained and rebooted. Engineers are currently working on this issue.
11/17/16 9:00 PST11/17/16 11:30 PSTscidb2.nersc.gov (Science Database)Unavailable. Emergency maintenance.
11/17/16 7:00 PST11/18/16 17:05 PSTCori Scratch FilesystemScheduled maintenance. Jobs that require a CSCRATCH license will not run during this time due to filesystem maintenance
11/17/16 7:00 PST11/19/16 14:39 PSTCoriScheduled maintenance. 11:40 PST - Logins are now available, estimated system uptime is 15:00 PST.
11/16/16 9:00 PST11/16/16 12:03 PSTHPSS UserScheduled maintenance.
11/12/16 9:55 PST11/12/16 10:00 PSTCoriLogins available, batch jobs not running. Some jobs may have failed between 09:55 and 10:00 PST due to internal high-speed-network issues.
11/10/16 7:42 PST11/10/16 8:15 PSTCoriLogins unavailable, batch jobs running. Logins are hanging due to filesystem issues. Engineers are actively working to resolve this issue.
11/08/16 22:42 PST11/08/16 22:53 PSTCoriSystem in degraded mode. /cscratch1 is not available. Engineers are investigating the issue
11/08/16 19:30 PST11/08/16 22:42 PSTCoriUnavailable. Cori is unavailable due to filesystem errors. Engineers are investigating the issue.
11/08/16 19:30 PST11/08/16 22:53 PSTEdisonSystem in degraded mode. /cscratch1 is unavailable. Engineers are investigating the issue.
11/08/16 13:00 PST11/08/16 13:40 PSTNX ServicesScheduled maintenance. Quarterly Maintenance.
11/08/16 12:10 PST11/08/16 12:50 PSTCoriSystem in degraded mode. Cori is degraded. /cscratch1 not available.
11/08/16 12:10 PST11/08/16 12:50 PSTEdisonSystem in degraded mode. Edison is degraded. /cscratch1 not available
11/08/16 11:00 PST11/08/16 12:13 PSTLDAPScheduled maintenance. Quarterly Maintenance
11/08/16 9:00 PST11/08/16 13:13 PSTISGScheduled maintenance. Quarterly Maintenance: Regular updates will be applied and services may be briefly inaccessible during system reboots. Affected services include Science Gateways, Genepool Database and Web Servers, the NERSC web site, License managers, MyProxy, NIM, and ftp/upload.
11/08/16 8:00 PST11/08/16 10:47 PSTEdisonScheduled maintenance. All compute nodes will be unavailable for network maintenance. Users will still be able to log in and submit jobs, but jobs will not start until after this window.
11/08/16 8:00 PST11/08/16 10:52 PSTCoriScheduled maintenance. All compute nodes will be unavailable for network maintenance. Users will still be able to log in and submit jobs, but jobs will not start until after this window.
11/08/16 8:00 PST11/08/16 10:44 PSTNetworkScheduled maintenance. NERSC Network maintenance.
11/08/16 7:00 PST11/08/16 11:37 PSTGenepoolScheduled maintenance. Quarterly Maintenance.
11/08/16 7:00 PST11/08/16 14:22 PSTData Transfer NodesScheduled maintenance. Quarterly Maintenance.
11/08/16 7:00 PST11/08/16 11:15 PSTSeqFSScheduled maintenance. Quarterly Maintenance
11/07/16 10:57 PST11/07/16 12:30 PSTmy.nersc.govScheduled maintenance. Possible intermittent disruption.
11/05/16 8:00 PDT11/06/16 19:00 PSTCoriScheduled maintenance. Cori hardware scheduled maintenance. Login nodes will be available.
11/03/16 10:30 PDT11/03/16 11:15 PDTCoriSystem in degraded mode. The batch system is currently down since 10:30. Engineers are currently investigating.
11/02/16 15:15 PDT11/02/16 16:43 PDTCoriSystem in degraded mode. Job scheduling is paused.
11/02/16 8:00 PDT11/02/16 16:02 PDTEdisonScheduled maintenance. Edison will be undergoing emergency maintenance. Logins will be available during this time.
10/31/16 17:26 PDT-CoriPlease see http://www.nersc.gov/users/computational-systems/cori/updates-and-status/ for current Cori issues and status.
10/31/16 1:25 PDT10/31/16 15:43 PDTEdisonSystem in degraded mode. /cscratch1 is curently unavailabe on Edison, jobs that use that filesystem will not run. Engineers are currently working to resolve the issue.
10/27/16 12:00 PDT10/27/16 15:39 PDTScience Gateway ServicesSystem in degraded mode.
10/27/16 8:00 PDT10/27/16 8:45 PDTEdisonScheduled maintenance. Logins will be available, and jobs may be submitted, but jobs will not run during this window.
10/27/16 8:00 PDT10/28/16 18:28 PDT/global/cscratch1Unavailable. The "cscratch1" filesystem will be unavailable during this window. Jobs using this filesystem may be submitted, but they will be held. Jobs using this filesystem at the beginning of this window will be terminated.
10/26/16 9:00 PDT10/26/16 12:50 PDTHPSS BackupScheduled maintenance.
10/25/16 17:33 PDT-EdisonAccess to collaboration accounts, NEWT, and globus transfers to Edison are currently unavailable. Engineers are investigating the problem.
10/25/16 10:14 PDT10/25/16 14:58 PDTEdisonUnavailable. Filesystem access problems are causing logins and jobs to hang. Engineers are actively working to resolve this issue.
10/22/16 6:00 PDT10/22/16 12:51 PDTEdisonScheduled maintenance. Power maintenance. Users will still be able to log in and submit jobs.
10/22/16 5:00 PDT10/22/16 14:30 PDTNX ServicesScheduled maintenance. NX Services will be unavailable due to a planned power maintenance.
10/22/16 5:00 PDT10/22/16 15:49 PDTPDSFScheduled maintenance. PDSF will be unavailable during the planned power maintenance. Logins will be unavailable and no jobs will be running.
10/22/16 5:00 PDT10/22/16 16:00 PDTGenepoolScheduled maintenance. Genepool will be down during the planned power maintenance. Logins will be unavailable and no jobs will be running.
10/22/16 5:00 PDT10/22/16 14:30 PDTMongoDB ServersScheduled maintenance. MongoDB Servers (mongodb01-04) will be unavailable during planned power maintenance.
10/22/16 5:00 PDT10/22/16 14:30 PDTWald SciDB clusterScheduled maintenance. Wald SciDB cluster will be unavailable during planned power maintenance.
10/22/16 5:00 PDT10/22/16 15:15 PDTMatgenScheduled maintenance. Matgen will be unavailable for planned power maintenance. Logins will be unavailable and no jobs will be running.
10/17/16 13:55 PDT10/17/16 14:05 PDTEdisonSystem in degraded mode. Batch system issues - new jobs will NOT run but existing jobs will continue to run. Users may experience issues querying the batch system.
10/12/16 8:00 PDT10/12/16 9:29 PDTNetworkScheduled maintenance. Engineers will be performing maintenance on our border router. Users will experience reduced bandwidth, however there should be no interruption.
10/09/16 9:55 PDT10/09/16 11:00 PDTScience Gateway ServicesSystem in degraded mode. Several Science Gateway sites may be unavailable. Engineers are actively working to resolve this issue.
10/05/16 11:00 PDT10/05/16 11:30 PDTNIMScheduled maintenance.

Show Older Outages: