NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
07/29/15 14:07 PDT07/29/15 16:10 PDTGlobusUnavailable.
07/29/15 9:00 PDT07/29/15 16:00 PDTBabbageScheduled maintenance.
07/29/15 8:00 PDT07/29/15 18:10 PDTEdisonScheduled maintenance.
07/25/15 16:30 PDT07/25/15 18:35 PDTHopperUnavailable. The Hopper /scratch and /scratch2 files systems were down.
07/21/15 14:04 PDT-DNAAs part of scheduled maintenance Engineers will be rebooting controllers on a storage array for DNA filesystem. There is a risk of hangs or degraded DNA filesystem.
07/21/15 14:00 PDT07/21/15 14:24 PDTDNAScheduled maintenance. /global/dna maintenance for hardware replacement.
07/19/15 9:02 PDT07/19/15 10:30 PDTHopperUnavailable.
07/19/15 5:00 PDT07/19/15 9:02 PDTHopperLogins available, batch jobs not running. Engineers are investigating a problem with /scratch & /scratch2, jobs may be impacted.
07/18/15 20:10 PDT07/18/15 22:05 PDTHopperUnavailable. Logins were unavailable. Engineers fixed a problem with the /scratch filesystem.
07/15/15 8:00 PDT07/15/15 15:30 PDTHopperScheduled maintenance.
07/14/15 8:00 PDT07/14/15 9:00 PDTNIMScheduled maintenance. Minor modifications to NIM interface. No outage expected.
07/13/15 11:50 PDT07/15/15 17:00 PDTMongodb03Unavailable.
07/10/15 10:00 PDT07/10/15 13:30 PDTEliza11 File SystemUnavailable.
07/09/15 18:15 PDT07/09/15 18:50 PDTEliza11 file systemSystem in degraded mode. Eliza11 File System degraded due to hardware issue.
07/08/15 8:24 PDT-EdisonEdison may experience job submittal issues. Engineers are working on the issue.
07/06/15 8:30 PDT07/06/15 13:45 PDTEdisonSystem in degraded mode. Edison degraded. Batch system issues being worked on. Job query, submission may be intermittently affected.
07/05/15 11:34 PDT-PDSFEliza11 is having H/W issues.
07/05/15 11:25 PDT07/05/15 12:28 PDTPDSFSystem in degraded mode. PDSF eliza11 is having H/W issues.
07/05/15 11:19 PDT07/05/15 12:30 PDTEdisonSystem in degraded mode. Edison degraded. Batch system issues being worked on. Job query, submission may be intermittently affected.
07/04/15 18:06 PDT07/04/15 18:13 PDTGlobal HomesUnavailable. The NGF "/global/homes" filesystem was unavailable.
07/04/15 18:05 PDT07/04/15 23:30 PDTCarverUnavailable. The `/global/common' filesystem was down. This Problem has been resolved.
07/04/15 18:05 PDT07/04/15 23:10 PDTEdisonUnavailable. The `/global/common' filesystem was down. This problem has been resolved.
07/04/15 18:05 PDT07/04/15 23:45 PDTGenepoolUnavailable. The `/global/common' filesystem was down. This problem has been resolved.
07/04/15 18:05 PDT07/04/15 23:10 PDTHopperUnavailable. The `/global/common' filesystem was down. This problem has been resolved.
07/04/15 18:05 PDT07/04/15 23:45 PDTPDSFUnavailable. The `/global/common' filesystem was down. This problem has been resolved.
07/04/15 18:05 PDT07/04/15 22:36 PDTGlobal CommonUnavailable. NGF `/global/common' filesystem was down.
07/04/15 18:05 PDT07/04/15 18:05 PDTScience Gateway ServicesUnavailable. SGN services were unavailable due to NGF /global/common outage.
07/04/15 18:05 PDT07/04/15 22:36 PDTData Transfer NodesUnavailable. Data Transfer Node services were unavailable due to NGF /global/common outage.
07/02/15 19:42 PDT07/02/15 23:05 PDTHopperSystem in degraded mode. Access to NGF filesystems on compute nodes may have been delayed due to a hardware problem.
07/01/15 9:00 PDT07/01/15 12:48 PDTHPSS UserScheduled maintenance.
06/26/15 7:30 PDT06/26/15 8:35 PDTNERSC WebsiteUnavailable.
06/24/15 21:52 PDT06/24/15 23:08 PDTHopperSystem in degraded mode.
06/22/15 17:00 PDT07/22/15 21:30 PDTEdisonScheduled maintenance. Dedicated Testing. Login nodes will be available for users, but no user jobs will be running during the reservation.
06/17/15 9:00 PDT06/17/15 11:27 PDTHPSS BackupScheduled maintenance.
06/16/15 10:00 PDT06/16/15 12:00 PDTNERSC WebsiteScheduled maintenance. One of the web server will be removed from the load balancer which willl affect the redundancy to NERSC website (http://nersc.gov); however, the site is expected to remain up and functional during the time
06/12/15 2:35 PDT06/12/15 3:00 PDTCarverUnavailable. Unscheduled Maintenance.
06/05/15 13:45 PDT06/05/15 14:20 PDTHopperSystem in degraded mode. Logins and jobs are hanging due to problems with the Lustre filesystem.
06/04/15 8:30 PDT06/04/15 12:10 PDTHopperScheduled maintenance. Logins are available.
06/03/15 9:00 PDT06/03/15 9:26 PDTNX ServicesScheduled maintenance. The NX services will be unavailable.
06/03/15 5:00 PDT06/03/15 13:15 PDTGrid Services, MyNERSC and Science GatewSystem in degraded mode. Engineers are investigating reports of failed logins to multiple grid based services. Globus, MyNERSC and Science Gateways are affected.
05/28/15 17:05 PDT05/28/15 20:50 PDTHopperSystem in degraded mode. Users can still log on and jobs are still running, Intermittent slow response.
05/27/15 9:00 PDT05/27/15 10:18 PDTHPSS UserScheduled maintenance.
05/27/15 9:00 PDT05/27/15 10:18 PDTHPSS BackupScheduled maintenance.
05/27/15 7:00 PDT05/27/15 18:00 PDTEdisonScheduled maintenance. The Login nodes will be unavailable.
05/21/15 23:11 PDT05/22/15 10:30 PDTportal-auth web servicesUnavailable. portal-auth.nersc.gov (sgn02) is currently unavailable.
05/20/15 8:00 PDT05/20/15 23:10 PDTEdisonScheduled maintenance.
05/19/15 9:25 PDT05/19/15 15:30 PDTLDAPUnavailable. Logins to most NERSC services are unavailable or unreliable. Engineers are actively investigating this issue. UPDATE: logins are currently working, but may be unreliable until the issue is fully resolved.
05/17/15 21:22 PDT05/18/15 12:30 PDTNIMUnavailable. Our engineers are currently working to resolve the issue.
05/14/15 11:56 PDT05/14/15 12:06 PDTProjectSystem in degraded mode. /project hang for writing.
05/14/15 11:00 PDT05/14/15 13:17 PDThpweb1Scheduled maintenance. We don’t expect any web site outage as a result of this since the web sites for nersc.gov, crd.lbl.gov and cs.lbl.gov will still be served by hpweb2.nersc.gov .
05/13/15 10:15 PDT05/13/15 14:09 PDTEdisonUnavailable. Engineers are working in resolving HSN (High Speed Network) issue.
05/13/15 10:00 PDT05/13/15 11:00 PDTProjectSystem in degraded mode. File creation and deletion was hanging during that time. There was no issue for reads
05/13/15 9:00 PDT05/13/15 11:30 PDTHPSS BackupScheduled maintenance.
05/13/15 9:00 PDT05/13/15 11:30 PDTHPSS UserScheduled maintenance.
05/12/15 12:00 PDT05/12/15 18:45 PDTNERSC MailmanScheduled maintenance. The maintenance on website https://mailman.nersc.gov/ has been completed.
05/12/15 8:00 PDT05/12/15 15:00 PDTGenepoolScheduled maintenance. All maintenance work on genepool, gpweb and gpdb hosts has been completed.
05/12/15 8:00 PDT05/12/15 8:52 PDTNIMScheduled maintenance. NIM interface was down.
05/12/15 7:00 PDT05/12/15 15:00 PDTScience Gateway ServicesScheduled maintenance. All maintenance work on the following hosts and services has been completed: 07:00-08:00 - Some Science Gateways that require authentication may be unavailable during this period. 13:00-15:00 - NERSC Certificate Services will be down for system upgrades. Globus Online/NEWT and Grid Services may be unavailable during this period. 07:00-15:00 - MySQL and Postgres DBs will be unavailable due OS upgrades on host machines scidb1/scidb2.
05/11/15 8:00 PDT05/11/15 17:00 PDTMATLAB License ServerScheduled maintenance.
05/11/15 6:40 PDT05/11/15 8:00 PDTHopperUnavailable. /scratch2 was not responding. Engineers resolved the issue.
05/07/15 6:00 PDT05/08/15 18:00 PDTEdisonSystem in degraded mode. /scratch1 file operations were slower than usual. Engineers resolved the issue.
05/05/15 12:06 PDT05/05/15 12:50 PDTNERSC WebsiteUnavailable. Intermittent access problems. Engineers are working to resolve this issue.
04/30/15 13:00 PDT04/30/15 14:36 PDTScience Gateway ServicesScheduled maintenance. Some science gateway services (openmsi, ptf) will be unavailable due to an urgent GPFS upgrade.

Show Older Outages: