NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
07/29/14 8:45 PDT07/29/14 11:55 PDTHopperScheduled maintenance. Logins were not available during the maintenance window.
07/25/14 7:50 PDT-PDSF7/31 (all day): pdsfdtn2 will be unavailable due to maintenance. Please use pdsfdtn1 instead.
07/23/14 7:30 PDT07/23/14 12:10 PDTEdisonScheduled maintenance. Logins will not be available during the maintenance.
07/22/14 19:10 PDT07/23/14 7:15 PDTHopperUnavailable. Hopper was unavailable due to a management switch failure.
07/22/14 0:49 PDT07/22/14 1:06 PDTHopperUnavailable. Login access to Hopper and access to scratch filesystems was unavailable
07/16/14 15:33 PDT07/16/14 16:04 PDTGlobal Scratch/global/scratch unavailable.
07/16/14 9:00 PDT07/16/14 11:45 PDTHPSS UserScheduled maintenance. This was previously scheduled but inadvertently deleted.
07/16/14 9:00 PDT07/16/14 11:15 PDTHPSS BackupScheduled maintenance. This was previously scheduled but inadvertently deleted.
07/12/14 10:52 PDT07/12/14 11:15 PDTGlobal ScratchUnavailable. Global Scratch filesystem (/global/scratch2) was unavailable to Hopper, Edison and Carver. The problem has been resolved.
07/11/14 10:30 PDT07/11/14 12:30 PDTNetworkScheduled maintenance. ESnet will be performing maintenance on the fiber carrying NERSC's 100G connection to the Internet. On Friday 07/11 the traffic will be moved back to the original path, which will cause a 15 minute disruption to the 100G link. During the outage, NERSC will use our 10G connection to the Internet, and we expect no loss of connectivity. Until the original path is restored on 07/11, users may notice a small increase in latency and/or a reduction in total available bandwidth available into and out of NERSC.
07/10/14 11:32 PDT-Edisonedisongrid may be inaccessible until 2014-07-10 13:00 PDT.
07/10/14 10:30 PDT07/10/14 12:31 PDTNetworkScheduled maintenance. ESnet will be performing maintenance on the fiber carrying NERSC's 100G connection to the Internet. The maintenance is scheduled for Thursday 07/10 between 10:30AM and 4:00PM, with a 15 minute outage at 10:30AM while Esnet reroutes traffic. During the outage, NERSC will use our 10G connection to the Internet, and we expect no loss of connectivity. Until the original path is restored on 07/11, users may notice a small increase in latency and/or a reduction in total available bandwidth available into and out of NERSC. Update: NERSC successfully failed over to the 10G circuit as of 10:35 PDT. Connectivity via the 100G circuit was reestablished at approximately 12:31 PDT.
07/07/14 10:00 PDT-Edison/scratch1 problems are causing intermittent hangs on login nodes. No estimated for fix available.
07/03/14 11:04 PDT-Edison/scratch1 problems are causing intermittent hangs on login nodes. No estimated time for fix available. Grid Services on Edison Down. No estimated time for fix available.
07/01/14 10:51 PDT07/01/14 10:52 PDTNetworkScheduled maintenance. ESnet is performing maintenance on our border Internet connection today between 0900 and 1600. Networking will be migrating our production traffic to a back-up path to try to minimize any disruptions. There may be up to a 10 minute outage between NERSC and the Internet. Update: The actual outage started at 10:51 and ended at 10:52 PDT.
06/30/14 14:45 PDT06/30/14 15:05 PDTCarverSystem in degraded mode.
06/30/14 9:25 PDT06/30/14 11:20 PDTGlobal ScratchUnavailable.
06/28/14 9:00 PDT07/01/14 17:30 PDTProjectSystem in degraded mode. Global PROJECT is at 80% of peak aggregate performance due to hardware replacement.
06/27/14 10:22 PDT-Edison6 login nodes are currently available. The other 6 login nodes should also returned to service on 10:10PDT, 7/2.
06/25/14 23:02 PDT-Edison6 login nodes are available. Update: /scratch3 is available as of 10:40 PDT, 06/26/2014.
06/25/14 19:02 PDT06/25/14 19:42 PDTProjectB/projectb unavailable. /projectb degraded due to Nexsan16 hung.
06/25/14 9:00 PDT06/25/14 10:55 PDTHPSS UserScheduled maintenance.
06/25/14 7:00 PDT06/25/14 22:45 PDTEdisonScheduled maintenance.
06/24/14 7:45 PDT06/24/14 8:15 PDTEdisonSystem in degraded mode. Edison system and /scratch1 was in degraded mode.
06/18/14 9:00 PDT06/18/14 14:04 PDTEdisonScheduled maintenance. Dedicated Testing. Please note that this maintenance has been extended until 14:00 PDT. However, logins are now available.
06/18/14 9:00 PDT06/18/14 13:15 PDTHPSS UserScheduled maintenance.
06/16/14 13:00 PDT06/16/14 14:25 PDTNetworkScheduled maintenance. Networking will be upgrading software on the NERSC load balancers on Monday 06/16 from 13:00 - 15:00 to address a bug which is causing instability in the load balancer configuration. The load balancers which provide ssh access to the large compute platforms as well as web access for NERSC and JGI may be affected. We anticipate the maintenance to have no impact on services.
06/13/14 18:00 PDT06/14/14 14:00 PDTGlobal DNASystem in degraded mode. System in degraded mode due to hardware replacement. Global DNA in 75% of peak aggregate performance during the maintenance.
06/11/14 10:00 PDT06/11/14 11:00 PDTMongoDBScheduled maintenance. Mongo dev node (mongodb03.nersc.gov) will be offline for network maintenance.
06/11/14 9:00 PDT06/11/14 11:00 PDTHPSS BackupScheduled maintenance. Maintenance rescheduled to 6/11.
06/11/14 9:00 PDT06/11/14 9:50 PDTHPSS UserScheduled maintenance.
06/10/14 11:02 PDT-Genepool/global/dna is back up
06/10/14 11:01 PDT-Genepool/global/dna is now back up.
06/10/14 10:46 PDT-Genepool/global/dna is degraded. Next update by 12:00 PDT 2014-06-10.
06/05/14 23:28 PDT-Edison(06/04/14 22:24 PDT) - /scratch3 file system will be down until 06/05/14 18:00PDT.
06/04/14 22:24 PDT-Edison /scratch3 file system will be down until 06/05/14 18:00PDT. /scratch3 file system is in degraded mode on the interactive nodes.
06/04/14 7:30 PDT06/04/14 22:19 PDTEdisonScheduled maintenance. Dedicated testing from 20:30 - 22:30. NOTE: /scratch3 file system will be down until 06/05/14 18:00PDT.
06/02/14 10:45 PDT06/24/14 10:00 PDTGlobal ScratchSystem in degraded mode. System in 60 percent of peak aggregate performance due to rolling firmware upgrade.
05/29/14 16:22 PDT-EdisonWe are debugging Intel license issues for Edison now. Users may experience intermittent interruptions.
05/28/14 23:15 PDT05/28/14 2:30 PDTProjectBUnavailable.
05/27/14 23:00 PDT05/28/14 2:30 PDTProjectBUnavailable.
05/27/14 10:00 PDT05/27/14 11:07 PDTNX ServicesScheduled maintenance.
05/27/14 8:37 PDT-EdisonEngineers have resolved issues with the /scratch2 filesystem starting at noon on Monday May 26 and lasting until 9am Tuesday May 27. Jobs and compilations for users with directories on this filesystem were affected. Engineers are continuing to investigate issues with intel compilers.
05/23/14 9:57 PDT-EdisonProblems with Intel compiler licenses; engineers are investigating.
05/21/14 9:00 PDT05/21/14 12:00 PDTHPSS BackupScheduled maintenance.
05/13/14 13:00 PDT05/13/14 13:21 PDTLicense Server: DMV1/DMV2Scheduled maintenance. OS update.
05/13/14 11:00 PDT-CarverNotes: (05/13/14 10:47 PDT) - Scheduled License Servers work from 13:00-14:00 today. Allinea tools (DDT, MAP) and HMPP may be unavailable briefly during this period.
05/13/14 10:47 PDT-EdisonScheduled License Servers work from 13:00-14:00 today. Most compilers (except GNU), Allinea tools (DDT, MAP) and Perftools may be unavailable briefly during this period.
05/13/14 10:47 PDT-HopperScheduled License Servers work from 13:00-14:00 today. Most compilers (except GNU), Allinea tools (DDT, MAP) and Perftools may be unavailable briefly during this period.
05/13/14 10:00 PDT05/13/14 13:38 PDTNIMScheduled maintenance. NIM database update.
05/13/14 10:00 PDT05/13/14 15:59 PDTJESUPScheduled maintenance. JESUP testbed will be unavailable. This will also affect the RStudio Services.
05/13/14 9:00 PDT05/13/14 11:00 PDTProjectB/projectb scheduled maintenance.
05/13/14 8:00 PDT05/13/14 19:00 PDTPDSFScheduled maintenance. PDSF was unavailable for necessary GPFS, kernel, and network upgrades.
05/13/14 7:00 PDT05/13/14 20:45 PDTGenepoolScheduled maintenance. The genepool computational system, login nodes, and gpint nodes will not be available. NOTE: gpweb GPFS is now available at 12:00PT noon.
05/13/14 7:00 PDT05/13/14 20:00 PDTBabbageScheduled maintenance.
04/29/14 9:00 PDT04/29/14 11:40 PDTDatawarehouse oracle (AKA dwprd1)Scheduled maintenance. During the maintenance period, all connections to the database server (farina) will disconnect, and some services may need to be restarted to get the new network address once farina is available again.
04/28/14 13:48 PDT-ProjectBFilesystem slowness caused by restriping of data.
04/27/14 5:46 PDT04/27/14 10:55 PDTProjectBSystem in degraded mode. PROJECTB (/projectb) was in a degraded state due to failing storage controller affecting Genepool nodes causing long I/O waiters. The problem has been resolved and no further outage is expected.
04/23/14 9:00 PDT04/23/14 13:20 PDTHPSS BackupScheduled maintenance.
04/21/14 14:00 PDT04/21/14 16:00 PDTLDAPScheduled maintenance. Updates made through https://nim.nersc.gov may be delayed.
04/21/14 13:20 PDT-PDSFApril 23 11:00 - 12:00 PDT: Rolling reboot of login nodes for software upgrade.
04/21/14 11:00 PDT04/21/14 12:07 PDTnerscca2Scheduled maintenance. NERSC Certificate Services nerscca2.nersc.gov was down for system upgrades. Globus Online / NEWT and Grid Services may be unavailable during this period.
04/16/14 14:50 PDT04/16/14 15:20 PDTCarverSystem in degraded mode. Intermittent login problems to Carver. Engineers are actively investigating.
04/16/14 9:00 PDT04/16/14 11:16 PDTHPSS UserScheduled maintenance.
04/15/14 14:00 PDT04/15/14 16:00 PDTMongoDBScheduled maintenance. MongoDB service will be unavailable intermittently from 2-4pm today for security patches and service upgrade.
04/14/14 14:00 PDT04/14/14 15:00 PDTnerscca1Scheduled maintenance. NERSC Certificate Services nerscca1.nersc.gov will be be down for system upgrades. Globus Online / NEWT and Grid Services may be unavailable during this period.
04/14/14 9:00 PDT04/14/14 11:19 PDTEdisonScheduled maintenance.
04/09/14 13:35 PDT04/09/14 15:40 PDTgenepool12Unavailable. Node genepool12 was unavailable. Some processes terminated prior to recovery.
04/09/14 10:00 PDT04/09/14 10:20 PDTNIMScheduled maintenance.
04/09/14 9:00 PDT04/09/14 10:45 PDTHPSS UserScheduled maintenance.
04/08/14 14:00 PDT04/08/14 17:00 PDTmailmanUnavailable. Mailman server (mailman.nersc.gov) will be down for maintenance.
04/07/14 6:42 PDT04/07/14 8:35 PDTGlobal HomesSystem in degraded mode. Global Homes (/global/homes) filesystem was experiencing problems resulting to intermittent logins to Hopper, Edison, Carver, and Genepool. Problem has been resolved.
04/03/14 8:00 PDT04/03/14 14:10 PDTHopperScheduled maintenance.
03/26/14 7:30 PDT03/26/14 16:00 PDTEdisonScheduled maintenance.
03/25/14 8:00 PDT03/25/14 18:20 PDTCarverScheduled maintenance.
03/22/14 21:51 PDT-ProjectFrom 20:18 PDT, 03/22/2014, until 00:30 PDT, 03/23/2014, /project was unavailable due to an NGF problem. The problem has been fixed.
03/22/14 20:18 PDT03/23/14 0:30 PDTProject/project unavailable.
03/22/14 3:15 PDT03/22/14 10:30 PDTLDAPSystem in degraded mode. LDAP was having intermittent authentication login issues to NERSC resources since 03:15PT, 03/22. Engineers have resolved the issues.
03/19/14 12:20 PDT03/19/14 12:50 PDTLicense ServerScheduled maintenance. DDT, IDL, MAP, MATLAB, Mathematica and TotalView will be unavailable.
03/19/14 9:00 PDT03/19/14 11:26 PDTHPSS BackupScheduled maintenance.

Show Older Outages: