NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
04/17/15 15:56 PDT-EdisonDedicated Science Run on Wednesday, 04/22/2015, 16:00-20:00 PDT. Logins will be available and the debug queue will be open, so users can run debug jobs during the reservation.
04/14/15 10:05 PDT04/14/15 11:00 PDTHopperUnavailable. The system Queues were stopped at 10:05 and were turned back on at 11:15. /scratch2 unavailable.
04/14/15 8:00 PDT04/14/15 9:00 PDTNIMScheduled maintenance.
04/11/15 8:00 PDT04/11/15 12:00 PDTEdisonLogins available, batch jobs not running. Dedicated science runs.
04/09/15 22:39 PDT04/09/15 22:56 PDTHPSS UserUnavailable. hpss_user (Archive) was down for unscheduled maintenance.
04/01/15 13:35 PDT-EdisonEdison /scratch1 filesystem appears to be slow. The issue is being looked into. Updates will be provided when available.
03/30/15 8:00 PDT03/30/15 9:47 PDTScience Gateway ServicesUnavailable.
03/26/15 9:02 PDT03/26/15 9:24 PDTData Transfer NodesSystem in degraded mode. The DTN02 had login issues.
03/25/15 14:45 PDT03/25/15 16:20 PDTEdisonUnavailable. Dedicated testing.
03/25/15 9:00 PDT03/25/15 12:30 PDTHPSS UserScheduled maintenance.
03/25/15 8:00 PDT03/25/15 14:45 PDTEdisonScheduled maintenance.
03/18/15 9:00 PDT03/18/15 12:50 PDTHPSS UserScheduled maintenance.
03/16/15 8:30 PDT03/16/15 9:15 PDTHopperSystem in degraded mode. Lustre/scratch2 unavailable
03/15/15 9:00 PDT03/15/15 10:20 PDTProjectBSystem in degraded mode. PROJECTB was in degraded mode with some lost disks.
03/12/15 17:00 PDT03/12/15 20:40 PDTProjectBSystem in degraded mode. /projectb was degraded for Genepool nodes.
03/11/15 10:00 PDT03/11/15 17:25 PDTData Transfer NodesScheduled maintenance. DTN03 and DTN04 will be unavailable; however, DTN01 and DTN02 will remain available.
03/11/15 10:00 PDT03/11/15 12:22 PDTMongoDB01Scheduled maintenance. MongoDB01 read/write operations will be locked during this maintenance.
03/11/15 9:00 PDT03/11/15 13:00 PDTHPSS BackupScheduled maintenance.
03/11/15 8:00 PDT03/11/15 22:45 PDTHopperScheduled maintenance. Hopper OS Upgrade to CLE 5.2 UP02.
03/06/15 17:56 PST03/06/15 20:06 PSTHPSS UserUnavailable. Unscheduled Maintenance.
03/05/15 17:28 PST-EdisonDedicated Large Scale Science Run on 3/6/15 08:00-16:30PT. User jobs will not be running but logins will be available.
03/04/15 15:00 PST03/04/15 17:00 PSTNIMScheduled maintenance.
03/03/15 13:36 PST-PDSF3/4/15 10am - 12pm pdsfdtn2 (PDSF newt and globus endpoint) will be unavailable.
03/01/15 9:39 PST03/01/15 10:55 PSTEdisonSystem in degraded mode. Hardware disk issues with /scratch1
02/28/15 8:39 PST02/28/15 9:22 PSTCarverSystem in degraded mode. Batch system degraded, logins are available.
02/27/15 13:28 PST02/27/15 16:17 PSTProjectBUnavailable. Genepool queues were being drained to avoid job failures due to projectb outage. Queues are reenabled.
02/25/15 13:50 PST02/25/15 14:10 PSTEdisonSystem in degraded mode.
02/25/15 10:00 PST02/25/15 16:30 PSTData Transfer NodesScheduled maintenance.
02/25/15 9:00 PST02/25/15 11:00 PSTHPSS UserScheduled maintenance.
02/24/15 10:52 PST02/24/15 12:26 PSTDNAUnavailable. The NFS mount on /global/dna was down since 10:52 AM and came back up at 12:26 PT. Engineers have resolved the issue.
02/24/15 8:00 PST02/24/15 9:11 PSTNIMScheduled maintenance.
02/23/15 17:46 PST02/23/15 18:17 PSTDNASystem in degraded mode. NFS export was not available.
02/22/15 14:23 PST02/22/15 15:30 PSTCarverSystem in degraded mode. Carver batch was down.
02/21/15 11:50 PST02/21/15 12:32 PSTHPSS BackupUnavailable.
02/20/15 15:42 PST-CarverNode
02/19/15 8:44 PST-CarverNode "carvergrid" maintenance extended; status update by noon, Fri Feb 20. Use node "carversl6" for grid services until carvergrid returns to production.
02/18/15 10:00 PST02/18/15 12:00 PSTHPSS BackupScheduled maintenance.
02/18/15 8:00 PST02/18/15 17:06 PSTHopperScheduled maintenance. ALL DATA on HOPPER /scratch (your $SCRATCH) WILL BE DELETED
02/14/15 12:55 PST02/14/15 13:36 PSTProjectUnavailable. /global/project was unavailable. Jobs may have been affected.
02/08/15 8:51 PST-EdisonEdison degraded - /scratch3 may be unavailable. Engineers are working on the issue.
02/08/15 3:14 PST02/08/15 9:25 PSTEdisonSystem in degraded mode. Edison degraded - /scratch3 may be unavailable.
02/06/15 12:20 PST-EdisonEdison is experiencing a slow I/O performance with the global file systems. We are aware of the issue and are actively investigating the problem.
02/04/15 17:23 PST-Edison/scratch3 is currently unavailable. /scratch3 recovery efforts ongoing with no data loss expected. Next status update will be by 16:00 PST, 02/05.
02/03/15 23:20 PST-PDSFSome older compute nodes are unavailable. There will be an update by 5 pm 2/4/15.
02/03/15 21:17 PST-Edison
02/03/15 7:00 PST02/03/15 23:15 PSTNERSC Quarterly MaintenanceScheduled maintenance.
02/03/15 7:00 PST02/03/15 19:10 PSTScience Gateway ServicesScheduled maintenance.
02/03/15 7:00 PST02/03/15 23:15 PSTCarverScheduled maintenance.
02/03/15 7:00 PST02/03/15 21:10 PSTEdisonScheduled maintenance.
02/03/15 7:00 PST02/03/15 21:15 PSTGenepoolScheduled maintenance.
02/03/15 7:00 PST02/03/15 19:40 PSTHopperScheduled maintenance.
02/03/15 7:00 PST02/03/15 18:45 PSTPDSFScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTDNAScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTGlobal CommonScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTGlobal HomesScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTGlobal ScratchScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTProjectScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTProjectBScheduled maintenance.
02/03/15 7:00 PST02/03/15 16:20 PSTHPSS BackupScheduled maintenance.
02/03/15 7:00 PST02/03/15 16:20 PSTHPSS UserScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:35 PSTData Transfer NodesScheduled maintenance.
01/28/15 9:00 PST01/28/15 13:01 PSTHPSS BackupScheduled maintenance.
01/24/15 16:30 PST01/24/15 17:10 PSTdtn03Unavailable. Dtn03 was rebooted to restore ssh access.
01/23/15 10:41 PST-GenepoolWe are experiencing some Genepool expels due to hiccups with /project. Your jobs might be affected if you are using this file system.
01/23/15 10:28 PST-Genepool/project filesystem was experiencing NFS stale handle since 09:50PT. Our engineers have found the problem and the issue has been resolved.
01/23/15 10:14 PST-Project/project filesystem was experiencing NFS stale handle. Our engineers have found the problem and the issue has been resolved.
01/23/15 10:14 PST-Hopper/project filesystem was experiencing NFS stale handle since 09:50PT. Our engineers have found the problem and the issue has been resolved.
01/23/15 10:14 PST-Edison/project filesystem was experiencing NFS stale handle since 09:50PT. Our engineers have found the problem and the issue has been resolved.
01/23/15 9:47 PST01/23/15 10:30 PSTProjectUnavailable. /project was unavailable on all systems due to system hardware issue. The problem has been identified and resolved.
01/21/15 9:00 PST01/21/15 13:00 PSTHPSS BackupScheduled maintenance.
01/20/15 9:07 PST-PDSF1/20/14 6:00 - 6:30 am. Several nodes had issues due to a power event. About 100 jobs were lost on these nodes.
01/20/15 8:00 PST01/20/15 10:00 PSTNIMScheduled maintenance.
01/20/15 7:28 PST-HopperQueues were stopped from 06:20 - 06:50 PST due to a power event. Jobs did not stop running.
01/13/15 12:50 PST-PDSF1/14/2014 10:00am - 10:00pm pdsfdtn2 unavailable
01/13/15 7:30 PST01/16/15 13:15 PSTEdisonScheduled maintenance.
01/07/15 13:35 PST02/07/15 15:52 PSTProjectSystem in degraded mode. Reads may be impacted to some portions of the file system. Writes are unaffected.
01/07/15 13:35 PST01/07/15 15:59 PSTScience Gateway ServicesUnavailable. Due to /project in degraded state.
01/07/15 9:00 PST01/07/15 11:00 PSTHPSS BackupScheduled maintenance.
01/06/15 10:00 PST-Edisonedisongrid (i.e. edison01) will be down for maintenance on 01/07/2015 from 08:00 - 12:00 PST.
01/05/15 8:00 PST01/05/15 8:49 PSTNIMScheduled maintenance.

Show Older Outages: