NERSCPowering Scientific Discovery Since 1974

Outage Log

This table lists events reported, ongoing, and resolved on NERSC systems. It is a historical record and may not be updated while a system event is in progress.

Event Date/TimeUp Date/TimeSystemComment
03/01/15 9:39 PST03/01/15 10:55 PSTEdisonSystem in degraded mode. Hardware disk issues with /scratch1
02/28/15 8:39 PST02/28/15 9:22 PSTCarverSystem in degraded mode. Batch system degraded, logins are available.
02/27/15 13:28 PST02/27/15 16:17 PSTProjectBUnavailable. Genepool queues were being drained to avoid job failures due to projectb outage. Queues are reenabled.
02/25/15 13:50 PST02/25/15 14:10 PSTEdisonSystem in degraded mode.
02/25/15 10:00 PST02/25/15 16:30 PSTData Transfer NodesScheduled maintenance.
02/25/15 9:00 PST02/25/15 11:00 PSTHPSS UserScheduled maintenance.
02/24/15 10:52 PST02/24/15 12:26 PSTDNAUnavailable. The NFS mount on /global/dna was down since 10:52 AM and came back up at 12:26 PT. Engineers have resolved the issue.
02/24/15 8:00 PST02/24/15 9:11 PSTNIMScheduled maintenance.
02/23/15 17:46 PST02/23/15 18:17 PSTDNASystem in degraded mode. NFS export was not available.
02/22/15 14:23 PST02/22/15 15:30 PSTCarverSystem in degraded mode. Carver batch was down.
02/21/15 11:50 PST02/21/15 12:32 PSTHPSS BackupUnavailable.
02/20/15 15:42 PST-CarverNode
02/19/15 8:44 PST-CarverNode "carvergrid" maintenance extended; status update by noon, Fri Feb 20. Use node "carversl6" for grid services until carvergrid returns to production.
02/18/15 10:00 PST02/18/15 12:00 PSTHPSS BackupScheduled maintenance.
02/18/15 8:00 PST02/18/15 17:06 PSTHopperScheduled maintenance. ALL DATA on HOPPER /scratch (your $SCRATCH) WILL BE DELETED
02/14/15 12:55 PST02/14/15 13:36 PSTProjectUnavailable. /global/project was unavailable. Jobs may have been affected.
02/08/15 8:51 PST-EdisonEdison degraded - /scratch3 may be unavailable. Engineers are working on the issue.
02/08/15 3:14 PST02/08/15 9:25 PSTEdisonSystem in degraded mode. Edison degraded - /scratch3 may be unavailable.
02/06/15 12:20 PST-EdisonEdison is experiencing a slow I/O performance with the global file systems. We are aware of the issue and are actively investigating the problem.
02/04/15 17:23 PST-Edison/scratch3 is currently unavailable. /scratch3 recovery efforts ongoing with no data loss expected. Next status update will be by 16:00 PST, 02/05.
02/03/15 23:20 PST-PDSFSome older compute nodes are unavailable. There will be an update by 5 pm 2/4/15.
02/03/15 21:17 PST-Edison
02/03/15 7:00 PST02/03/15 23:15 PSTNERSC Quarterly MaintenanceScheduled maintenance.
02/03/15 7:00 PST02/03/15 19:10 PSTScience Gateway ServicesScheduled maintenance.
02/03/15 7:00 PST02/03/15 23:15 PSTCarverScheduled maintenance.
02/03/15 7:00 PST02/03/15 21:10 PSTEdisonScheduled maintenance.
02/03/15 7:00 PST02/03/15 21:15 PSTGenepoolScheduled maintenance.
02/03/15 7:00 PST02/03/15 19:40 PSTHopperScheduled maintenance.
02/03/15 7:00 PST02/03/15 18:45 PSTPDSFScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTDNAScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTGlobal CommonScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTGlobal HomesScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTGlobal ScratchScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTProjectScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:25 PSTProjectBScheduled maintenance.
02/03/15 7:00 PST02/03/15 16:20 PSTHPSS BackupScheduled maintenance.
02/03/15 7:00 PST02/03/15 16:20 PSTHPSS UserScheduled maintenance.
02/03/15 7:00 PST02/03/15 15:35 PSTData Transfer NodesScheduled maintenance.
01/28/15 9:00 PST01/28/15 13:01 PSTHPSS BackupScheduled maintenance.
01/24/15 16:30 PST01/24/15 17:10 PSTdtn03Unavailable. Dtn03 was rebooted to restore ssh access.
01/23/15 10:41 PST-GenepoolWe are experiencing some Genepool expels due to hiccups with /project. Your jobs might be affected if you are using this file system.
01/23/15 10:28 PST-Genepool/project filesystem was experiencing NFS stale handle since 09:50PT. Our engineers have found the problem and the issue has been resolved.
01/23/15 10:14 PST-Project/project filesystem was experiencing NFS stale handle. Our engineers have found the problem and the issue has been resolved.
01/23/15 10:14 PST-Hopper/project filesystem was experiencing NFS stale handle since 09:50PT. Our engineers have found the problem and the issue has been resolved.
01/23/15 10:14 PST-Edison/project filesystem was experiencing NFS stale handle since 09:50PT. Our engineers have found the problem and the issue has been resolved.
01/23/15 9:47 PST01/23/15 10:30 PSTProjectUnavailable. /project was unavailable on all systems due to system hardware issue. The problem has been identified and resolved.
01/21/15 9:00 PST01/21/15 13:00 PSTHPSS BackupScheduled maintenance.
01/20/15 9:07 PST-PDSF1/20/14 6:00 - 6:30 am. Several nodes had issues due to a power event. About 100 jobs were lost on these nodes.
01/20/15 8:00 PST01/20/15 10:00 PSTNIMScheduled maintenance.
01/20/15 7:28 PST-HopperQueues were stopped from 06:20 - 06:50 PST due to a power event. Jobs did not stop running.
01/13/15 12:50 PST-PDSF1/14/2014 10:00am - 10:00pm pdsfdtn2 unavailable
01/13/15 7:30 PST01/16/15 13:15 PSTEdisonScheduled maintenance.
01/07/15 13:35 PST02/07/15 15:52 PSTProjectSystem in degraded mode. Reads may be impacted to some portions of the file system. Writes are unaffected.
01/07/15 13:35 PST01/07/15 15:59 PSTScience Gateway ServicesUnavailable. Due to /project in degraded state.
01/07/15 9:00 PST01/07/15 11:00 PSTHPSS BackupScheduled maintenance.
01/06/15 10:00 PST-Edisonedisongrid (i.e. edison01) will be down for maintenance on 01/07/2015 from 08:00 - 12:00 PST.
01/05/15 8:00 PST01/05/15 8:49 PSTNIMScheduled maintenance.
01/03/15 17:40 PST01/04/15 1:30 PSTEdisonUnavailable. The system went down due to a communication problem. The system was rebooted.
01/03/15 12:21 PST01/03/15 13:00 PSTData Transfer NodesSystem in degraded mode. dtn03 is currently unavailable. The other DTN nodes are in service.
12/23/14 5:46 PST12/23/14 6:20 PSTData Transfer NodesSystem in degraded mode. DTN03 is down but other DTN are available.
12/19/14 16:13 PST-EdisonEdison is running several large jobs. Debug jobs may take longer to start than usual.
12/17/14 9:00 PST12/17/14 12:17 PSTHPSS UserScheduled maintenance.
12/16/14 11:30 PST12/16/14 14:00 PSTEdisonUnavailable.
12/10/14 10:00 PST12/10/14 10:44 PSTData Transfer NodesScheduled maintenance. DTN01 and DTN02 scheduled maintenance.
12/10/14 9:00 PST12/10/14 15:43 PSTHPSS BackupScheduled maintenance.
12/09/14 15:38 PST-EdisonEdison is experiencing issues with the /scratch3 filesystem. We are actively investigating.
12/03/14 9:00 PST12/03/14 10:50 PSTHPSS BackupScheduled maintenance.
12/03/14 9:00 PST12/03/14 11:00 PSTHPSS UserScheduled maintenance.
11/30/14 13:40 PST11/30/14 17:20 PSTEdisonSystem in degraded mode. Edison was degraded due to batch system issues; intermittent problems with job submission and scheduling.
11/26/14 10:55 PST11/26/14 12:00 PSTHPSS BackupUnavailable.
11/26/14 9:00 PST11/26/14 10:20 PSTHPSS BackupScheduled maintenance.
11/19/14 11:00 PST11/19/14 11:30 PSTNIMScheduled maintenance.
11/18/14 7:20 PST11/18/14 10:37 PSTCarverSystem in degraded mode. Previously-submitted jobs are running and users may submit new jobs, however new jobs will not start due to /project unavailability.
11/18/14 4:45 PST11/18/14 10:18 PSTProjectUnavailable.
11/12/14 0:22 PST-HopperIDL/MATLAB/Mathematica unavailable on Hopper.
11/12/14 0:22 PST-EdisonIDL/MATLAB/Mathematica unavailable on Edison.
11/12/14 0:05 PST-CarverPGI and Intel compilers unavailable on Carver. IDL/MATLAB/Mathematica unavailable on Carver.
11/12/14 0:01 PST11/12/14 9:30 PSTLicense ServerScheduled maintenance. License Server DMV has been returned to production service. Licensed software is now available.
11/11/14 11:00 PST11/11/14 13:00 PSTNIMScheduled maintenance. Web interface to NIM unavailable.
11/11/14 10:00 PST11/11/14 10:30 PSTNX ServicesScheduled maintenance. The NX service will be unavailable.
11/11/14 9:00 PST11/11/14 11:18 PSTHPSS UserScheduled maintenance.

Show Older Outages: