NERSCPowering Scientific Discovery Since 1974

Franklin Timeline

This page records a brief timeline of significant events and user environment changes on Franklin. Franklin compute nodes have been upgraded from dual core to quad core from July to October 2008.

Apr 30, 2012
Franklin is retired.
Apr 5, 19, 27, 2012
Reminder announcements on Franklin retirement schedule.
-- Thurs Apr 26, 23:59: Batch system is drained, batch queues are stopped (no jobs will be running at this point)
-- Mon Apr 30: Last day to retrieve files from Franklin scratch file systems
-- Mon Apr 30, 23:59: User logins are disabled
Mar 6, 2012
Announcement on Franklin retirement date set: 04/30/2012.
Feb 22, 2012
HW and SW maintenance. Set to default versions: pgi/11.10.0 and xt-mpt/5.3.5.
Feb 21, 2012
Update on Franklin earliest retirement date: 4/30/2012.
Dec 20, 2011
First announcement: Franklin earliest retirement date: 03/31/2012.
Oct 3, 2011
Set to default versions: pathscale/4.0.9.
Sep 14, 2011
HW and SW maintenance.  Installed field notices and software patches.  OS upgraded to CLE2.2UP03B.  GPFS upgraded to 3.3. Set to default versions: xt-asyncpe/5.0, xt-mpt/5.3.2, pgi/11.7.0, cce/7.4.2, gcc/4.5.3, xt-libsci/10.5.02, petsc/3.1.08, petsc-complex/3.1.08, tpsl/1.1.0, atp/1.3.0, xt-lgdb/1.4, perftools/5.2.2, papi/4.1.3, libfast/1.0.9.
Jun 15, 2011
HW and SW maintenance.  Installed field notices and software patches. Moab updated to 6.0.4.  Converted to new LDAP server. Set to default versions: pgi/11.3.0, perftools/5.2.0. CPU limit on Freedom set to 24 hrs. 
Apr 20, 2011
HW and SW maintenance. Installed field notices.  Set to default versions: xt-asyncpe/4.9, pgi/11.2.0, xt-mpt/5.2.1, cce/7.3.3, xt-libsci/10.5.01, petsc/3.1.05, petsc-complex/3.1.05, gcc/4.5.2, gdb/7.2.
Feb 16, 2011
HW and SW maintenance. Installed field notices. Updated system softwares: Moab, GPFS, myquota, chkmyqta. Deconfigured RDAC, configured GRES for /project. Set to default versions: xt-asyncpe/4.6, pgi/10.9.0, xt-mpt/5.1.3, cce/7.3.0, xt-libsci/10.5.0, petsc/3.1.04, petsc-complex/3.1.04, perftools/5.1.3 (includes xt-craypat/5.1.3, apprentice2/5.1.3), trilinos/10.6.0, chapel/1.2.1.
Nov 10, 2010
HW and SW maintenance. Installed field notices. Set to default versions: xt-asyncpe/4.3, pgi/10.8.0, xt-mpt/5.1.0, cce/7.2.7, xt-libsci/10.4.8, petsc/3.1.02, petsc-complex/3.1.02, hdf5/1.8.5, hdf5-parallel/1.8.5, netcdf/4.1.1, perftools/5.1.2 (includes xt-craypat/5.1.2, apprentice2/5.1.2, and xt-papi/4.1.0), modules/, atp/1.0.3.
Sep 8, 2010
HW and SW maintenance. Installed field notices. Set to default versions: xt-asyncpe-4.0, pgi/10.5.0, xt-mpt/5.0.0, acml/4.4.0, xt-craypat/5.1.0, apprentice2/5.1.0, gcc/4.4.4, cce/7.2.4, petsc/3.1.00, petsc-complex/3.1.00, pathscale/3.2.99, java/6.0_20.
Jul 27-31, 2010
Franklin down due to NERSC Center Power upgrade and system stress test afterwards.
Jun 23, 2010
HW and SW maintenance. Installed field notices. Installed DVS patch. Upgraded SSH, Moab, and nscd. Changed Torque configuration so that batch job stdout/stderr spool files appear and progress in user job submit directory while the job is running.
May 12, 2010
HW and SW maintenance. Installed field notices. Installed ldap, gpfs, and nscd patches. Added Torque "SERVERHOST" setting.
May 3, 2010
Decreased the boundary between reg_small to reg_med from 512 to 256 nodes, increased the boundary between the reg_med and reg_big queues from 1024 to 2000 nodes, and changed the queue charge factor for reg_med from 0.8 to 0.75 (more discount).
Apr 21, 2010
HW and SW maintenance. Installed field notices. Installed torque/2.4.7. Lustre configuration (disable statahead) changes. Set to default versions: pgi/10.0.0, xt-libsci/10.4.3, xt-mpt/4.0.3, xt-asyncpe/3.7, fftw/, xt-craypat/5.0.2, apprentice2/5.0.2, gcc/4.4.3, cce/7.2.1, netcdf/, netcdf-hdf5parallel/, hdf5/ and hdf5-parallel.
Apr 1, 2010
The Franklin external login node "Freedom" entered into production.
Mar 24, 2010
OS upgraded to CLE 2.2UP02. OS version is 2.2.48B.
Feb 10, 2010
Hardware maintenance. Virtual Channel 2 (VC2) enabled. Idle loop set to halt.
Jan 21, 2010
Hardware maintenance. Franklin home converted to global homes.
Dec 9, 2009
Hardware maintenance. Installed field notices include new nscd. Preparation work for Global Home conversion. New modules that has options for "module avail" installed.
Nov 17, 2009
Installed field notice (Linux security fix). Set hdf5 and hdf5-parallel to default. Installed portals bug fix, OS version is now 2.1.50HDB_PS13A.
Nov 4, 2009
Hardware maintenance. Installed field notices. New nscd and glic. Set to default versions: pgi/9.0.4, xt-mpt/3.5.0, xt-asyncpe/3.3, xt-libsci/10.4.0, xt-craypat/apprentice2 5.0.0, gcc/4.4.1, cce/7.1.4, chapel/1.0, Cray hdf5_netcdf-1.5. Implement Pathscale no license checking. GPFS upgrade. Configured to prepare for future /global/home conversion.
Oct 1, 2009
Charging discount reg_med queue decreased from 25% to 20% (charging factor is 0.8).
Sept 30, 2009
Hardware maintenance. Installed DVS and portals patches. Connected the eslogin node (freedom) to Franklin (routing and network changes needed).
Sept 16, 2009
Hardware maintenance. Installed field notices. Retired "feature=quad" for batch script. Installed new NERSC SSH. /usr/common mounted on NGF as /global/common/franklin. Set to default versions: pgi/9.0.2, xt-mpt/3.4.1, xt-asyncpe/3.2, xt-libsci/10.3.8, acml/4.3.0, cce/7.1.2, xt-mpt/3.4.1, Cray hdf5 and hdf5-parallel/, and Cray netcdf/
Sept 4-6, 2009
Linux Kernel, RDAC, GPFS, DVS, portals, and lustre patches are upgraded on Franklin due to a security vulnerability in Linux kernel. OS level is upgraded from 2.1.50HD to 2.1.50_HDA_PS12A.
Sept 2, 2009
Hardware maintenance. Installed field notices. Restarted LDAP server.
Aug 19, 2009
Hardware maintenance. Installed field notices. Installed DVS patch and software portals patch for MPICH PtlEQPoll "unresponsive nodes in the system" error. Added new batch queue execution class reg_short with max user run limit of 12. and max user idle limit of 8. xt-asyncpe/3.0, xt-mpt/3.3.0, and xt-libsci/10.3.6 are set to default.
Aug 5, 2009
Hardware maintenance. Installed field notices. Franklin internal networking changed to free up two network nodes for eslogin.
Jul 22, 2009
Hardware maintenance. Installed field notices. Installed DVS and HSS patches. Installed RDAC. Upgraded Torque/Moab to use node counts basis (instead of mppwidth) for execution class routing.
Jul 8, 2009
Hardware maintenance. Installed field notices. GPFS upgraded to PTF12. Torque upgrade to resolve the core dump at reservation problem.
Jun 19, 2009
xt-asyncpe/2.5, pgi/8.0.6, xt-mpt/3.2.0, xt-libsci/10.3.5, and gcc/4.3.3 are set to default.
Jun 18, 2009
Daily puring on scratch file systems starts. Cut off date is 12 weeks as defined by last access time.
Jun 2, 2009
Hardware maintenance. DDN upgrade for one cabinet. Installed software patch for UPC.
May 12, 2009
Introduced reg_med queue (using 2,045-4,092 cores) with 25% charging discount. Jobs running in reg_big (using 4,093-24,572 cores) and reg_xbig (using >=24,573 cores) queues now have 50% charging discount.
Apr 27, 2009
Made some changes on ldap parameter settings to mitigate the "identifier removed" problem. Installed new security patch for udev. Changes in ports reservation for GPFS.
Apr 21, 2009
Set maxproc limit on login/MOM nodes to 150 soft and 200 hard.
Apr 16, 2009
libfast/1.0.2, xt-libsci/10.3.3, xt-mpt/3.1.2, pgi/8.0.4 and xt-asyncpe/2.3 are set to default. Installed security patch for udev.
Mar 31 - Apr 1, 2009
Hardware maintenance. New IO hardware added. /scratch reformatted. /scratch2 created. Service nodes redistributed. High Speed Network recabled.
Mar 24, 2009
Hardware maintenance. Hardware upgrade login nodes, MOM nodes and network nodes to PCI-Express. Redistribute login nodes, Mom nodes and network nodes in the torus. Change settings for two HSN Tuning parameters: dateline and aging settings.
Mar 17, 2009
OS upgrade from CLE 2.1 to CLE 2.1UP01 with patch sets 01, 01A and 02. Hardware repairs. PAM/LDAP related changes. Network configuration change (from 42 net to 41 net). Reduce max wallclock time for regular queues from 36 hrs to 24 hrs. Insert new SIO modules (converted from compute modules). Franklin has 9,532 compute nodes (with 60 spare nodes).
Mar 10, 2009
Hardware maintenance. Upgrade version to Moab/5.2.5-s12920. Updated a lustre timeout parameter (kptllnd) from 50 sec to 250 sec to reduce the chances of user jobs being terminated by Lustre timeout.
Mar 9, 2009
Free charging (due to /scratch reformat) ends.
Mar 3, 2009
Franklin rebooted to clear situation of several login nodes out-of-memory conditions and lost connection to /scratch, which also caused some jobs hung exiting. Also made the following changes: Separate login and mom nodes (10 login nodes and 6 MOM nodes); Set a 512MB limit on /tmp for login and mom nodes; And set the following new memory limits (unit in bytes) on the login nodes:
                Type    Item         Current      New
Soft stack 2097152 131072
Hard stack 2097152 262144
Soft data 2097152 1048576
Soft memoryuse 2097152 1048576
Soft vmemoryuse unlimited 2097152
Hard vmemoryuse unlimited 2097152
Feb 25, 2009
Due to disk controller problem caused Franklin /scratch to be reformatted. Charging is free until further notice.
Feb 22, 2009
Installed the new firmware patch that caused SWOs caused by CAM overflow condition.
Feb 21, 2009
Installed firmware patch that caused SWOs caused by CAM overflow condition.
Feb 7, 2009
Turned off virtual Channel 2 (VC2) setting.
Feb 4, 2009
Hardware and Software maintenance. Installed secutiry patches. Enabled VC2. Installed new torque/2.4.0b1. Set usecp for /project. Started to enforce job submission filter by checking user quota.
Jan 29, 2009
Reinstalled a Moab scheduler version 5.2.4-s11143 with 10k group entries.
Jan 27, 2009
xt-craypat/4.4.0 and apprentice2/4.4.0 are set to default.
Jan 26, 2009
Installed two security patches. Installed the corrected patch for LBUG. Hardware maintenance.
Jan 18, 2009
Removed the software patch installed for LBUG on 01/15/09.
Jan 16, 2009
Installed a revised patch for the ssnal bug.
Jan 15, 2009
Installed two software patches, one to fix an ssnal bug that caused login/network nodes to become unresponsive, and the other to fix an LBUG. Set parameter max_rw_chunka to zero as a workaround for slower single stream IO performance.
Jan 13, 2009
Allocation Year 2009 starts. Charging factor is now based on Franklin hours. 50% Quad core discount ended (Each quad core node is now charged fully). reg_big discount is reduced from 50% to 25%. pgi/8.0.1 , xt-mpt/3.1.0, acml/4.2.0 , and fftw/ are set to default.
Jan 12, 2009
Allocation Year 2008 ends. Hardware maintenance.
Jan 6, 2009
Franklin software maintenance. Installed the lustre portals patches removed on Dec 25, plus more software patches. Put into place the aprun wrapper to reject MPT2 parallel executables.
Dec 29, 2008
xt-mpt/2.x.x versions removed from the system. Installed xt-mpt/3.0.4, xt-mpt/3.1.0, acml/4.2.0, xt-libsci/10.3.1, fftw/, pgi/8.0.1, xt-asyncpe/2.0, fast_mv/1.0.1.
Dec 25, 2008
Softwares patches installed on Dec 16 were removed to help diagnose system crashes.
Dec 21, 2008
Turned off virtual Channel 2 (VC2) setting.
Dec 16, 2008
pgi/7.2.5, xt-mpt/3.0.3 , and acml/4.1.0 set to default. Turned on virtual Channel 2 (VC2) setting. Installed several bug software patches (2 lustre and 1 portals patches), including warmbooting down nodes, MDS node panic related to quota enforcement, and the nodes marked down de to lustre apps hanging.
Dec 8, 2008
Changed the SeaStar netmask from to to fix a communication problem (for example login and ssh to Franklin) from IP address 192.x.x.x. Changed the nfsserver services on the service nodes. Installed new sshd.
Dec 3-4, 2008
Franklin OS was upgraded from 2.0 to 2.1. Service nodes now run SLES 10 Service Pack 1 (upgraded from SLES 9.2). Lustre upgraded from 1.4.12 to 1.6.5. New SSHD installed.
Nov 17, 2008
Franklin was down to fix hardware for a Link Inactive, to upgrade SMW software and Lustre software conversions.
Oct 29, 2008
Franklin quad core upgrade final configuration. The completely upgraded Franklin has 9,660 quad core compute nodes (38,640 quad cores).
Oct 17-19, 2008
Franklin quad core upgrade phase 4 started. Hardware maintenance. 7 columns of quad core modules from the test system merged to the production system. Modules from columns 14 and 16 swapped with those in columns 0 and 1. Franklin production environment is now a pure quad core system: columns 0-13 and 15 quad core, total of 34,032 compute cores. Torque upgraded to version 2.3.4-snap.20080901135 and Moab upgraded to version 5.2.4-s11143.
Oct 16, 2008
Started to implement franklin /scratch purge policy. Start from files older than 24 weeks, defined by last access date, down to files older than 16 weeks if necessary.
Oct 15, 2008
Started to implement the policy to kill jobs with stdout/stderr file sizes bigger than 100 MB.
Oct 13, 2008
pathscale/3.2 set to default.
Oct 6, 2008
pgi/7.2.4 set to default.
Sept 30, 2008
Hardware maintenance. Also installed securities patches from field notices.
Sept 19, 2008
License server moved from login01 to login11.
Sept 17, 2008
Franklin quad core upgrade phase 3b started. Hardware maintenance. Franklin full configuration in production: even columns 2-16 quad core, columns 0, and columns 0 and 1 dual core, total of 20,392 compute cores. pgi/7.2.4 installed.
Sept 10, 2008
Franklin quad core upgrade phase 3a started. Hardware maintenance. Franklin full configuration in production: even columns 2-16 quad core, columns 0, and odd columns 1-15 dual core, total of 28,456 compute cores. Default environment set to quad core. Franklin quad core started charging (in dual core rate). Queue structure adjusted by doubling min core and max cores for each queue class. 50% discount for reg_big with >= 4829 cores (instead of 2415 cores).
Aug 21, 2008
Franklin quad core upgrade phase 2b started. Hardware maintenance. Franklin full configuration in production: cols 12,14,16 quad core, cols 0-10,11,13,15 dual core, total of 17,016 compute cores. Installed new SSH. Set xt-mpt/3.0.2 as default.
Aug 13, 2008
Franklin quad core upgrade phase 2a started. Hardware maintenance. Franklin full configuration in production: cols 12,14,16 quad core, cols 0-10,11,13,15 dual core, total of 22,776 compute cores.
Aug 11, 2008
Installed xtpe-quadcore, xt-async/1.0, gcc/4.2.0.quadcore, libsci/10.2.1. Also set gcc/4.2.0.quadcore and libsci/10.2.1 to be default. Set xt-craypat/4.3.1and apprentice2/4.3.0 to be default. Reduced the process memory hard limit on the service nodes from 4 GB to 2 GB.
Jul 15, 2008
Franklin quadcore upgrade started. 4 columns were removed from Franklin production system. A total of 14,712 compute cores are available. (cols 0-11,13 dual cores). Installed new portals firmware patch. Installed ALPS with new patches. Installed Field Notice. Performed hardware maintenance for failed nodes.
Jun 28, 2008
Torque/2.2.2 rolled back to torque/2.2.0.
Jun 22-24, 2008
/var mount options added; NERSC sshd installed on service nodes; We have 10 login nodes, and 16 MOM nodes. Upgraded DDN fimware to 3.11.h and rolled back to 3.11.i due to potentiality of data corruption.
Jun 18, 2008
Hardware maintenance (nodes and DDN controller). Installed portals fix for SPR 743158.
Jun 12, 2008
Added two more pathscale compiler licenses (total of 4 now).
Jun 11, 2008
OS upgraded from 2.0.44a2 to 2.0.53. Lustre upgraded from 1.4.9 to 1.4.12+ with bug fixes. Torque upgraded to a newer version 2.2.2-200805051805. Login nodes and MOM nodes are separated. (later found out not effective) We now have 10 login nodes, and 10 MOM nodes.
May 30, 2008
Installed a DDN bug fix to correct /scratch IO problem.
May 28, 2008
Set pgi/7.1.6 to be the default pgi compiler.
May 27, 2008
Upgraded DDN controller firmware to correct /scratch IO problem. Had some other HW actions for the maintenance.
May 22, 2008
Increase regular queue wall clock limit from 24 to 36 hours. Increase low queue wall clock limit from 12 to 24 hours.
May 21, 2008
Started to enforce 2 GB soft memory limit and 4 GB hard memory limit on processes running on login/service nodes. Started to enforce 8-hour idle session logout. The 60 min CPU time limit was put onto batch jobs in the morning and backed out in the afternoon.
May 19, 2008
nscd (name service caching daemon) time-to-live increased to 600 sec for password, and 3600 sec for group change.
May 12, 2008
Implemented automatic login node failure clearing after Franklin comes back up from either a crash or a scheduled down.
Apr 28, 2008
Hardware and software maintenance. Installed Field Notice. Installed additional patch for SeaStar HB fault. Reconfigured Lustre to allow access from 4 network nodes. NERSC reserves the right to kill jobs that are over the limit of 8 for multiple simultaneous apruns in a batch job from today.
Apr 22, 2008
Started to enforce 60-min CPU time limit on processes running on login/service nodes from interactive jobs.
Apr 21, 2008
Fixed the firmware change defect for SPR 741141. Set the limit of 8 for multiple simultaneous apruns in a batch job, will enforce from 04/28/08.
Apr 2, 2008
OS upgrade to 2.0.44a2. Installed a portals patch (firmware change) for SPR 741141 to increase system stability. Installed patches for two Lustre quota bugs.
Mar 26, 2008
Hardware and software maintenance. More power modules swapped. Updated PIC code for voltage settings. Set pgi/7.1.4, craypat/4.1.1, Apprentice2/4.1, and papi/3.5.99b to default.
Mar 22, 2008
Set CNL kernel boot parameter idle=poll.
Mar 19, 2008
gcc/4.2.3 and acml/4.0.1a set to default. Torque rolled back to version 2.2.0
Mar 12, 2008
Hardware and software maintenance. More power modules swapped. Torque upgraded to version 2.2.2-200712171618. Security field notice FN5509 applied.
Feb 27, 2008
Hardware maintenance. More power modules swapped.
Feb 13, 2008
OS upgraded to CNL version 2.0.39. Has the multiple apruns in background feature. ACML is not automatically invoked by compiler driver any more. Fixed the Lustre falsely thinks user inode over quota problem. Installed patch for degraded IOR read and write performance due to quota enabled. "2s" firmware setting in place to fix a portal's issue related to seastar non-deadbeaf hang. set CNL kernel boot parameter idle=halt.
Feb 7, 2008
Hardware maintenance. Swapped some power modules.
Jan 22, 2007
MPT portals fix for the MPI send-to-self deadlock problemi, also solved the problem of MPI_allreduce call needing an explicit barrier.
Jan 9, 2008
xt-craypat/4.1 and apprentice/4.1 installed as non-default.
Jan 8, 2008
FY08 production year starts. Production queues max wall clock limit increased from 12 hours to 24 hours. Reduced the reg_small maximum tasks to 2416. Enabled premium queue. Franklin starts official charging with charging factor of 6.5 and 50% discount for jobs using 1208+ nodes.
Dec 13, 2007
Moab/5.1.0p8 installed and set to default.
Dec 10, 2007
GCC/4.1.2 installed.
Dec 5, 2007
UPC/1.0.2 installed.
Dec 3, 2007
Set apprentice/4.0 as default.
Nov 29, 2007
Moab configure changes. Max user idle limit for reg_big increased from 1 to 2.
Nov 16, 2007
Installed CrayPat 4.0.0, made module xt-craypat/4.0 the default. Installed Apprentice 4.0.0, still set apprentice2/3.2.3 the default.
Nov 7, 2007
Moab configuration changes for enforcing gloabl limit, and for settings per CLASS.
Nov 2, 2007
Set Pathscale default version from 3.0 to 3.1.
Oct 31, 2007
Libsci default changed from 10.0.1 to 10.2.0.
Oct 26, 2007
Franklin officially accepted by NERSC form Cray.
Oct 23, 2007
Fix for NWCHEM, GAMESS and Global Arrays codes: The default xt-mpt module has been set from 2.0.24b to 2.0.24d, with a new environment variable SHMEM_SWAP_BACKOFF set to 100.
Oct 19, 2007
Default compiler versions upgraded: pgi from 6.2.5 to 7.0.7, and gcc from 3.2.3 to 4.2.1.
Oct 17, 2007
Franklin some Moab config changes.
Oct 15-22 2007
Production CNL benchmark runs.
Oct 14, 2007
User quota enforced. Franklin queue structure regarding regular batch classes was simplified to have only "reg_small", "reg_big" and "reg_xbig" classes to replace the original 10+ buckets.
Oct 13-14 2007
Dedicated CNL witness benchmark runs.
Oct 8 2007
OS upgrade to OS 2.0.24b+ (GA version). Installed ALPS 9/5 RPM, which includes fix for MPMD aprun command line issues, and aprun -L issues.
Oct 5, 2007
Installed patch on Franklin for SHMEM portals problem
Sept 24-26, 2007
All NERSC Users enabled on Franklin
Sept 24, 2007
Trap installed for Shmem atomic problem on Franklin
Sept 14, 2007
Installed new Moab version 5.1.0p7. /dd>
Sept 12, 2007
Installed pgi/7.0.7 and gcc/4.2.1. Not set as default.
Sept 10, 2007
OS 2.0.14 patched for enabling "aprun -L" option.
Sept 7, 2007
OS 2.0.14 patched for enabling aprun MPMD mode.
Aug 3, 2007
Acceptance period begins. The system is tested and tuned and is required to pass strict performance and functionality tests.
Early July 2007
Early users started to have access on Franklin.
June to August, 2007
Assessed CNL with initial testing, testing at scale and improvements.
First two weeks in June 2007
Franklin CNL evaluation, installed CNL the week it was released from Cray Develop to Cray Testing.
March to May, 2007
System integration and partial acceptance testing with CVN.
Early March, 2007
Very-early staff users started to have access to Franklin.
Feb 10-12, 2007
Franklin phase 2a (30 cabinets) and phase 2b delivered (36 cabinets).
Jan 29-31, 2007
Franklin Isobase installed.
Jan 16, 2007
Franklin phase 1 delivered (36 XT4 cabinets, 11 DDN cabinets).
Dec 7, 2006
Silence upgraded to XT4.
Aug 21, 2006
Test System "Silence" (1 XT3 cabinet) arrived at OSF.
Aug 10, 2006
Franklin contract awarded to Cray by DOE.