Systems
Franklin Home
About Franklin
Access to Franklin
User Environment
Accounts/Charges
File Storage
Programming
Running Jobs
Software
Performance Tools
Performance Monitoring
IO Strategies
Franklin Timeline
Vendor Manuals
Status & Stats
NERSC MOTD
Announcements
Known Problems
Current Queue Look
Completed Jobs List
Job Stats
Benjamin Franklin, today known primarily for his central role in founding the United States, was in his own time internationally recognized as one of America’s leading scientists, primarily for his work with electricity and energy. Franklin's theory of electricity laid the foundation for the science of electrostatics. He coined many of the electrical terms we use today, such as battery, conductor, condenser, positive and negative charge, electric shock, and electrician. He is also remembered as the inventor of the Franklin stove, lightning rod, bifocals, medical catheter, swim fins, glass armonica, and odometer.
Franklin’s scientific interests were wide-ranging and pioneering. He was the first to realize that the Earth's climate could change, first to plot the course of a hurricane, and first to chart the Gulf Stream. He studied cooling by evaporation, experimented with oil films on water, warned of the dangers of lead poisoning, and hypothesized about continental drift and the wave theory of light. Franklin founded the American Philosophical Society to help scientists share their discoveries.
Franklin won every major scientific honor of his time, including the Royal Society's Copley Medal (the equivalent of today’s Nobel Prize); election to the Royal Societies of England, France, and Germany; and honorary degrees from Oxford and other universities. His fame as a scientist opened doors and paved the way for his success as a diplomat during the American Revolution and the negotiation of the Treaty of Paris.
(Portrait: 1777, by Jean-Baptiste Greuze)
About Franklin
Cray XT4
The NERSC Cray XT4 system, named Franklin, is a massively parallel processing (MPP) system with 9,572 compute nodes. Each node has quad processor cores, and the entire system has a total of 38,288 processor cores available for scientific applications. The system is named in honor of Benjamin Franklin.
Each of Franklin's compute nodes consists of a 2.3 GHz single socket quad-core AMD Opteron processor (Budapest) with a theoretical peak performance of 9.2 GFlop/sec per core (4 flops/cycle if using SSE128 instructions). Each compute node has 8 GB of memory (2 GB of memory per core), and each service node (e.g. login node) has 8 GB of memory. The full system consists of 102 cabinets with 78 TBytes of aggregate memory. The memory speed is 800 MHz. The theoretical peak performance of Franklin is about 352 TFlop/sec. Each compute node is connected to a dedicated SeaStar2 router through Hypertransport with a 3D torus topology which ensures high performance, low-latency communication for MPI and SHMEM jobs.
Franklin uses two different operating systems. Full-featured SuSE Linux is run on service nodes (10 login nodes and other I/O, network and system nodes). A light weight OS based on Linux, Compute Node Linux (CNL), is run on each compute node. CNL reduces system overhead, and is critical for the system to scale to very large concurrency. The overall operating system on Franklin is called Cray Linux Environment (CLE). The Parallel file system on Franklin is provided by Lustre with approximately 436 TB of user disk space.
Franklin Configuration
Franklin Specifications | |
|---|---|
| Number of compute nodes | 9,572 |
| Processor cores per node | 4 |
| Number of compute processor cores | 38,288 |
| Number of spare compute nodes | 20 |
| Processor Core type | Opteron 2.3 GHz Quad Core |
| Processor core theoretical peak | 9.2 GFlop/sec |
| System theoretical peak (compute nodes only) | 352 TFlop/sec |
| Physical memory per compute node | 8 GB |
| Memory usable by applications per node | 7.38 GB |
| Number of login nodes | 10 |
| Switch Interconnect | SeaStar2 |
| Measured MPI Point-to-Point Bandwidth | 1.6 GB/sec |
| Measured MPI Point-to-Point Latency | 6.5-8.5 μs |
| File System | Lustre |
| Usable disk space | 436 TB |
| Theoretical IO Bandwidth | 16 GB/sec to each of two /scratch file systems; 32 GB/sec aggregate |
| Batch system | Torque/Moab |
Software Configuration
- SuSE SLES10 SP1 Linux on service nodes
- Compute Node Linux (CNL) for all compute nodes
- Portals communication layer
- Lustre Parallel File System
- ALPS utility (aprun) to launch compute node applications
- Torque resource management system with the Moab scheduler
- Programming Environment
- PGI compilers: assembler, Fortran, C, and C++
- Gnu compilers: C, C++, and Fortran F77
- Pathscale compilers: Fortran, C, and C++
- Parallel Programming Models: Cray MPICH2 MPI and Cray SHMEM
- AMD Core Math Library (ACML): BLAS, LAPACK, FFT, Math transcendental libraries, Random Number generators, GNU Fortran libraries
- LibSci scientific library: ScaLAPACK, BLACS, SuperLU
- A special port of the glibc GNU C library routines for compute node applications
- CrayPat and Cray Apprentice2
- Performance API (PAPI)
- Modules
Franklin Production Compute Node Configuration
| Dates | Quad-Core Nodes | Dual-Core Nodes | Total Cores |
|---|---|---|---|
| 7/14/2008 and before | 0 | 9,660 | 19,320 |
| 7/15/2008 - 8/12/2008 | 0 | 7,356 | 14,712 |
| 8/13/2008 - 8/20/2008 | 1,728 | 7,932 | 22,776 |
| 8/21/2008 - 9/9/2008 | 1,728 | 5,052 | 17,016 |
| 9/10/2008 - 9/17/2008 | 4,588 | 5,052 | 28,456 |
| 9/18/2008 - 10/16/2008 | 4,588 | 1,040 | 20,392 |
| 10/17/2008 - 10/28/2008 | 8,630 | 0 | 34,520 |
| 10/29/2008 - 3/31/2009 | 9,660 | 0 | 38,640 |
| 4/1/2009 to present | 9,582 | 0 | 38,328 |
IO Upgrade and System Stabilization
NERSC improved Franklin's stability and enhanced the I/O bandwidth to scratch file systems during March to May 2009. The details are described on the Franklin IO upgrade and system stabilization page.
Quad-Core Upgrade
Franklin's processors were upgraded from dual-core to quad-core Opterons from July 15 to October 29, 2008. The details are described on the Franklin quad-core upgrade page.
Franklin Dual-Core Configuration (Prior to July 15, 2008)
Franklin Specifications | |
|---|---|
| Number of compute nodes | 9,660 |
| Processor cores per node | 2 |
| Number of compute processor cores | 19,320 |
| Number of spare compute nodes | 20 |
| Processor Core type | Opteron 2.6 GHz Dual Core |
| Processor core theoretical peak | 5.2 GFlop/sec |
| System theoretical peak (compute nodes only) | 101.5 TFlop/sec |
| Physical memory per compute node | 4 GB |
| Memory usable by applications per node | 3.75 GB |
| Number of login nodes | 10 |
| Switch Interconnect | SeaStar2 |
| File System | Lustre |
| Usable disk space | 350 TB |
| Batch system | Torque/Moab |
Software Configuration
- SuSE SLES 9.2 Linux on service nodes
- Compute Node Linux (CNL) for all compute nodes
- Portals communication layer
- Lustre Parallel File System
- ALPS utility (aprun) to launch compute node applications
- Torque resource management system with the Moab scheduler
- Programming Environment
- PGI compilers: assembler, Fortran, C, and C++
- Gnu compilers: C, C++, and Fortran F77
- Pathscale compilers: Fortran, C, and C++
- Parallel Programming Models: Cray MPICH2 MPI and Cray SHMEM
- AMD Core Math Library (ACML): BLAS, LAPACK, FFT, Math transcendental libraries, Random Number generators, GNU Fortran libraries
- LibSci scientific library: ScaLAPACK, BLACS, SuperLU
- A special port of the glibc GNU C library routines for compute node applications
- CrayPat and Cray Apprentice2
- Performance API (PAPI)
- Modules
