Systems
Jacquard
For IBM SP UsersConnecting
Accounts/Charges
File Storage
Programming
Running Jobs
Software
Linux Environment
Vendor Manuals (PDF)
PathScale User GuideACML User Guide
PBS Pro User Guide
Status & Statistics
System Status: UPNERSC MOTD
Announcements
Jacquard Queue Status
Completed Jobs
Jacquard Job Stats
Jacquard News
Jacquard will be decommissioned on Thursday April 8, 2010. The last day for user logins will be April 7 and the last batch jobs will end at 11:59 PM Pacific Time, April 6. Jacquard is being replaced by the new Carver cluster.
No Jacquard files will be retained or archived - this applies to both home directory files as well as scratch files. You will need to transfer any files you wish to retain to other systems, for example HPSS, the NERSC Global File (NGF) system, Carver, Hopper or Franklin.
Jacquard is currently running with limited support. Please note:
- No software upgrades are planned.
- Failed hardware may not be replaced.
- A major hardware failure may be terminal.
- Users should back up important files to HPSS often.
Jacquard — Opteron Cluster
Jacquard is a 712-CPU Opteron cluster running a Linux operating system. The machine is named in honor of inventor Joseph Marie Jacquard, whose loom was the first machine to use punch cards to control a sequence of operations.
Jacquard has 356 dual-processor nodes available for scientific calculations. An additional 8 spare nodes will be in service if available. Four nodes are dedicated as login nodes. There are additional I/O and service nodes. Each processor runs at a clock speed of 2.2GHz, and has a theoretical peak performance of 4.4 GFlop/s. Processors on each node share 6GB of memory. The nodes are interconnected with a high-speed InfiniBand network. Shared file storage is provided by a GPFS file system.
| Jacquard Characteristics | |
|---|---|
| Processor type | Opteron 2.2 GHz |
| Processor theoretical peak | 4.4 GFlops/sec |
| Number of application processors | 712 |
| System theoretical peak (computational nodes) | 3.13 TFlops/sec |
| Number of shared-memory application nodes | 356 |
| Processors per node | 2 |
| Physical memory per node | 6 GBytes |
| Usable memory per node | 3-5 GBytes |
| Number of shared-memory spare application nodes (In service when possible.) |
8 |
| Number of login nodes | 4 |
| Switch Interconnect | InfiniBand |
| Switch MPI Unidirectional Latency | 4.5 usec |
| Switch MPI Unidirectional Bandwidth (peak) | 620 MB/s |
| Global shared disk | GPFS |
| Usable disk space | 30 TBytes |
| Batch system | PBS Pro |
Jacquard I/O Subsystem Configuration
- 16 of the 20 I/O servers are reserved for $SCRATCH
- Each node (compute, storage and login) has 4x connection to a 'leaf' IB switch, with 12 nodes per switch.
- Each switch has a 12x connection to 4 "top-level" switches (4 ports per leaf switch).
- Each I/O node has 1 dual-port 2Gbps QLogic Fibre Channel card (400MBps) hooked up to a S2A8500 DDN controller.
- Every DDN contoller is a "couplet" with 8 2Gbps ports which hook up to 4 I/O nodes. There are a total of 5 couplet controllers.
- In total for $SCRATCH there are 32 2 Gbps links to disk and a total of 40 Fibre Channel links to the DDNs.
- There is no failover at each link. The only failover is that two nodes can talk to the same LUNs. If one node goes down access to the disk is not lost although performance will be degraded.
