Systems
Jacquard
For IBM SP UsersConnecting
Accounts/Charges
File Storage
Programming
Running Jobs
Software
Linux Environment
Vendor Manuals (PDF)
PathScale User GuideACML User Guide
PBS Pro User Guide
Status & Statistics
System Status: UPNERSC MOTD
Announcements
Jacquard Queue Status
Completed Jobs
Jacquard Job Stats
Jacquard News
Retirement Status
Jacquard is currently running with limited support and will be retired in early 2010. A precise date will be announced at least several weeks prior to Jacquard's retirement. Please note:- No software upgrades are planned.
- Failed hardware may not be replaced.
- A major hardware failure may be terminal.
- Users should back up important files to HPSS often.
Survey: Why do you use Jacquard?
Jacquard user survey: Why Do You Use Jacquard? [MORE]Jacquard — Opteron Cluster
Jacquard is a 712-CPU Opteron cluster running a Linux operating system. The machine is named in honor of inventor Joseph Marie Jacquard, whose loom was the first machine to use punch cards to control a sequence of operations.
Jacquard has 356 dual-processor nodes available for scientific calculations. An additional 8 spare nodes will be in service if available. Four nodes are dedicated as login nodes. There are additional I/O and service nodes. Each processor runs at a clock speed of 2.2GHz, and has a theoretical peak performance of 4.4 GFlop/s. Processors on each node share 6GB of memory. The nodes are interconnected with a high-speed InfiniBand network. Shared file storage is provided by a GPFS file system.
| Jacquard Characteristics | |
|---|---|
| Processor type | Opteron 2.2 GHz |
| Processor theoretical peak | 4.4 GFlops/sec |
| Number of application processors | 712 |
| System theoretical peak (computational nodes) | 3.13 TFlops/sec |
| Number of shared-memory application nodes | 356 |
| Processors per node | 2 |
| Physical memory per node | 6 GBytes |
| Usable memory per node | 3-5 GBytes |
| Number of shared-memory spare application nodes (In service when possible.) |
8 |
| Number of login nodes | 4 |
| Switch Interconnect | InfiniBand |
| Switch MPI Unidirectional Latency | 4.5 usec |
| Switch MPI Unidirectional Bandwidth (peak) | 620 MB/s |
| Global shared disk | GPFS |
| Usable disk space | 30 TBytes |
| Batch system | PBS Pro |
Jacquard I/O Subsystem Configuration
- 16 of the 20 I/O servers are reserved for $SCRATCH
- Each node (compute, storage and login) has 4x connection to a 'leaf' IB switch, with 12 nodes per switch.
- Each switch has a 12x connection to 4 "top-level" switches (4 ports per leaf switch).
- Each I/O node has 1 dual-port 2Gbps QLogic Fibre Channel card (400MBps) hooked up to a S2A8500 DDN controller.
- Every DDN contoller is a "couplet" with 8 2Gbps ports which hook up to 4 I/O nodes. There are a total of 5 couplet controllers.
- In total for $SCRATCH there are 32 2 Gbps links to disk and a total of 40 Fibre Channel links to the DDNs.
- There is no failover at each link. The only failover is that two nodes can talk to the same LUNs. If one node goes down access to the disk is not lost although performance will be degraded.
