Reasons for Using Jacquard

• 1. Overview
• 2. Summary of User Responses to "Why do you use Jacquard?"
• 3. Jacquard's Unique Features
• 4. Jacquard 2009 Usage Profile
• 5. Top Projects Using Jacquard
• 6. Case Studies
  • 6.1 Chemistry
  • 6.2 Materials Sciences
  • 6.3 Fusion Energy Research
  • 6.4 Geosciences

1. Overview

In the spring and summer of 2009, NERSC asked members of the top 20 projects using Jacquard "Why do you use Jacquard?" Representatives from 13 projects responded. If you would like to contribute to the survey, we welcome email to ragerber@lbl.gov.

2. Summary of User Responses to "Why do you use Jacquard?"

3. Jacquard's Unique Features

• Full Linux OS on compute nodes.
• 3 GB memory per core (2 GB per core on Franklin)

4. Jacquard 2009 Usage Profile

5. Top Projects Using Jacquard

Repositories with the most usage on Jacquard through the first quarter of the 2009 Allocation Year.

6. Case Studies

6.1 Chemistry

My reason for using Jacquard is that it is the only platform at NERSC on which I can carry out my calculations.

I do electronic structure calculations on rather small molecules, usually involving 5-8 atoms; for example CH4, C2H4, C2H4O, 2(H2O), 3(H2O), that kind of size. For each molecule I need to calculate the electronic energy for many different geometries, several 104 geometries. Each individual calculation takes, depending on the system and on the precise electronic structure method, anywhere from a minute to an hour; 15 minutes is typical. I do many such calculations in parallel, and it is embarrassingly parallel, with one job per CPU, so two jobs per node on Jacquard.

For the calculations I use the Molpro package almost exclusively, although I might also use Gaussian, Aces, NWChem, or any of a number of other electronic structure packages. To the best of my knowledge, whatever package I use, the hardware issues are the same and I could only use Jacquard at NERSC to make progress. The critical hardware issue is that these calculations require disk space for scratch use. I believe it has to be that way, it is a common feature of electronic structure codes; in any, case, the use of disk space is outside my control.

The electronic structure calculation involves two main phases. The first phase is called integral evaluation, and this is where the scratch disk space is involved. The second phase is an iterative solution of large-scale nonlinear equations. I believe that the second phase does not involve disk.

On Jacquard, by special permission of the consultants, I have set things up so that Molpro uses local disk on each node, so the /tmp filesystem. On Franklin this is not available. Without use of local disk my batch of Molpro jobs, even if it is just 32 of them, completely overwhelms the central filesystem, so /scratch on Jacquard, and my calculations grind to a halt. Worse, of course, my jobs grind the /scratch filesystem to a halt for all users. With use of local disk I can run as many jobs in parallel as the scheduler will give to me. (Even so, the small size of /tmp on Jacquard has been a problem for me that prevents me from doing some of my larger calculations at NERSC.)

I do not know any way around this reliance on /tmp, and I hope that the cluster that is now being procured will offer local disk. Certainly the inner workings of Molpro, Gaussian, NWChem, etc., are beyond my control; and in any case, the use of scratch disk space in those codes is for good reasons, as can be seen in that it is a common feature of all such codes.

I often use Jacquard to run direct ab initio molecular dynamics (AIMD) for studying the dynamics and kinetics of radical reactions. In the calculations, we use the home-developed code (DualOrthGT) that calls a quantum chemistry program such as Gaussian (or MolPro). Both DualOrthGT and Gaussian are well parallelized via MPI. Although Franklin provides the same features as Jacquard, Franklin like IBM Blue Gen does not support "Call SYSTEM()" which is the key function to build a link between MD and ab initio codes. Thus, it's rare to use Franklin. Thank you very much for your kindly supports.

There are several reasons for us to run on Jacquard:

1. Jacquard has more scientific software ready to be used, in particular Gaussian 03. Bassi also has it, but the waiting time for the job to run is much longer.
2. Related to the above, Jacquard has long (48-hour) queues, which is ideal for running programs like G03 (fewer nodes but longer running time). I don't think there is a substitute (yet) for Jacquard for this purpose. These Gaussian jobs are too demanding for us to run on local computers (8-16 processors parallel for each job), but are very important for our research on large organic/inorganic molecules. Without Jacquard, a lot of such research could not have been done.
3. Jacquard has more usable memory per node that Franklin.
4. Most times Jacquard is more stable than Franklin.

I use Jacquard exclusively to run Gaussian 03 jobs of small-medium size in parallel mode using Linda. So, the availability and stability of G03 and Linda on Jacquard is the most important factor. The system seems to be more stable than Franklin and more convinient than Bassi. Queue seems to move faster than on Franklin and admits more tasks than Bassi.

I use Gaussian 03, a computational chemistry software package, for the majority of my calculations at NERSC. There are two reasons I like to use Jacquard instead of Bassi (Gaussian is not available on Franklin, Ed.)

1. Jacquard has a shorter wait time.
2. Gaussian 03 does not parallelize well on Bassi. My personal benchmark has shown that jobs parallelize well, under reasonable scaling factor, up to 16 nodes on Jacquard, but scale very badly if one tries to run multi-node G03 calculations on Bassi.

I saw the announcement of replacing Jacquard by Cray's machine. I just worry that I will have to run all of my Gaussian jobs on Bassi (which is planned for retirement January 2010, Ed.), which is limited to one node, once Jacquard is retired. Since Gaussian is one of the most popular quantum chemistry packages in the computational chemistry community, I really hope NERSC can work with Gaussian Inc. together to figure out a way of putting well-parallel Gaussian program on the new cluster.

For me, the most efficient way to use NERSC resources is the following:

6.2 Materials Sciences

I have moved some applications to Franklin. I kept most of my jobs on Jacquard because the charge factor is lower and I usually use fewer than 64 processors. I run large jobs on Franklin, it works great!

We mainly use GROMACS to run Molecular dynamic simulations. We found that GROMACS does not parallelize well beyond 30-40 cores. So, I used Jacquard to run longer simulations (up to 48 hrs) using 32 cores whereas on Franklin I can only run jobs for 24 hrs. That is one of the main advantages.

Secondly, GROMACS was available on Jacquard (I assume that it was compiled to get best efficiency) whereas on Franklin I compiled GROMACS myself and I am not sure how efficient it is.

I can run on Franklin, but the charge rate is lower on Jacquard. The jobs I run are small enough that jacquard is suitable. I run VASP with usually >24 processors. My larger runs are done on Franklin.

The most important things for me are: throughput (mostly, how long is the queue time), availability of VASP, and charge factor.

We use an in-house FORTRAN parallel code to run many jobs investigating the hydration mechanism of different metal ions. We prefer Jacquard because it is about 25% faster than Franklin. This was observed in early 2007 and I had substantial discussion with NERSC staff members. Unfortunately, I did not get very clear explanations. I have not run jobs in Franklin since early 2008 and don't know its current performance. Also, Jacquard is cheaper than Franklin regarding computing time charge.

6.3 Fusion Energy Research

I use Jacquard for small jobs (usually 32 processors) because these jobs have a more rapid turnaround time on Jacquard than on Franklin. I use Franklin for jobs requiring in excess of 100 processors.

Much of the work done on GYRO is done with 32 or 64 processors, but often requires long run times (more than 24 or 48 hours). We generally accomplish this by saving an intermediate phase and resubmitting the job. Naturally, this strategy is facilitated by short wait times in the queue. Emphasis in the PBS algorithm on running large jobs restricts our ability to use the full NERSC capacity. For the 32 processor jobs I run on Jacquard, my needs are well served by increased capacity at the expense of capability. The replacement for Jacquard should emphasize rapid turnaround on small jobs, with a low job size cap (around 100 processors or less). For these needs, a large number of small machines governed by a single batch system would be more than satisfactory. Let users choose between the high capability option (Franklin) and high capacity with low capability.

The other reason I used Jacquard, particularly earlier in the year, was that it was much more stable than Franklin.

The main reason I have been using Jacquard is that I run a lot of small 32 processor jobs. Jacquard has been more reliable and wait time shorter than for Franklin.

6.4 Geosciences

I think there are four main reasons that we use Jacquard as opposed to Bassi, Franklin or DaVinci.

1. We have already done code development, debugging, verification and large scale runs on it;
2. It has more memory per core than Franklin (or Hopper!!!)
3. The two-cores per processor makes it easy for our run to use the entire node. From other clusters, we have experience that the code performance can vary greatly based on which other processes are running on the same node.
4. Machine charge factor is only 60% of Bassi's.