The NERSC GAMESS README file
GAMESS: ab initio Molecular Quantum Chemistry
The benchmark code GAMESS (General Atomic and Molecular Electronic Structure System) performs various ab-initio quantum chemistry simulations. Several kinds of SCF wavefunctions can be computed with correlation, CI, Correlation corrections to these SCF wavefunctions include Configuration Interaction, second order Perturbation Theory, and Coupled-Cluster approaches, as well as the Density Functional Theory approximation. A variety of molecular properties, ranging from simple dipole moments to frequency dependent hyperpolarizabilities may be computed. Many basis sets are stored internally, together with effective core potentials, so all elements up to Radon may be included in molecules. [GAM] The two benchmarks used here test DFT energy and gradient, RHF energy, MP2 energy, and MP2 gradient.
Relation to DOD MOD TI-06
This version of GAMESS is the same as that in the DOD TI-06 (22 November 2004 R1) procurement [TI-06] except that ddi/shmem/ddishm.src and ddi/shmem/ddio3k.src were modified to support shmem targets for Cray and SGI. These modified files are allowed by TI-06. The medium and large test cases correspond to STD and LARGE test cases of TI-06. See Running and Reporting, below, for the differences in reporting results between Ti-06 and NERSC5.
The code consists of about 550,000 lines of Fortran90 code. It includes a communication layer called DDI (Distributed Data Interface) that makes use of highly optimized vendor libraries. BLAS routines are also used.
Relationship to NERSC Workload
GAMESS is available and is used on three NERSC systems: Jacquard, Seaborg, and Bassi. The NERSC workload that uses it is funded by BES.
For NERSC-related procurements please visit the procurement site.
Due to licensing restrictions GAMESS source code cannot be downloaded from the NERSC site. To obtain the code visit the ISUQCG site.
You can obtain the NERSC-5 GAMESS benchmark input data files here (gzip tar file).
The following steps required to build the code.
- Go into the tools directory. Edit the file actvte.code, replace all strings *UNX with 4 blanks and save the file as actvte.f. There are additional vendor specific substitutions to be performed for Cray and NEC - please read the instructions at the top of actvte.code for additional information. Compile actvte.f with a Fortran compiler to produce the GAMESS source preprocessor "activator," actvte.x, e.g.
- Return to the main GAMESS directory. Edit the files compall, comp, and lked, changing the values of the TARGET variable to one appropriate for your architecture. Some targets available include ibm64-sp, amd64, linux-ia64, and cray-x1; the complete list is in the file misc/readme.unix. In the same three files also change the chdir <directory> command to point to where you unpacked the GAMESS source code. Also correct the names and options of the compilers in the comp, and lked scripts. Note that compiler options for a given target architecture can be in many places in the script; e.g., some files are compiled with increased optimization, others with decreased optimization.
- GAMESS provides its own, "plain vanilla" Fortran source code for BLAS. If you have fast vendor provided BLAS routines available, remove the compilation of 'blas' from 'compall' (around line 74) and check that the optimized versions are linked correctly in lked. If you are compiling with long integers and you have a BLAS library that doesn't support these take a look at what happens with "blaswrap" in compall, comp, and lked. The "ibm64" target is an example of how this problem is fixed via blaswrap.
- Go to the ddi directory and edit the compddi script. Change line 17 to the correct architecture. The choices are the same as in step 2, above; common ones might include amd64, compaq-axp, cray-x1, ibm64-sp, or linux-ia64. Change line 34 to be the correct communications architecture; likely choices are "sockets," "mpi," "lapi," or "shmem." Change lines 45-46 to the appropriate values for number of CPUs and number of nodes in your system. Full instructions are in readme.ddi.
- Keep in mind that the compiler used in comp and compall is for single-processor compiles (we think) while the compiler used in compddi and in lked is probably for MPI compiles and is generally a wrapper script such as mpicc.
- Execute the compddi script to compile and build the ddi library. You can edit the compddi script to enable shmem or mpi for an new architecture but you should read the instructions in the ddi.readme file to determine the correct settings. The relocatable object files from this step go in a ddi/obj directory.
- After ddi/libddi.a is produced return to the main directory and execute the script compall to compile the GAMESS source code. After compall is finished, produce a GAMESS executable by executing lked. The executable is called "gamess.00.x" and it is in the same directory.
f77 -o actvte.x -O actvte.f
A description of the directory structure and some important files is given in the following table.
|Directory or File||Description|
|./||Top-level directory with README|
|./src.pmemd||Directory with top-level Makefile|
|./src.pmemd/pmemd||Directory: All source files, Makefile|
|./src.pmemd/Machines||Directory: Contains several Machine.<X> files with machine-specific environment variable definitions.|
|./src.pmemd/Machine||File: soft link to ./src.pmemd/Machines/.<X>|
|./src.pmemd/Machines/X||Directories for various machines containing system-dependent source code (sys_proto.f90 and erfcfun.c|
|exe||Directory: Executable is put here|
The code is run by executing a shell script (provided) called rungms. Instructions:
- Edit rungms. Set the TARGET variable on line 37 to the correct value for your architecture. Set the SCR variable on line 38 to a directory where large temporary files can reside. Set the ERICFMT variable on line 89 to point to the ericfmt.dat file included in top-level directory of the distribution. Also make sure that the path to the GAMESS executable is correct; this will be in several places and depends on your OS. May have to change SMP_SIZE.
- Invoke the application with the following syntax (csh example):
rungms <size>.inp 00 <procs> >& <size>.logwhere <size> is "small," "medium" or "large," and <procs> is the correct concurrency. Example:
rungms small.inp 00 4 >& small.log
Computational nodes employed in the benchmark must be fully-packed; that is, the number GAMESS "computational tasks" must be equal to the number of physical processors on the node. (If you use "generic" MPI-DDI then the number of MPI processes will be double the number of GAMESS "computational tasks."
Almost all arrays in GAMESS are allocated dynamically although some are held in Fortran COMMON.Memory Required By The Sample Problems:
|Small||To be determined GB|
|Medium||To be determined GB|
|Large||To be determined GB|
|XL||To be determined GB|
The minumum memory configuration required to run the problems in each configuration must be reported (OS + buffers + code + data + ...).
There are three input decks supplied in the benchmark directory: small, medium, and large. Their intent is not to measure scalability; in fact, they compute different systems and do different kinds of computations.
The small case should be used for testing. The benchmark timings are required for the medium and large cases. Each case must be executed with a the concurrency shown in the following table.
The GAMESS benchmark directory contains a verify script that you should run on the output from a run. The syntax is
verify <size> <your output file>
Succesful verification requires self checking.
[GAM] GAMESS Home Page http://www.msg.ameslab.gov/GAMESS/
[TI-06] Technology Insertion-06 Documentation http://www.hpcmo.hpc.mil/Htdocs/TI/TI06/index.html