Bassi Quick Start Guide for Seaborg Users

Bassi is an IBM parallel computer system and its computing environment is very similiar to Seaborg. If your code compiles and runs on Seaborg, you may have very little trouble doing the same on Bassi.

There are some differences and the most important ones are described here. After consideration of these items you should be running quickly on Bassi.

Passwords and Default Shells

Your Bassi password is your NIM password. Password and login shell changes must be done via NIM. Shell changes may take up to two hours to take effect. This procedure is different than on Seaborg, but it is the same as that used for Jacquard, PDSF, and DaVinci.

Compiling and Linking

The recommended standard compiler options are almost the same as on Seaborg: -qtune=auto, -qarch=auto -O3. Two examples:

%  mpxlf90_r  -qtune=auto -qarch=auto -O3 -c source_file.f
%  mpcc_r -qtune=auto -qarch=auto -O3 -c source_file.f

Most, if not all, codes will benefit greatly from using the "large page" memory available on the POWER 5 (see Programming on Bassi). Large page use is enabled automatically for parallel codes in the default NERSC environment. We have not observed any HPC application that suffers from using large pages, however you can experiment by disabling large page support by linking with the option "-bnolpdata".

64-bit Environment

The default environment on Bassi is 64-bit; on Seaborg the default is 32-bit. The environment variable OBJECT_MODE is set to 64, which makes all code builds produce 64-bit executables. Bassi performs best in 64-bit mode. This difference should be transparent to most to most users. Bassi is binary compatible with Seaborg, but if you try to move libraries and objects, you may encounter 32/64-bit issues.

If you are moving code from Seaborg to Bassi, check your makefile and remove linker option that manipulate the size of data and stack memory segments. In most cases you do not want to use the -bmaxstack and -bmaxdata options; they will only limit the amount of memory you can access in 64-bit mode.

If you must build 32-bit executables, set OBJECT_MODE=32 first. Most NERSC libraries contain both 64-bit and 32-bit symbols for linking against. However, for some libraries this is not possible (e.g. netCDF and HDF5). In these cases you must link against the separate 32-bit versions. These libraries have separately named 32-bit modules you can load (e.g. hdf5_32).

Running Jobs

The following environment variables are set by default, because we think they will benefit most codes. The default values may be adjusted as we gain experience with the machines. You may override any of these values, but first see Running Jobs: Runtime Options for more information.

MEMORY_AFFINITY=MCM
MP_TASK_AFFINITY=MCM
MP_SHARED_MEMORY=yes
MP_USE_BULK_XFER=yes
OBJECT_MODE=64

OpenMP Codes

When running codes that use OpenMP, unset the following environment variable before running your code:

MP_TASK_AFFINITY

Batch Scripts

There are very few differences between Bassi and Seaborg batch scripts. The LoadLeveler classes on Bassi have different limits than on Seaborg. See Class Info and Policies.

Bassi has 8 CPUs per node, not 16 like Seaborg. If you specify tasks_per_node, it must be set to no more than 8.

The dual-link HPS network adapter is specified as sn_all on Bassi. On Seaborg you use csss. For backward compatibility with LoadLeveler scripts, csss is equivalent to sn_all. You may use either of the following in LoadLeveler scripts to run MPI jobs over HPS on Bassi:

#@network.MPI = sn_all,not_shared,US
#@network.MPI = csss,not_shared,US

File Systems

The Bassi file systems are similar to those on Seaborg. You have a $HOME directory and a $SCRATCH directory. The /projects center-wide file system is also mounted on Bassi.

Quotas on the $HOME file system are 5 GB space and 7,500 inodes. Your $HOME directory and your default file group is your personal file group, which has the same name as your login name. On Seaborg, your default file group has the same name as your default repository name.