R.K.'s NetCDF pages
This page documents some of the findings, recommendations, notes, and procedures regarding NERSC's involvement with NetCDF.
Currently, a major milestone has been reached. The netCDF library can now allow a subset of PEs to open a netCDF dataset when using the SGI/Cray ``global'' FFIO layer. Prior to this, either all the PEs had to open the file, or not use global FFIO.
The library performs without incident on the T3E and in all test cases so far. However, there are some practices that need to be followed as documented in the usage section.
The first time, you need to check-out the sources from CVS you need to do the following:
It's important to do the configure soon after the check-out, because the .cvsrc file defines the proper compiler environment variables to use. If you don't do this, then the configure script will find the minimum configuration (i.e. fort77, c89, and no C++ compiler). These minimal compilers are guarenteed by POSIX compliance, but do not have the full optimization capabilities as the standard Cray compilers. Hence, you need to define certain environment variables, which I've done for you in .cvsrc. The .cvsrc should look something like this for a T3E:
# setenv CVSROOT /usr/local/pkg/usg/netcdf3.4x/cvs/ # # set these environment variables prior to doing "configure" # # sn6602 t3e 2.0.2.15 unicosmk CRAY T3E (from INSTALL file) # Cray Standard C Version 5.0.3.0 (d29p35m275a35) Mar 7 1998 00:42:04 # Cray CF90 Version 2.0.3.4 03/07/98 00:42:05 # Cray C++ Version 2.0.3 03/07/98 00:42:06 setenv CC cc setenv CFLAGS -O3 setenv FC f90 setenv FFLAGS '-g -F -M1110' setenv CXX CC setenv CXXFLAGS "-h char" # needed with this version setenv CXXFLAGS "" #RKO> the above cause warnings!? #'C' compiler still chokes on libsrc/putget.c # c89 -c -O3 -I. -DNDEBUG putget.c # cc-7951 c89: LIMIT File = putget.c, Line = 6617 # Insufficient memory is available for compiler to continue. #(56% of the way through the file. This is with a pragma to #prevent inlining of odo1.)
The CVSROOT environment variable is only necessary if you're
using the NERSC sources under CVS control. The key items are the other
environment variables. If these are not set then configure
will not intelligently find the optimal compilers, and will default
to fort77 and c89.
Modification History
I'll generally send email out to the email list
and with the details on how to update things.
I've applied a couple of patches and made several code modifications.
The following list details some the changes from the base library.
cvs tag: unidata
This patch can be found at $NETCDF_DIR/macros.make.ffflush.patch and applied by going into the src directory and doing--- macros.make 1998/06/25 19:30:47 1.1 +++ macros.make 1998/06/25 19:31:24 @@ -21,7 +21,7 @@ M4 = m4 M4FLAGS = -B10000 CPP = cc -E -CPPFLAGS = $(INCLUDES) $(DEFINES) -DNDEBUG +CPPFLAGS = $(INCLUDES) $(DEFINES) -DNDEBUG -DNOFFFLUSH FPP = FPPFLAGS = CXXCPPFLAGS = $(CPPFLAGS)
You will also need to patch the macros.make file like this:
--- macros.make.Thu Jul 16 11:53:49 1998 +++ macros.make Tue Aug 18 10:49:08 1998 @@ -21,7 +21,7 @@ M4 = m4 M4FLAGS = -B10000 CPP = cc -E -CPPFLAGS = $(INCLUDES) $(DEFINES) -DNDEBUG +CPPFLAGS = $(INCLUDES) $(DEFINES) -DNDEBUG -DLOCKNUMREC FPP = FPPFLAGS = CXXCPPFLAGS = $(CPPFLAGS)
- int nc_base_pe(const char path[], int pe)
- Notifies netCDF which PE to use as the base for MPP use. This is only necessary if using the SGI/Cray ``global'' Flexible File I/O layer and desire to have only a subset of PEs to open the specif- ic netCDF file. The path given must be identical to that given in a following nc_open() or nc_create(). To use this feature, you need to specify a communicator group and call glio_group_mpi() or glio_group_shmem() prior to the netCDF nc_open() and nc_create() calls.
- int nc_inq_base_pe(int ncid, int* pe)
- Inquires of the netCDF dataset which PE is being used as the base for MPP use. This is safe to use at any time.
- int nc_set_base_pe(int ncid, int pe)
- Resets the base PE for the netCDF dataset. Only perform this operation when the affected communicator group syncronizes before and after the call. This operation is very risky and should only be contemplated under only the most extreme cases.
Look at the sources in: $USGROOT/examples/netcdf/doubleu/ for a working example. The ``odd'' and ``even'' PEs open separate netCDF files and add records along the unlimited dimension axis. Be sure to look at the README file and use the Makefile.
The basic recipe is:
/* define a communicator group */
...
/* notify the ``global'' layer */
glio_group_{mpi|shmem}(mpi_or_shmem_communicator);
/* notify netCDF to tag which PE to use as the ``base'' */
nc_base_pe(filename, basepe);
/* open or create the netCDF dataset */
stat = nc_open(filename, NC_WRITE, &ncid);
...
/* netCDF I/O restricted to only those PEs tagged in the
communicator group */
...
The output will look something like this:cvs status | grep Status
The status Needs Patch says that the base file has been modified and you can retrieve it by doing ``cvs update file_name''.File: nc.h Status: Up-to-date File: ncconfig.in Status: Up-to-date File: ncio.c Status: Locally Modified File: ncio.h Status: Needs Patch ...
cvs log file_name
cvs status -v any_file
This module will set up the necessary environment variables, most notably: $NETCDF and $NETCDF_DIR, which defines the libraries to link with and the ``root'' of the netcdf directory.module load USG netcdf3.4x # most up-to-date module load USG netcdf3.4x_noffflush # avoids ffflush() module load USG netcdf3.4x_par # uses PARIO library
Then to link with this library:
linker options... $NETCDF
To find out further info regarding the modules do:
module help netcdf3.4x # show message module display netcdf3.4x # display environment variables, etc.
| R.K. Owen | rkowen@nersc.gov |
| Chris Ding | cding@nersc.gov |
| Steve Luzmoor | luzmoor@cray.com |
| Harsh Anand | harsh@nersc.gov |
| Mark Zeng | markz@nersc.gov |
| Jonathan Carter | jcarter@nersc.gov |
| Venkatramani Balaji | vb@gfdl.gov |
| Glenn Davis | davis@binnie.unidata.ucar.edu |