Parallel Climate Model
Understanding the response of the Earth's climate system to both natural and anthropogenic (man-made) stimuli, together with understanding the natural variability of the climate system, is part of the science base need for rational formulation of energy policy. Developing this knowledge requires a combination of field observations and numerical modeling, as well as theoretical advances. A comprehensive climate system model must describe the behavior of the atmospheric, oceanic, cryospheric and biospheric systems and the interactions between them.
The principal tools used to simulate the behavior of the global atmosphere and oceans are known as general circulation models. Large numbers of cells or degrees of freedom are needed to adequately resolve the circulation on the global scale.
The NCAR Community Climate Model version3 has been coupled with the Parallel Ocean Program (POP) and a sea ice model from the Naval Postgraduate School in a massively parallel computer environment. The target machines are the CRAY T3D/T3E and SGI Origin 2000. With the cooperation and assistance of NCAR's Scientific Computing Division, the Parallel Climate Model (PCM) model code has been ported to the Hewlett-Packard and SGI Origin 2000.
Based on the experience with the NCAR Climate System Model, in order to minimize the initial drift of the coupled system, the ocean/ice can be spun-up with forcing from previous CCM3 runs with prescribed sea surface temperature. This has also been useful in demonstrating and improving the kind of adjustments that occur in the ocean and ice due to coupling the CCM3, without having to run the more expensive coupled system. Lastly, the full system has been run and tested.
Atmospheric Model
The atmospheric component is the massively parallel version of the NCAR Community Climate Model version 3 (CCM3). This model includes the latest versions of radiation, boundary physics, and precipitation physics and is a state-of-the-art atmospheric component. The CCM3 also includes the land surface model (LSM) which takes into account soil physics and vegetation. This model has been coded to run on the T3D/E and SGI Origin 2000.
Ocean Model
The POP model's grid is 384
28832, with an average
resolution of 
degree latitude and longitude with
increased latitudinal resolution near the equator of approximately
degree. Because of the displaced pole, there is
relatively higher horizontal resolution in the eastern North Pacific,
in the Arctic Straits near northern Canada and Greenland, and in the
Gulf Stream area. In addition, the continents and bottom topography
were carefully modified to obtain realistic flow in many regions
throughout the globe. This model is being spun up with observed
surface and subsurface forcing in preparation for coupling. The model
is presently running efficiently on the NERSC CRAY T3E and LANL SGI
Origin 2000. Plans are being made to add more realistic upper ocean
parameterizations to the model. The model in its present form yields
an extraordinary simulation of the Arctic Ocean, tropical Pacific, and
boundary currents, such as the Gulf Stream, with eddies resolved in
most basins (see figures 35-37). Tools have
been developed to interpolate the model output to regular grids. More
documentation on the POP model can be found in Los Alamos report
LA-UR-95-1146 by Richard D. Smith, Samuel Kortas, Bertrand Meitz,
Curvilinear Coordinates for Global Ocean Models, pp 38.
Sea Ice Model
The sea ice component was adapted for PCM from the Arctic model implemented by Yuxia Zhang (NPS) in an eddy-resolving ocean model. The Zhang and Hibler viscous-plastic ice rheology is used for dynamical processes, with thermodynamics from the Semtner and Parkinson-Washington models. The grid is transformed such that the horizontal resolution is constant (27 km), thus avoiding the problem of convergence near the pole as on a latitude-longitude grid. The model is configured for efficient computation on the CRAY T3X architectures, and is being converted for use on the SGI Origin and HP/Convex platforms by the SCD Computational Science Section.
Computing
The PCM has several configurations that can be run on the T3E - mainly, they involve activating/deactivating the various component models. The T3E requirements are then dictated by the specific PCM configuration that has been set-up. For example, the fully coupled PCM is restricted to 64 PEs because CCM3 is restricted to 64 PEs. If the inactive CCM3 version is used to spin-up the ocean and ice systems, is possible to run with 512 PEs with very good scalability.
The Parallel Ocean Program (POP) developed at LANL is one of many
efforts in the Department of Energy's CHAMMP![[*]](http://www.emsl.pnl.gov/images/latex2html/foot_motif.gif)
program aimed at increased
understanding of the earth's climate through the use of modeling and
high performance computing. POP has been used to perform the highest
resolution (
degree on average) global ocean simulation
ever undertaken using the Thinking Machines CM5 computer located at
LANL's Advanced Computing Laboratory (ACL).
POP is a descendant of the Bryan-Cox model that is used frequently in ocean simulations. This earlier model has been substantially improved and adapted for use with massively parallel computers. Some of these improvements are:
A primary motivation for performing such high-resolution simulations is to resolve eddy motions that can play an important role in the dynamics of the ocean.
Project Participants
The following are the scientists/programmers involved in the coupled climate model effort in alphabetical order: J. Arblaster (NCAR), T. Bettge (NCAR), R. Chervin (NCAR), T. Craig (NCAR), J. Dennis (NCAR), J. Dukowicz (LANL), J. Hack (NCAR), S. Hammond (NCAR), E. Hunke (LANL), R. James (NCAR), P. Jones (LANL), R. Loft (NCAR), R. Malone (LANL), M. Maltrud (LANL), W. Maslowski (NPS), J. Meehl (NCAR), A. Semtner (NPS), R. Smith (LANL), G. Strand (NCAR), W. Washington (NCAR), V. Wayland, (NCAR), J. Weatherly (NCAR), D. Williamson (NCAR), and Y. Zhang (NPS).
Recommendations
The major issue is the 64 PE queues. Higher priority (i.e., more time) given to that queue each day would make the simulations much more efficient. To utilize the allocation, it would be necessary to have two 4-hour jobs make it through that queue through the allocation period. Another ocean spin-up phase will be initiated that will use the 128 PE queue.
The PCM requires no special compilers (we use the standard F90), and no special math libraries.
Our memory use on the T3E with 64 PEs is 9 Mwords per PE. We require 4 GB of temporary disk space during each job submission. Disk space is not currently a problem.
Each year we simulate we save, on average (again, depends upon active components), 2 GBytes. We will run in increments of 100-year experiments, so each experiment will be 200 GBytes of storage.
The NERSC T3E is regarded as the workhorse and NCAR is very satisfied with the performance. Long runs have been initiated in the 64 PE and 128 PE queues. NCAR is concentrating on long control and climate change experiments. Extensive simulations are being planned at NERSC over the next 6 months. Further down on the list of things to do is that we expect to bring up version of PCM that can run on IBM SP systems. Finally, Rodney James of NCAR and John Drake of Oak Ridge are updating the parallel version of CCM so that two-dimensional decomposition is used in CCM3. This will allow CCM3 to run efficiently at T42 on more than 64 PEs on a T3E. This is ongoing work.
Finally, concerning our performance on the T3E, the full PCM achieves about 40 Mflops/PE on 64 PEs - that's 2.5 Gflops. Without CCM3, on 512 PEs, NCAR achieves nearly 40x512= 20 Gflops in tests without I/O.
