NERSCPowering Scientific Discovery Since 1974

IMPACT_T

General Description

IMPACT-T (Integrated Map and Particle Accelerator Tracking-Time) is a parallel, three-dimensional, quasi-static beam dynamics code used to study dynamics in photoinjectors and RF linear accelerators. Developed under the US DOE Scientific Discovery through Advanced Computing (SciDAC) program, it uses a Particle-in-Cell method to model the space-charge forces of the beam(s) self-consistently. It solves the electro-static Poisson equation in the moving beam frame and allows an arbitrary overlap of the fields for a variety of beam-line element types. It also includes an integrated Green function method, multiple energy bins for beams with large energy spread, and models for treating RF standing wave and traveling wave structures.

IMPACT-T is one member of a suite of computational tools for the prediction and performance enhancement of accelerators. These tools are intended to predict the dynamics of beams in the presence of optical elements and space charge forces, the calculation of electromagnetic modes and wakefields of cavities, the cooling induced by co-moving beams, and the acceleration of beams by intense fields in plasmas generated by beams or lasers. The Impact-T code uses a parallel, relativistic PIC method for simulating 3-D beam dynamics using time as the independent variable. (The design of RF accelerators is normally performed using position, either the arc length or z-direction, rather than the time, as the independent variable.) IMPACT-T includes the arbitrary overlap of fields from a comprehensive set of beamline elements, can model both standing and traveling wave structure, and can include a wide variety of external magnetic focusing elements such as solenoids, dipoles, quadrupoles, etc.

IMPACT-T is unique in the accelerator modeling field in several ways. It uses space-charge solvers based on an integrated Green function to efficiently and accurately treat beams with large aspect ratios, and a shifted Green function to efficiently treat image charge effects of a cathode. It uses energy binning in the space-charge calculation to model beams with large energy spread. It also has a flexible data structure (implemented in object-oriented Fortran) that allows particles to be stored in containers with common characteristics; for photoinjector simulations the containers represent multiple slices, but in other applications they could correspond, e.g., to particles of different species.

The benchmark problem is for a photoelectron beam transported through a section of a photoinjector similar to the one at the Stanford Linac Coherent Light Source. It describes a cylindrical electron beam initially 1.2mm in radius and 2 mm long, initial energy 1 eV, going through a 1.6 cell RF cavity for acceleration and solenoid focusing and then through a traveling wave structure acceleration to reach a 62-MeV final energy.

Coding

The code consists of about 33,000 lines of Fortran90 in 36 files in a single source subdirectory.

MPI is required. None of the files are preprocessed; all have the '.f90' suffix. The FFTW library is also required.

64-bit floating-point precision is used for most physical quantities.

IMPACT-T uses an object-oriented Fortran90 coding style that is critical for providing some of its key features. For example, it has a flexible data structure that allows particles to be stored in "containers" with common characteristics such as particle species or photoinjector slices; this allows binning in the space-charge calculation for beams with large energy spreads.

Authorship

IMPACT-T is part of a suite of accelerator modeling codes developed by Ji Qiang and Robert Ryne of Lawrence Berkeley National Laboratory.

Relationship to NERSC Workload

Particle accelerator research is funded at NERSC by the DOE Office of Science (DOE/SC) High Energy Physics (HEP) and Nuclear Physics (NP) programs. Its direct allocation from those sources amounted to about 7% of the NERSC cycles in 2007 but it contributes to other areas of importance, such as Fusion Energy Science (FES) and Basic Energy Science (BES). A DOE INCITE grant entitled "Beam Delivery System Optimization for X-Ray Free Electron Lasers" was awarded in 2008 for 5 million MPP hours, most of which will be consumed by codes similar to IMPACT-T. NERSC resources were used to perform simulations of the first 1000 turns (~2 ms) after injection of the Fermilab Booster, using a fully 3D space charge model. The simulation results were then compared with results from Booster experiments and made significant contributions in understanding and calibrating Booster diagnostic devices. Simulation studies are now being used to help optimize the injected beam parameters to achieve a matched beam in the presence of space charge and large momentum spread.

Past applications of the IMPACT code suite include the SNS linac modeling, the J-PARC linac commissioning, the RIA driver linac design, the CERN superconducting linac design, the LEDA halo experiment, the BNL photoinjector, and the FNAL NICADD photoinjector. Recent code enhancements have expanded IMPACT-T's applicability, and it is now being used (along with other codes) on the LCLS project and the FERMI@Elettra project. A recent DOE INCITE grant allowed IMPACT to be used in the design of free electron lasers.

Parallelization

The IMPACT code was developed as a parallel particle in- cell code using a domain decomposition method. In this method, the spatial computational domain is divided and assigned to individual processors. Macroparticles with physical position in that computational domain are assigned to that processor. During each time step, when a particle moves out of the original computational domain, it will be sent to the other processor owning that spatial domain containing the particle's new position. This method has the advantage of avoiding global communication during the charge deposition to obtain the charge density distribution on the grid and during the field interpolation to obtain the space-charge fields from the grid. Only neighboring communication is needed in these stages and in the particle moving stage.

However, this method may suffer from unbalanced work since each processor may have a different number of macroparticles for a non-uniform particle distribution. In order tomaintain a roughly equal number of particles on each processor, the local computational domain boundary of each processor has to adjusted frequently. This introduces extra computational cost and may not work well for some types of distributions. The benchmark problem provided here does frequent load balance updates.

Obtaining the Code

IMPACT-T was used at NERSC expressly for the purpose of supercomputer performance evaluation related to the NERSC-6 acquisition project.  If you are interested in IMPACT-T for accelerator simulation studies you must contact the code's author.  More information is available on the IMPACT-T web page.

Building the Code

Makefiles for a few machines and all the source files are included in the 'source' subdirectory. Change to that subdirectory and type, e.g., make -f Makefile.franklin. The build procedure will produce the executable "xmain" there.

Running the Code

All input data files and sample run scripts are in the "benchmark" subdirectory. To run the code type something like "mpirun -np 256 xmain" and "mpirun -np 1024 xmain."

IMPACT-T requires an input file that contains certain run parameters (grid size, # of particles, # of timesteps, etc.), beamline characteristics, and also a 2-D logical processor configuration. The input file called, ImpactT.in must be in the same directory as the executable.

Timing Issues

The code is well instrumented to separate out timings for various portions of the calculation but the benchmark time is reported as "NERSC_time."

Storage Issues

Memory Required By The Sample Problems:

Medium

y GB

Large

z GB

Required Runs

The same executable is used for the two required runs so "strong" scaling will be measured. The NERSC-6 "base case" runs are for 256- and 1024-process configurations. Separate input files are provided for these.

If you want to run other concurrencies you need to change the values of onpcol,onprow on the first line of the input file to the corresponding values for the 2-D processor decomposition you will use.

Verifying Results

The code checks itself after the last time step and compares a computed value with the corresponding value calculated on the Cray XT4 system at NERSC using the PG compiler.

Modification Record

Record of Formal Questions and Answers

No entries as yet.

Bibliography

Y. Ding, Z. Huang, C. Limborg-Deprey, and J. Qiang, LCLS Beam Dynamics Studies with the 3-D Parallel Impact-T Code, Stanford Liniear Accelerator publication SLAC-PUB-12673, July, 2007.

Ji Qiang Ilya v. Pogorelov, and Robert D. Ryne, "Parallel Beam Dynamics Simulation Tools for Future Light Source Linac Modeling," Lawrence Berkeley National Laboratory Report LBNL-62109, 2007.

Ji Qiang, Steve Lidia, Robert D. Ryne, and Cecile Limborg-Deprey, "Three-dimensional quasistatic model for high brightness beam dynamics simulation," Phys. Rev. Special Topics - Accel. Beams 9, 044204, (2006).

Ji. Qiang, "IMPACT-T User Guide," LBNL Report LBNL-62326, 2007.

J. Qiang I.V. Pogorelov, and R. Ryne, "Recent Improvements to the IMPACT-T Parallel Particle Tracking Code," LBNL Report LBNL-62010, 2006.

J Qiang, M Borland, A Kabel, R Li, R Ryne, E Stern, Y Wang, H Wasserman and Y Zhang, "SciDAC advances in beam dynamics simulation: from light sources to colliders," Journal of Physics: Conference Series 125 (2008) 012004.