DLFM library tools for large scale dynamic applications
DLFM is no longer actively supported on NERSC hardware. Users looking to scale up Python and other dynamic applications and avoid the start up overhead should instead user Shifter.
Large scale Python and other dynamic applications may spend huge time at startup. The DLFM library, developed by Mike Davis at Cray, Inc., is a set of functions that can be incorporated into a dynamically-linked application to provide improved performance during the loading of dynamic libraries when running the application at large scale on Edison. To access this library, do
module load dlfm
- Pilot run with small node count, e.g., using 2 nodes collect the needed shared libraries and python modules imported
- Real run with large number of cores, only one core read in the shared libraries and python imported modules
This figure shows the startup time for Warp, a python application. With DLFM the code can startup in about 5 minutes with 96K cores! The startup time with 48 cores is the ones without using DLFM for comparison.