NERSCPowering Scientific Discovery Since 1974

Quantum ESPRESSO/PWscf

Description

Quantum ESPRESSO is an integrated suite of computer codes for electronic structure calculations and materials modeling at the nanoscale. It builds on the electronic structure codes PWscf, PHONON, CP90, FPMD, and Wannier.  It is based on density-functional theory, plane waves, and pseudopotentials (both norm-conserving and ultrasoft).

How to Access Quantum ESPRESSO

NERSC uses modules to manage access to software. To use the default version of Espresso, type:

% module avail espresso

% module load espresso/<version>

Using Quantum ESPRESSO on a Cray System

Recently, we have optimized the hybrid DFT calculations in Quantum ESPRESSO (pw.x) for hybrid OpenMP+MPI applications. These changes are described in our Quantum ESPRESSO case study and available in the module version 6.1 we provide. In order to maximize the pw.x performance, you should use the following example script:

#!/bin/bash
#SBATCH -p regular
#SBATCH -C haswell
#SBATCH -N 2
#SBATCH -t 02:00:00
#SBATCH -J my_job

export OMP_NUM_THREADS=8
export OMP_PLACES=threads
export OMP_PROC_BIND=spread

module load espresso/6.1
srun -n 8 -c 16 --cpu_bind=cores pw.x -nbgrp 8 -input test.in

Here we are running 8 MPI ranks on 2 nodes, thus 4 MPI ranks per node and therefore 2 MPI ranks per socket. Since the total number of logical cores per node is 64, we request 64/4=16 cores in the -c statement. This will make all cores available to the program. Since QE does not benefit much from hyperthreading, we set the number of OpenMP threads to 8 and declare spread thread binding so that we have 1 thread running on each physical processor.

For Xeon Phi 7250 (Knight's Landing), the same script should look like

#!/bin/bash
#SBATCH -p regular
#SBATCH -C knl,quad,cache
#SBATCH -N 2
#SBATCH -t 02:00:00
#SBATCH -J my_job

export OMP_NUM_THREADS=16
export OMP_PLACES=threads
export OMP_PROC_BIND=spread

module load espresso/6.1
srun -n 8 -c 68 --cpu_bind=cores pw.x -nbgrp 8 -input test.in

Note that there are in total 272 logical cores, so running 4 ranks per node results in 272/4=68 available cores per rank and only a quarter can be used reasonably. To make the number of used cores even, i.e. making sure that all tiles are either empty or packed, we are using only 64/4=16 threads per rank. Nevertheless, to allow for proper thread binding, -c is set to 34. 

Concerning band-group parallelization, it is recommended to run one band group per MPI rank. However, please keep in mind that you cannot use more band-group than there are bands in your system, so adjust the number accordingly if you run into issues. Also note that the new implementation is much more efficient, so you might be able to use much fewer nodes and still get the solution within the same wallclock time.

For all routines except pw.x, please run QE in full MPI mode as there is currently no efficient OpenMP implementation available. Therefore, please use the following script (for Haswell, e.g. running ph.x):

#!/bin/bash
#SBATCH -p regular
#SBATCH -C haswell
#SBATCH -N 2
#SBATCH -t 02:00:00
#SBATCH -J my_job

export OMP_NUM_THREADS=1

module load espresso/6.1
srun -n 128 -c 4 --cpu_bind=cores ph.x -input test.in

Pay close to attention to the fact that we explicitly set OMP_NUM_THREADS=1 when running in pure MPI mode. This is optimal when intending to run with only MPI tasks.

independently of what application you want to run and how, submit the job script using the sbatch command, e.g., assuming the job script name is test_espresso.pbs:

% sbatch test_espresso.sl

Documentation

Quantum ESPRESSO Online Manual

Compilation Instructions

Some advanced users may be interested in tweaking the Quantum ESPRESSO build parameters and building QE themselves in their own directory. In order to aid in this process, and to provide a greater degree of transparency, the build instructions for the QE module are listed below. The following procedure was used to build Quantum ESPRESSO versions >5.4 on Cori. In the root QE do:

% ./configure
% cp /usr/common/software/espresso/<version>/make.inc .
% make <application-name, e.g. pw>

Availability

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
Quantum ESPRESSO cori applications/ material sciences 5.2.0 espresso/5.2.0 2015-09-24 2015-09-24
Quantum ESPRESSO cori applications/ material sciences 5.2.1 espresso/5.2.1 2016-01-16 2016-06-29
Quantum ESPRESSO cori applications/ material sciences 5.4.0 espresso/5.4.0 2016-06-29 2016-06-29
Quantum ESPRESSO cori applications/ material sciences 5.4.0-comp espresso/5.4.0-comp 2016-08-05 2016-08-05
Quantum ESPRESSO cori applications/ material sciences 5.4.0-dfti espresso/5.4.0-dfti 2016-08-05
Quantum ESPRESSO cori applications/ material sciences 6.1 espresso/6.1 2017-01-18 2017-01-18
Quantum ESPRESSO edison applications/ material sciences 6.1.0-scalapack espresso/6.1.0-scalapack 2017-01-19 2017-01-19