NERSCPowering Scientific Discovery Since 1974

Access, Compiling and Running Jobs

Access Dirac

Dirac can be accessed by logging into carver.nersc.gov. 

Compile

To compile your code, you need to land on a dirac compute node 1st:

qsub -q dirac_reg -l nodes=1 -l walltime=00:30:00 -I

After you are inside the job, you can load the necessary module for compile:

module unload pgi 
module unload openmpi
module unload cuda
module load gcc-sl6
module load openmpi-gcc-sl6
module load cuda

Now you can compile your code. Note that gcc is the only supported compilier, it supports OpenCL and CUDA. Currently there is no support for PGI, CudaFortran, or OpenACC. 

After compiling you can run your code inside the same interactive job. 

Submitting Batch Job

To submit a batch job, please put the following block to the beginning of your PBS script:

module unload pgi
module unload openmpi
module unload cuda
module load gcc-sl6
module load openmpi-gcc-sl6
module load cuda

Then you can submit your job with qsub.  

Queue Classes

Jobs must be submitted to a valid Submit Queue. Upon submission the job is routed to the appropriate Execution Queue. You can not directly submit a job to an Execution Queue.

Submit QueueNodesAvailable ProcessorsMax WallclockRelative PriorityRun Limit
dirac_int 1 1-8 30 mins 1 1
dirac_reg 1-12 1-96 6 hrs 2 2
dirac_small 1 1-8 6 hrs 2 4
dirac_special 1-48 1-384 Contact consult@nersc.gov to arrange

Special Queue for higher concurrency jobs

For jobs that need more than 32 nodes, please contact consult@nersc.gov with the subject "Special queue request for Dirac". Note that these jobs might take some time to run depending on the load on Dirac.

Quad-Fermi and Quad-Tesla Nodes

To request a node with 4 Fermi C 2050 GPUs use resource mfermiTo request a node with 4 Tesla C 1060 GPUs, use resource mtesla

qsub -I -q dirac_reg -l nodes=1:ppn=8:mfermi
qsub -I -q dirac_reg -l nodes=1:ppn=8:mtesla 

Note that these two nodes only support single node jobs.

Tesla K20Xm and K40c Nodes

There is one Tesla K20Xm node and one Tesla K40c node in dirac, to request please use the k20x/k40 resource, e.g. 

qsub -I -q dirac_reg -l nodes=1:ppn=8:k20x
qsub -I -q dirac_reg -l nodes=1:ppn=8:k40

Local Hard Drive

There is a local hard drive on each node. Access /lscratch.