NERSCPowering Scientific Discovery Since 1974

qdo

Description and Overview

QDO (kew-doo) is a toolkit for managing many many small tasks within a larger batch framework. QDO separates the queue of tasks to perform from the batch jobs that actually perform the tasks. This simplifies managing tasks as a group, and provides greater flexibility for scaling batch worker jobs up and down or adding additional tasks to the queue even after workers have started processing them.

The qdo module provides an API for interacting with task queues. The qdo script uses this same API to provide a command line interface that can be used interchangeably with the python API. Run "qdo -h" to see the task line options.

How to Access qdo on Edison/Hopper/Carver

#- On edison, hopper, or carver:
module use /project/projectdirs/cosmo/software/modules/$NERSC_HOST
module load qdo/0.5
pydoc qdo
qdo --help

  

 

Further Information

qdo was developed to support workflows in processing images for cosmological surveys. For more information please contact Stephen Bailey in the Cosmology Group at Lawrence Berkeley Lab.