NERSCPowering Scientific Discovery Since 1974

TOKIO: Total Knowledge of I/O

The Total Knowledge of I/O (TOKIO) project is developing algorithms and a software framework that collects and correlates I/O workload data from production HPC resources at multiple system levels to provide a dramatically clearer view of system behavior, and the causes of behavior, to application scientists, facility operators and computer science researchers in the field. TOKIO is a collaboration between the Lawrence Berkeley and Argonne National Laboratories and is funded by the DOE Office of Science through the Office of Advanced Scientific Computing Research.

Architecture

The framework combines a multitude of component-level I/O characterization utilities and a scalable collection framework to continuously monitor I/O at various levels including application profiling with Darshan and back-end storage server monitoring using file system-specific tools.

TOKIO scalable collection framework

TOKIO's component-level monitoring and scalable collection framework

 

Once collected, data is retained on-disk, and views are created that serve as queryable indices of salient measurements across the different component-level monitoring outputs.

TOKIO analysis components

TOKIO analysis framework

 

These views are then used by analysis modules that present the correlated data in a meaningful way through standard query interfaces for users and applications.

Participants

Publications and Presentations

Related Work