NERSCPowering Scientific Discovery Since 1974

TOKIO: Total Knowledge of I/O

The Total Knowledge of I/O (TOKIO) project is developing algorithms and a software framework that collects and correlates I/O workload data from production HPC resources at multiple system levels to provide a dramatically clearer view of system behavior, and the causes of behavior, to application scientists, facility operators and computer science researchers in the field. TOKIO is a collaboration between the Lawrence Berkeley and Argonne National Laboratories and is funded by the DOE Office of Science through the Office of Advanced Scientific Computing Research.


The framework combines a multitude of component-level I/O characterization utilities and a scalable collection framework to continuously monitor I/O at various levels including application profiling with Darshan and back-end storage server monitoring using file system-specific tools.

TOKIO scalable collection framework

TOKIO's component-level monitoring and scalable collection framework


Once collected, data is retained on-disk, and views are created that serve as queryable indices of salient measurements across the different component-level monitoring outputs.

TOKIO analysis components

TOKIO analysis framework


These views are then used by analysis modules that present the correlated data in a meaningful way through standard query interfaces for users and applications.


Publications and Presentations

Related Work