1. Introduction

This tutorial presents an overview of disk input and output (I/O) operations on the NERSC Cray T3E. It assumes you have some working knowledge of high-performance computing, UNIX, compiling environments, Fortran, and some familiarity with the Cray computing environment. Material in this document is presented in Fortran, with some examples in C.

Source code for all the examples can be found on mcurie. First load the training module and then look in the $EXAMPLES directory.

Overview

The Cray T3E is a machine consisting of a large number of autonomous processors. Each processor is essentially a self-contained computer, capable of performing independent I/O. Because of this capability I/O is inherently more complicated than on serial computers.

The purpose of this document is to describe the I/O environment on the NERSC Cray T3E. The goals of this document are to:

• introduce the basic design of the T3E and how it is connected to disks: The T3E is a multi-processor machine in which many processors act independently, but share a common file system.
• explain some of the basic strategies for dealing with I/O in parallel programming: using either one or many processors to perform I/O.
• introduce how Cray uses layers to control and format I/O: In particular, use of the assign command to specify these layers.
• emphasize that the use of the Cray Global I/O layer is the only safe way to write to a single disk file from multiple processors.
• describe how to use disk striping to speed up I/O for large files.

The sections of this tutorial are

Hardware

A simplified overview on the T3E hardware with an emphasis on concepts relevant to I/O.

Strategy

An overview of different strategies for dealing with input/output on the T3E.

Single-PE I/O

You can simplify I/O concerns by using a single PE to perform all disk access.

Multiple-PE I/O

Using multiple PEs for I/O can be convenient and fast, but you must exercise care.

Global I/O

Cray's Global I/O Library is the only way to safely write to a common file from many PEs.

The assign command

The assign command can be used to control how disk files are formatted and accessed

Disk striping

Significant I/O performance gains are possible by using disk striping to write and read files in parallel using multiple disk partitions.