NERSC 3rd Party IPI-3 Project

NERSC 3^rd Party IPI-3 Project

Phase III

3/5/2000

Summary of Changes

and New Development

I. Introduction

This document summarizes the modifications that were made to the HPSS source code in support of the 3^rd Party IPI-3 project. The changes were intentionally kept to a minimum to make it more likely that they would be accepted as part of the HPSS Baseline – the majority of the code to support this capability on Unicos platforms at NERSC was external to HPSS, primarily in the development of the Unicos IPI-3 libraries, the architecture-independent thread libraries, and the complete redesign/rewrite of the internal HSI I/O subsystem. These libraries are described in sections III – V, below.

The HSI I./O subsystem was rewritten to fully support the HPSS parallel I/O capabilities, as described in section VI.. Changes were also made to the non-DCE Client API library to implement the hpss_ReadList and hpss_WriteList calls.

II. HPSS Modifications

As described in the Phase II Feasibility Study, in order to use HIPPI Framing Protocol as User Level in order to do 3^rd Party IPI-3 transfers, the Unicos Client process must first reserve an Upper Layer Protocol identifier to associate inbound HIPPI messages with a particular opened device. The ANSI standard reserves low-numbered IDs, for example, ULP ID 6 and 7 are reserved for IPI-3 Master and Slave messages. Applications normally reserve ULP IDs > between 128 and 255, but this is a configurable parameter (on AIX, this can be changed via SMIT panel, and is settable at compile time for the Unicos IPI-3 ULP reservation library).

The HPSS mover and IPI-3 code changes were made with the following primary design goals in mind:

The changes must be transparent to existing IPI-3 clients. For example, it should not even be necessary to relink the AIX PFTP client in order to make it continue to work as it does currently.
The mover should be able to dynamically determine, on a request-by-request basis whether it is working on a request for a Framing Protocol client or a Master/Slave client, and automatically do the appropriate type of transfer.
The changes should be made in such a way as to make it simple later on to support Slave IPI-3 transfers to AIX HIPPI-attached clients that were not controlling Max Strats.

IPI3_INTERFACE_STRUCT

The IPI3_INTERFACE_STRUCT is currently defined as

typedef struct

{

short interface; /* interface type flag */

char *name; /* client name for data transfers */

}

The interface member currently has only one defined value:

/* supported values for interface type flag

#define IPI3_HIPPI 0 /* IPI3 Third Party over HIPPI */

The format of the "interface" member was redefined to contain 3 logical fields as follows (bit 0 is the rightmost, least-significant bit):

bits 2-0 (3 bits): "Interface type" (values 0-7). A value of 0 continues to mean IPI3_HIPPI (i.e., no change is necessary to existing code).

bits 7-3 (5 bits): Flags to support special features.

bits 15-8 (8 bits): Contents are defined differently, based upon the flag bits

For the IPI-3 project, if bit 7 is set to 1, then the ULP ID to use when sending HIPPI FP messages to the client is contained in bits 15-8.

The coding to support this is shown below:

/* supported values for interface type flag (values are 0-7) */

#define IPI3_IFTYPE_MASK 0x0007 /* mask for extracting the transport type field */

#define IPI3_HIPPI 0 /* IPI3 Third Party over HIPPI */

#define IPI3_FIBERCHANNEL 1 /* IPI3 Third Party over FIBERCHANNEL */

#define IPI3_ETHERNET 2 /* IPI3 Third Party over ETHERNET (e.g. GBit) */

/* flag bit definitions in the "interface" type flag */

#define IPI3_IFTYPE_ULP_PRESENT 0x0080 /* 1: ULP ID present in upper 8 bits */

#define IPI3_IFTYPE_BIT6 0x0040 /* unused */

#define IPI3_IFTYPE_BIT5 0x0020 /* unused */

#define IPI3_IFTYPE_BIT4 0x0010 /* unused */

#define IPI3_IFTYPE_BIT3 0x0008 /* unused */

/* right shift count for extracting the ULP ID from the upper bits of the interface value */

#define IPI3_IFTYPE_ULP_SHIFT 8

typedef struct

{

unsigned short interface; /* interface type flag */

char *name; /* client name for data transfers */

} IPI3_INTERFACE_STRUCT;

Note that a value of 0 for "interface" has the same meaning as it did previously, so existing code and executables will continue to work without requiring any changes.

Slave I/O Transfer Protocol Addition

The IPI-3 Slave interface in the AIX HIPPI driver has the same restrictions as the Max Strat, namely, it expects that ULP ID 6 and 7 are the only ULP IDs used for IPI-3 master/slave messages. This interface does not provide the "Alternate HIPPI-FP" parameter which was used to circumvent the restrictions in the Max Strat. Instead, a new Slave capability was added which makes use of the direct HIPPI Framing Protocol user-level interface provided by the AIX HIPPI driver. When this transport method is used, the slave IPI-3 code opens a direct HIPPI FP connection, obtaining a random ULP ID assigned by the driver. For complex writes, this ULP ID must be transmitted to the client, so that it can direct the data back to the mover using the correct ULP ID.

For Slave I/O transfers, a new IPI-3 protocol parameter was defined:

parameter name: IPI3_ALT_ULPID_PARM

parameter ID: 0xD4

parameter length: 3

All of the remaining HPSS changes described below were to support the use of the alternate ULP ID, if specified by the client. Additional detail regarding the use of the Alternate HIPPI-FP Header Parameter (0xD7) can be found in the Phase II Feasibility Study.

HPSS Changes – Detail

The following source files were modified (starting at the root of the HPSS source tree).

./Makefile.macros

./include/ipi3/ipi3.h

./include/ipi3/ipi3cmd.h

./include/ipi3/ipi3defs.h

./include/ipi3/ipi3en.h

./include/ipi3/ipi3slave.h

./include/hpss_iod.idl

./src/ipi3/ipi3.c

./src/ipi3/ipi3drvr.c

./src/ipi3/ipi3en.c

./src/ipi3/ipi3struct.c

./src/ipi3/ipi3log.c

./src/ipi3/ipi3slave.c

./src/mvr/mvr_clientif.c

./src/mvr/pre.ksh

The changes for each of these files are as follows:

Makefile.macros

Enable IPI drivers

Set other NERSC-specific options (disable DMG, HDM, ...)

include/ipi3/ipi3.h

add prototypes for new ipi3.c functions

ipi3_build_fp_parm() : function to build an alternate HIPPI FP parameter to send to the Max Strat for complex read and write functions

ipi3_build_alt_ulpid_parm() : function to build an "Alternate ULP ID" parameter to include in the TNR for Slave I/O complex writes

ipi3_parse_alt_ulpid_parm : function to parse an inbound "Alternate ULP ID" parameter (0xD4)

include/ipi3/ipi3cmd.h

Add structure definitions for the "alternate ULP ID" structure (IPI3_ALT_ULPID_STRUCT) and for the "Alternate HIPPI-FP Header" structure (IPI3_ALT_FPHDR_PARM_STRUCT)

include/ipi3/ipi3defs.h

Add definitions for the fields in the "interface" member variable in the IPI3_INTERFACE_STRUCT.

In order to avoid sign-extension problems with ULP IDs > 0x80, change the "interface" definition in both the IPI3_INTERFACE_STRUCT and in the IPI3_INTERFACE_STRUCT2 from

short interface;

unsigned short interface;

include/ipi3/ipi3en.h

Add new parameter "interface" to the ipi3_read_cmd() and ipi3_write_cmd() calls, to pass in the interface flags from the IPI-3 request. [These are internal functions that are used only within the IPI3 library].

include/ipi3/ipi3slave.h

Add new parameter "rinterface" (remote interface) to the ipi3_slave_open() function call.

include/hpss_iod.idl

change definition of "Interface" member of ipiaddress structure from

signed16 to unsigned16

src/ipi3/ipi3.c

Add macros IPI_FETCH32 and IPI_STORE32 to fetch and store unsigned long value from packed array of characters

Add function ipi3_build_fp_parm() to build the Alternate HIPPI-FP header to add to a Transfer Notification Response parameter. The Alternate HIPPI-FP header parameter contains the HIPPI FP header, the IPI3 response message, and the TNR to be transmitted to the client (all of the D1 data to be sent to the client by the Max Strat must be contained in this header)..

Add function ipi3_build_alt_ulpid_parm() to build the IPI3_ALT_ULPID_PARM parameter for output slave I/O complex write messages.

Add function ipi3_parse_alt_ulpid_parm() to parse received IPI3_ALT_ULPID_PARM parameters.

src/ipi3/ipi3drvr.c

Add "interface" parameter to ipi3_part_printf debug messages

Change checks for IPI3_HIPPI interface type to mask off the value using the new IPI3_IFTYPE_MASK definition

Add the "interface" parameter to the calls to ipi3_read_cmd() and ipi3_write_cmd()

src/ipi3/ipi3en.c

Function ipi3_read_cmd() – add new "interface" parameter to calling sequence. This parameter is passed from the client to the mover, which in turn passes it to the ipi3 library. The new code checks the IPI3_IFTYPE_ULP_PRESENT flag in the passed-in interface; if set, it calls ipi3_build_fp_parm() to add an alternate FP Header parameter to the request which it is sending to the device.

Function ipi3_write_cmd() - same as ipi3_read_cmd() changes

src/ipi3/ipi3struct.c

Function ipi3_client_conf() – mask off the interface type bits using IPI3_IFTYPE_MASK when searching for a specified device and client

src/ipi3/ipi3log.c

Function ipi3_log_command() – fix segmentation violation bug with printing long parameter(s).

Function ipi3_log_response() – same problem

src/ipi3/ipi3slave.c

Ad definition for HIPPI framing protocol device and max delay when waiting for complex write response. [Note: this might be better defined in a configuration file, e.g. ipi3slave.conf].

Add fields to the ipi3_slave[] structure to track:

whether ipi3Slavexxx() calls or FPxxx() calls are being used for the transfer

the interface type flag

the remote ULP ID [if direct Framing Protocol transfers are being used]

the local ULP ID [if direct FP transfers are being used]

Function ipi3_slave_open() – add "rinterface" (remote interface) parameter. This parameter is the same as the one passed in for non-slave transfers (originally sent by the client). If this parameter has the PI3_IFTYPE_ULP_PRESENT bit set, then the code opens a direct HIPPI Framing Protocol transport connection, using the ULP_ANON ULP ID (which allows the driver to select any available ULP ID from the allowable set as configured for the device).

Function ipi3_slave_close(): issue FPClose() if device was opened with FPOpen()

Function ipi3_slave_async_io(): use IPI3_IFTYPE_MASK when checking interface type

Function ipi3_slave_async_iov(): use IPI3_IFTYPE_MASK when checking interface type

Function ipi3_slave_next_io()

Send local ULP ID as part of the TNR if using direct HIPPI-FP, and this is a complex write

Call FPInitiateSend()or ipiSlaveInitiateTransfer()depending on how the device was opened (note: since HPSS does not issue IOV-style requests, FPInitiateSend() code was not added for the IOV case.

Function ipi3_slave_wait():

If using framing protocol, issue FPComplete() call vs.ipiSlaveWait() call

If this was a complex write operation, then wait for IPI_FP_MAX_WRITE_DELAY seconds for the client to send the data associated with the write, using FPGetPacket() calls. to retrieve the inbound data, and select() calls to delay

src/mvr/mvr_clientif.c

Function OpenClientConnection(): Add "Addr_u.IPIAddr.IPI3Addr.Interface" to _IODEBUG messages

Include AddrPtr->addr_u.IPIAddr.IPI3Addr.Interface in call to ipi3_slave_open()

src/mvr/pre.ksh

function Evaluate: Add library "fplink" to "OTHER_FLAGS" in the code to handle bool_IPI –eq 1

III. Non-DCE Library Changes

The Gleicher Enterprises-supported non-DCE client library (NDAPI) was augmented with the following capabilities:

support for the hpss_Readlist and hpss_Writelist calls
new XDR functions to pass the IOD/IOR structure and the various address structures.
all functions renamed to avoid collisions with identically-named functions which were introduced in HPSS for the DMAP Gateway
additional functions were defined to move the automatic Class of Service selection from HSI into the non-DCE server in order to support the capability for a single HSI binary to logon to multiple unrelated HPSS systems and automatically select the proper COS as defined by the remote site.

IV. Threads Library

In order to support the new multi-threaded HSI on architectures other than Unicos, an architecture-independent thread library was developed, to wrap the native thread implementation on the architectures for which HSI is supported. The names of all data types and functions in the library are prefixed by "ai_", e.g. "ai_thread_abort()", [Architecture Independent]. The thread library includes wrappers for:

architectures which are fully POSIX compliant
architectures which include a subset of the POSIX definitions
Mach "cthreads" [for the OpenStep and MacOSX operating systems]

Only a subset of the pthreads capability, as required for HSI, was implemented. This basic capability includes:

definitions for primitive data types [ai_addr_t, ai_condition_t, ai_condattr_t, ai_mutex_t, ai_mutexattr_t, ai_thread_t, ai_thread_attr_t]

thread control [ai_thread_abort, ai_thread_detach, ai_thread_exit, ai_thread_create, ai_thread_equal, ai_thread_join, ai_thread_self, ai_thread_sleep, ai_thread_yield]

mutex control [ai_mutex_destroy, ai_mutex_init, ai_mutex_lock, ai_mutex_try_lock, ai_mutex_unlock]

condition variable control [ai_condition_broadcast, ai_condition_init, ai_condition_signal, ai_condition_wait]

The threads library was adapted from the library developed for the Feasibility Study during Phase 2 of the NERSC IPI-3 Project. Only functions which were available in Unicos were implemented, since this was the primary platform to be supported by the new HSI.

V. Unicos IPI-3 Libraries

The Unicos version of the IPI-3 library, as developed for phase II of this project, was modified to support the IPI-3 Interface Struct changes described above. These changes were nearly identical to the changes made in the HPSS version of the IPI-3 library.

Administrative Controls.

A requirement for this project was the ability to disable either IPI-3 reads or writes if a software problem was suspected, or hardware problems require disabling the HIPPI channel or one of the HIPPI switches.

For the Unicos implementation, a new module, called ipi3AdminCtl.c, was added to the Unicos library. This module contains administrative control functions to allow the Unicos application to determine if IPI-3 transfers have been disabled. The following control flags are defined:

IPI3_READ_ENABLED - determines if IPI-3 reads are enabled

IPI3_WRITE_ENABLED – determines if IPI-3 writes are enabled.

These controls are implemented by dynamically checking for the presence of a file, which can be created by using the Unix touch(1) command. Unless the file(s) are present, it is assumed that IPI-3 reads and writes are both enabled. The filenames are defined in the Makefile for the IPI-3 library at compile time, as follows:

IPI3_ADMIN_MASTER_DISABLE_FILE : if this file exists, both IPI-3 reads and writes are disabled

IPI3_ADMIN_READ_DISABLE_FILE: if this file exists, IPI-3 reads are disabled

IPI3_ADMIN_WRITE_DISABLE_FILE: if this file exists, IPI-3 writes are disabled.

To re-enable reads or writes, it is sufficient to rename or remove the administrative control file.

The administrative capability is conditionally compiled into the IPI-3 library for the Unicos operating system, and the code to check the state of the administrative controls is conditionally compiled in the Parallel I/O file transfer code within HSI. Prior to the beginning of each new file transfer, the code checks the state of the controls, and will only perform IPI-3 transfers if the appropriate control (read or write) is still enabled.

AIX Library Compatibility

A goal of this project was to implement a Unicos library using the same IPI-3 function names and parameters as the existing AIX library. This was accomplished by the development of a new module which implements the "ipiMasterXXX" functions as a separate thread, which is invoked when an IPI-3 device is first opened. The new thread assumes the role of the AIX OS HIPPI driver, processing requests passed to it via a signaling mechanism in the main thread.

A restriction which was added for the Unicos implementation is that the I/O buffer which the application uses for IPI-3 I/O MUST be allocated via a call to an internal IPI-3 library function. This function allocates additional space to accommodate the extra space required for the I-field and FP Header, as described in the Phase II report. The new functions which were added are:

ipi3_buf_malloc - allocates a buffer

ipi3_buf_free – deallocates a buffer

ipi3_buf_ok – internal function to verify whether a buffer was allocated by ipi3_buf_malloc() call.

With the exception of the requirement to call ipi3_buf_malloc/ipi3_free_buf, the Unicos IPI-3 library is compatible with the AIX version. The HPSS PFTP client uses only a subset of the AIX IPI-3 library calls which are available, and only that subset was actually implemented in the Unicos version of the library. The remaining functions were implemented as stubs that can be expanded, if necessary, in the future.

VI. ULP Library

The Upper Layer Protocol library, used to manage the assignment of ULP IDs (using advisory locks on a globally-shared file in the Unicos file system), as developed in phase II of this project, was used without modification. No code changes to PFTP or HSI were necessary to make use of this library; all calls are internal to the IPI-3 master thread.

Two goals of this library were:

Control access via a locking mechanism to the ULP reservation file
Provide an automatic cleanup capability for the case where a ULP ID was reserved by an application that terminated without releasing the reservation.

These goals were met by implementing an advisory lock on the ULP reservation file, which is a small fixed-length file containing one entry for each possible ULP assignment (max of 256). This file is created, if it does not exist, by the first program which attempts to open a HIPPI device. Each time a new ULP reservation is requested, the library verifies that each reserved ULP ID is owned by a process that is still active. If it is determined that an entry belongs to a process that no longer exists, the entry is reassigned to the requesting process.

VII. HSI Changes

The HSI I/O subsystem was completely rewritten, involving approximately 7000 lines of new or changed code (including comments) in HSI.. The existing file transfer function (fileio.c) was replaced by modules to support the following new capabilities:

multithreaded I/O
full support of the HPSS mover protocol for parallel transfers
automatic detection of the appropriate stripe width and buffering parameters as configured for the Class of Service for the files being read or written
automatic use of parallel transfer mode vs. sequential transfer mode for piped files
addition of recursive copy within HPSS, in addition to the existing recursive get/put capabilities
ability to interactively pipe files into and out of HPSS, e.g., by using the "tar" command

VIII. Issues

During the testing of the new code, two problems were encountered which must be resolved before the new HSI can be put into production. Both of these problems are currently being investigated.

Corrupted I-field – In order to perform complex writes, the Ifield that is received as part of the Transfer Notification Response is used as a return addresses by complementing the <direction> bit. Under some as-yet undetermined conditions, the inbound Ifield received by the Master IPI Thread has a source address of zero, which causes the write to fail since the data is being written to an illegal switch address.

It is unknown at this time whether the problem is caused by a malfunctioning HIPPI switch, or by a problem in the Cray Gigaring software in the HPN.

Timeout problem – both IPI-3 reads and writes on the heavily loaded NERSC Cray systems occasionally time out and cause the transfers to fail. The timeouts occur more often a the I/O buffer size increases, particularly on the T3E. As described in the phase II study, the Cray operating system parameters may need to be adjusted to allow HSI to either be locked into memory, or otherwise be given priority in the CPU scheduler, in order to ensure that it can respond in a timely manner to read and write requests from the Maximum Strategy disk array.

Conclusion

The IPI-3 interface has been tested, using the new version of HSI, on both the J90 and T3E Unicos systems at NERSC. Significant improvements in I/O to and from HPSS have been observed on both of these architectures, particularly on the J90. Transfer rate improvements on the order of 10X have been achieved on the J90, despite the heavy loading on the system. In addition, the new HSI parallel I/O features have been tested on both DCE and non-DCE AIX platforms, Apple Mac OSX, HPUX, Sun Solaris, and SGI IRIX platforms; all architectures have demonstrated significantly improved I/O performance. .

The HPSS modifications may possibly be included in the HPSS baseline. An informal peer review of the changes has taken place with one of the primary HPSS mover developers , with positive results. In order to include the changes as part of the baseline, it will be necessary to go through the formal HPSS requirement and review/test process.

The primary goals, as set forth in the initial phase I requirements, have been met. In addition, other significant benefits have been realized as a result of this project:

The implementation of the IPI-3 Slave transfers enables 3^rd-party IPI-3 transfers to HIPPI-attached mover nodes which are controlling devices other than Maximum Strategy disk array, such as tape drives, SCSI or Fiberchannel-attached local disk drives, etc.
The DCE version of HSI, which runs on AIX platforms, now supports native IPI-3 transfers on nodes which are HIPPI-attached and fully supports parallel transfers for socket-based transfers.
The parallel I/O code developed for HSI can be adapted for use in the NDAPI server to provide high-speed parallel transfers between unrelated HPSS systems, a capability which is planned for the PROBE project.
It is anticipated that the new IPI-3 capabilities will be portable without significant effort to other architectures which support user-level HIPPI Framing Protocol access. Potential candidates include both Sun and SGI.

References