ATP 2.1.1 ========= Release Date: ------------- July 06, 2017 Release Purpose: ---------------- - Update MRNet with UW scaling fixes - Update to Stackwalker/Dyninst 9.3.2 - Enabled enhanced debugging (for tool maintenance). Bugs closed with this release: ------------------------------ BUG 844440 Theta - STAT Problems with 1024 Job - This resolves the ATP part of 844440. Product and OS Dependencies: ---------------------------- The Cray ATP 2.1.1 release is supported on the following Cray systems running Cray Linux Environment(CLE) operating system - Cray XE and XK systems with CLE version CLE 5.2 Running either ALPS or SLURM 14.11 - 16.05 - Cray XC systems with CLE version 5.1 or later Running either ALPS or SLURM 14.11 - 16.05 Newer versions of SLURM likely will work, but have not been tested. Product Dependencies: - craype 2.1.2 or later is required on Cray XE and Cray XK systems running CLE 5.2 and on all Cray XC systems. - stat is recommended for viewing the merged stack backtrace tree. ATP is supported on the host CPU but not on the accelerator on Cray XK systems. Known Issues: ------------- - Temp space must be writable on the login nodes (and compute nodes under SLURM). The search order is $TMPDIR, /tmp, $HOME. SLURM systems have the additional requirement of a writable /tmp on the compute nodes. See the man page for greater detail. - Currently not supporting simultaneous application runs on a set of nodes on SLURM. Documentation: -------------- For more information see the intro_atp man page. Installation instructions for CLE 5.2: -------------------------------------- rpm -ivh --prefix=/opt/cray atp-2.1.1-201706131319.d3b72849c10eb-0.x86_64.rpm To change the product version to default after installation, execute: /opt/cray/admin-pe/set_default_files/set_default_atp_2.1.1 Installation instructions for CLE 6.0: -------------------------------------- rpm -ivh atp-2.1.1-201706131319.d3b72849c10eb-0.x86_64.rpm To change the product version to default after installation, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_atp_2.1.1 When installing with the PE Installer on a SLURM system, ATP needs to add a file to the plugstack.conf.d sub-directory of the SLURM configuration directory. The installation will attempt to do this automatically, but this depends on the SLURM configuration files having been installed in one of: /etc/opt/slurm /opt/slurm/default/etc /opt/local/slurm/default/etc/sysconfig/slurm If the install can not find the SLURM configuration files, it will error out, delivering instructions to the installation log. The instructions amount to providing the location of the site specific SLURM configuration directory, using the environment variable ATP_SLURM_CONFIG_DIR. By adding this to the boot node 'rc' file (e.g., $HOME/.bashrc, $HOME/.cshrc), future installs will succeed without manual intervention. Without the delivery of $ATP_HOME/spank/99-atpSLaunch to plugstack.conf.d, The ATP framework will not launch automatically at the start of application "srun". (This discussion applies to ATP/1.8.3 and forward.) Should sites build alternative versions of SLURM after ATP has been installed, $ATP_HOME/spank/99-atpSLaunch will need to be manually copied to plugstack.conf.d or ATP will not be launched via srun when that version of SLURM is booted. Product Description: -------------------- Abnormal Termination Processing (ATP) is a system that monitors Cray XT System user applications, and should an application take a system trap, ATP preforms analysis on the dying application. All of the stack backtraces of the application processes are gathered into a merged stack backtrace tree and written to disk as the file "atpMergedBT.dot". The stack backtrace for the first process to die is sent to stderr as is the number of the signal that caused the death. If the core file size limit (RLIMIT_CORE) is non-zero, a heuristically selected set of processes dump their core. atpMergedBT.dot can be viewed with 'statview', a component of the STAT package (module load stat). The merged stack backtrace tree provides a concise, yet comprehensive, view of what the application was doing at the time of its death. License: ---------- Except for the third party modules and software licensed by Cray through proprietary agreements, components, files or programs contained within this package or product are Copyright 2001-2014 Cray Inc. All rights reserved. Attribution notices for open source licensed software contained in this package are detailed in the file: /opt/cray/atp/2.1.1/ATTRIBUTIONS_ATP.2.1.1.txt ################################################################################ Cray Compiling Environment (CCE) 8.6.0 ===================================== Release Date: ------------- June 29, 2017 Purpose: -------- CCE 8.6.0 provides Fortran, C, and C++ compilers for Cray XC and Cray CS systems. The following key enhancements are included in this CCE release: CCE 8.6.0 --------- Support for the Intel® Xeon® processor Scalable family C++14 support is added and enabled by default - See the crayCC(1) man page for more information. C11 support is enabled by default - See the craycc(1) man page for more information. The gcc emulation mode is gcc 6.1 Reduced the performance penalty of C++ exception handling for the common case where an exception is not thrown. OpenMP 4.5 full support - See the intro_openmp(7) man page for more information. Increased use of arithmetic approximations at default for some architectures - The -hfp2=noapprox command line option can be used to disable this optimization. New -h alias command line option controls how aliasing is performed - See the crayftn(1), craycc(1), and crayCC(1) man pages for more information. Optimization of omp simd private() constructs New 'capacity' attribute for the "memory" directive for placing data in DDR memory - See the memory(7) man page for more information. The -g option is now ignored if any optimization option is specified - See the crayftn(1), craycc(1), or crayCC(1) man page for more information. Improved vectorization of Fortran array syntax An API for "thread hot" UPC is introduced for mixed OpenMP threads/UPC performance - See the intro_pgas(7) man page for more information. Performance improvements for all supported systems Important information regarding compatibility: ---------------------------------------------- Please note that a cdt module is provided to help users load required product versions for compatibility with CCE 8.6. There are two important changes in the CCE 8.6 C++ compiler that will most likely require that users rebuild their C++ codes. First, the implementation of C++ exception handling in CCE has changed for the 8.6 release. Unlike CCE 8.5, C++ code compiled with CCE 8.6 is exception-compatible with code generated by the GNU C++ compiler. This means that most C++ code that was compiled with CCE 8.5 is incompatible with most C++ code compiled with CCE 8.6. If you try to link an older C++ library that makes use of C++ exception handling with newly compiled code, you will see errors about undefined references to __eh_curr_region and __curr_eh_stack_entry. The older code will need to be recompiled to be compatible with CCE 8.6. Second, the gcc emulation mode for CCE 8.6 is gcc 6.1.0. This means the default C++ mode is C++ 14, and system headers and libraries from /opt/gcc/6.1.0/ are used. Loading a gcc module that is older than 6.1.0 along with a CCE 8.6 module leads to linker errors. Please unload your gcc module, or swap to gcc/6.1.0, when using CCE 8.6. A new cray-mpich 7.6 module is required to work with CCE 8.6. All MPI codes (Fortran, C, and C++) built with CCE 8.6 must use cray-mpich 7.6. The CCE 8.6 C++ changes affect all MPI codes since MPI uses C++ exception handling. If you need to go back to a cray-mpich 7.5 module or a cce 8.5 module, you should change both cce and cray-mpich. New versions of HDF5, NetCDF, parallel-NetCDF, and Trilinos are required to work with CCE 8.6 for C++ applications. If you need to go back to a previous version of these products or a cce 8.5 module, you should change both cce and these products. For Intel MIC (KNL) targets where exact values are required for division, square root, and exponentiation at default optimization levels (specifically -hfp2), the use of -hfp2=noapprox may be required to disable the newly default arithmetic approximations. Dependencies: ------------- This CCE release is supported on - Cray XC systems with CLE 6.0 or CLE 5.2. - Cray CS systems running CentOS 7.1/7.2 or RedHat 7.1/7.2. For Cray XC systems the following products are required: - CrayPE 2.5.11 - GCC 6.1.0 must be installed but does not need to be the default GCC - GMP 6.1.1 - MPFR 3.1.4 - MPC 1.0.3 - LibSci 17.06.1 or later - MPT 7.6.0 - PMI 5.0.12 or later For Cray CS systems the following products are required: - CrayPE 2.5.11 - GCC 6.1.0 must be installed but does not need to be the default GCC - GMP 6.1.1 - MPFR 3.1.4 - MPC 1.0.3 - LibSci 17.06.1 or later - MVAPICH 2.2rc1.0.3 This CCE release requires the following minimum versions of these products if they are used: - HDF5 1.10.0.2 - NETcdf 4.4.1.1.2 - parallel-NETcdf 1.8.1 - GA 5.3.0.7 - LibSci_acc 17.03.1 - TPSL 17.06.1 - PETSc 3.7.6.0 - fftw3 3.3.6.2 - fftw2 2.1.5.9 - Trilinos 12.10.11 - Perftools 6.5.0 Installation instructions: -------------------------- For XC systems, Craype-installer 1.23.0 or later must be used to install this release. Using an earlier version of Craype-installer with this release will result in a non-functioning Cray Programming Environment installed on the system. For CS systems, please see S-2800 CS System Programming Environments Installation Guide. License: Except for the third party components and software licensed by Cray through proprietary agreements, components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. Attribution notices for open source licensed software contained in this package are detailed in the file: /opt/cray/pe/cce/8.6.0/ATTRIBUTIONS_8.6.txt Bugs Closed: ------------ CCE 8.6.0 --------- 851404 - perftools-lite-hbm fails to compile fortran code using ieee_arithmetic 850880 - cce 8.5.6 C++ too many compiler warnings 850586 - Invalid bitcast -- LLVM ERROR: Broken function found, compilation aborted! 849559 - Performance decrease in the OpenACC code (physics was about 4x slower). 849452 - cuMemFree returned CUDA_ERROR_INVALID_VALUE 849234 - 8.6.0 cc internal error 848709 - ftn-2116 crayftn: INTERNAL "/opt/cray/cce/8.5.7/cftn/x86-64/lib/optcg" was terminated 848418 - Cray fortran compiler cce/8.5.5 ignores DEFAULT(NONE) clause in the nested $OMP PARALLEL construct 847985 - Cray Fortran compiler does not vectorize a do loop nested within an openmp parallel do loop due to "potential hazard in conditional code" 847812 - OpenMP and perftools not honoring: CALL PAT_REGION_BEGIN(1,'MYREGION',pat_err) 846500 - cray c compiler generates inefficient binary from an user code 846104 - ICE in GPU code 845914 - Full c++11 support 845837 - Wrong parameter pass with optimization 845347 - AVX512 intrinsics needed for Agner Fog's C++ vector class library and UMESIMD 844658 - problem with std::regex_replace 841587 - ccfe process persists even after controlling CC process is killed 841582 - unusually long compile times 839653 - cray internal compiler error 836648 - CCE 8.4.1 (OpenACC) INTERNAL COMPILER ERROR(s) 835189 - CCE not respecting OpenMP collapse clause 830170 - code using OpenMP task depend does not distribute across threads 827905 - Cray ftn cannot compile code in a directory containing a space in the name 825947 - Symbol tables for libraries of C++ objects generated with -g exceed size limit 825353 - Cray OpenMP compiled code run with one thread is about 50% slower than the serial version 805385 - Cray Fortran compiler cannot be run in a directory with spaces 794639 - Runtime bug under Cray compile - unable to parse grass_rbcL.tre [Boost Ticket #9152] 792616 - Allow commas in directory paths passed to CCE compilers via options. 767546 - ERROR message for out-of-range constants ################################################################################ The modulefile defines the system paths and variables for the product, the Cray Comparative Debugger (ccdb). CCDB 3.0.3 ========== Release Date: -------------- July 06, 2017 Product Description: -------------------- CCDB, Cray's next generation debugging tool extends the comparative debugging capabilities of lgdb with a Graphical User Interface (GUI) enabling programmers to compare corresponding data structures between two executing applications. Comparative debugging assists users with locating sections of code containing data deviations introduced by algorithm changes, compiler differences, and porting to new architectures/libraries. Some features of ccdb include: * Side-by-side debugging session execution of two parallel applications. * Automatic creation of comparison statements for all local variables in scope. * Type templates for structured data types to selectively compare members. * Warning/error epsilon tolerance values for floating point comparison. * GPU debugging is supported for OpenACC and OpenMP 4.0 GPU directives. * PBS PRO, MOAB/TORQUE, and hybrid SLURM workload manager support. Purpose: --------- This is a feature release. 1. Provides support for Cray CS machines. 2. Provides support for SLURM 15. Bugs Closed with this release: ------------------------------ Other modifications: --------------------------- Fixed a number of problems found in internal testing. Rework breakpoint management so that CCDB and the CCDB server agree on the status of breakpoints. This fixes some cases which cause segfaults attempting to delete breakpoints. Prevent trying to load files we don't have the source for. This was causing a segfault when closing the file. Product and OS Dependencies: ----------------------------- This release depends on the following minimum product versions. cray-lgdb/3.0.1 Cray systems running CentOS operating system on CS. - Native SLURM is supported with version 15. Documentation: --------------- Type `man ccdb` with the cray-ccdb module loaded to read the ccdb(1) man page. Documentation is available through CCDB from the 'Help' menu. Installation instructions: -------------------------- Installation instructions for sles12: ============================================= rpm -ivh cray-ccdb-3.0.3-201706191347.7d8b68ee6ac22-0.sles12.x86_64.rpm To make ccdb 3.0.3 the default version of ccdb, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_cray-ccdb_3.0.3 Installation of ccdb on Mac systems: ------------------------------------ cray-ccdb-3.0.3-201706191347.7d8b68ee6ac22-0.dmg Installation of ccdb on Windows systems: ------------------------------------ cray-ccdb-3.0.3-201706191347.7d8b68ee6ac22-0.x86_64.exe Location of desktop installers: ------------------------------- /opt/cray/pe/ccdb/3.0.3/share/desktop_installers Download the Cray CCDB installer onto a desktop or laptop running Mac OS. Double click on installer to begin installation. The installer will walk you through the process for your system. Download the Cray CCDB installer onto a desktop or laptop running Windows 7 or higher. Double click on installer to begin installation. The installer will walk you through the process for your system. Certain components, files or programs contained within this package or product are Copyright 2007-2017 Cray Inc. All rights reserved. ################################################################################ CRAY CTI 1.0.6 ========== Release Date: -------------- July 06, 2017 Product Description: -------------------- CRAY CTI (Cray Tools Interface) is a library that abstracts debugger tool support to a common interface regardless of the underlying work load manager. It facilitates the staging of files and the launch of tool daemons on the compute nodes associated with a parallel job. ALPS and Cray SLURM are the currently supported WLMs. This is installed primarily for those Cray supplied tools that need such capabilities. The module is not typically loaded, but can be for the sake of experimental swapping to other CTI releases. Purpose: --------- This is a bugfix release. 1. Fix an issue related to dlopen of bundled libraries with the generic workload manager 2. Fix an issue with launching from whiteboxes on slurm Product and OS Dependencies: ----------------------------- The CRAY CTI 1.0.6 release is supported on the following Cray systems running Cray Linux Environment(CLE) operating system - Cray XE and XK systems running CLE 5.2 Running either ALPS or SLURM 15.8 - Cray XC systems with CLE version 5.1 or later Running either ALPS or SLURM 15.8 - Cray CS systems Product Dependencies: - craype 2.1.2 or later is required on Cray XE and Cray XK systems running CLE 5.2 and on all Cray XC systems. Documentation: --------------- Currently the only documentation available is found in craytools_fe.h, craytools_be.h, and the tests in the 'tests' directory. Installation instructions: -------------------------- Installation instructions for CLE 5.2: ============================================= rpm -ivh --prefix=/opt/cray cray-cti-1.0.6-201706191755.a96f83d04d25e-0.sles11.x86_64.rpm To make cti 1.0.6 the default version of cray-cti, execute: /opt/cray/admin-pe/set_default_files/set_default_cray-cti_1.0.6 Installation instructions for CLE 6.0: ============================================= rpm -ivh cray-cti-1.0.6-201706191755.a96f83d04d25e-0.sles11.x86_64.rpm To make cti 1.0.6 the default version of cray-cti, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_cray-cti_1.0.6 Certain components, files or programs contained within this package or product are Copyright 2007-2016 Cray Inc. All rights reserved. ################################################################################ FFTW 3.3.6.2 ============ Release Date: ------------- June 29, 2017 Purpose: -------- The Cray FFTW 3.3.6.2 release is supported on Cray XC and CS Systems. FFTW is supported on the host CPU but not on the accelerator of Cray systems. The Cray FFTW 3.3.6.2 release provides the following: - Added support for Intel Xeon processor Scalable family CPUs on XC systems. - Added FASTPLAN optimizations for Intel Xeon processor Scalable family CPUs on XC systems. The FASTPLAN feature is enabled by default. See the intro_fftw3 manual page for information on FASTPLAN usage. - For Broadwell CPU targets the cray-fftw module will now link to the OpenMP threaded FFTW libraries by default rather than the Pthreads implementation. See the intro_fftw3 manual page for information on default behavior and linking options. Product and OS Dependencies: ---------------------------- The Cray FFTW 3.3.6.2 release is supported on the following Cray systems: * Cray XC systems with CLE 5.2 (or later) or CLE 6.0 (or later). * Cray CS systems with CentOS and either MVAPICH2 2.2 or Intel MPI. The FFTW 3.3.6.2 release requires the following software products: For Cray XC series systems: craype 2.1.2 or later MPT 7.2.5 or later One or more of the following major compiler versions: CCE 8.6 GCC 4.9 GCC 5.1 or later GCC 6.1 or later GCC 7.1 or later Intel 17 For Cray CS systems: craype 2.1.2 or later MVAPICH2 2.2 or Intel MPI (cray-impi) as appropriate for the network target One or more of the following major compiler versions: CCE 8.6 Notes and Limitations: ---------------------- None. Documentation: -------------- http://www.fftw.org/index.html#documentation See the intro_fftw3 manual page for additional information. Modulefile: ----------- module load cray-fftw Installation ------------ On CLE 5.2: rpm -ivh --prefix=/opt/cray cray-fftw-3.3.6.2-201706211942.66c853703ecd2-7.x86_64.rpm On CLE 6.0: rpm -ivh cray-fftw-3.3.6.2-201706211942.66c853703ecd2-7.x86_64.rpm To make this the default version, On CLE 5.2 execute /opt/cray/admin-pe/set_default_files/set_default_fftw_3.3.6.2 On CLE 6.0 execute /opt/cray/pe/admin-pe/set_default_files/set_default_fftw_3.3.6.2 Certain components, files or programs contained within this package or product are Copyright 2011-2017 Cray Inc. All rights reserved. ################################################################################ gcc 6.1.0 ========= Release Date: ------------- June 29, 2017 Purpose: -------- The gcc 6.1.0 release. Product and OS Dependencies: ---------------------------- The gcc 6.1.0 release is supported on the following Cray systems: * Cray XE and XK systems with CLE 5.2 and higher. * Cray XC systems with CLE 5.2 and higher. Limitations: -------------- gcc 6.1.0 is not supported with CUDA Toolkit 7.0. The Cuda runtime header throws an error when compiling with nvcc. Cray recommends to only install gcc 6.1.0 as non-default on systems with GPU accelerators. Documentation: -------------- http://gcc.gnu.org/gcc-6 Modulefile: --------------------- module load gcc/6.1.0 This modulefile defines the system paths and environment variables needed to use gcc, gfortran and g++ on Cray systems. The gcc modulefile can be swapped for other gcc versions. This modulefile may be loaded as a standalone modulefile or as part of the GNU Programming Environment, PrgEnv-gnu. The CrayPE drivers, cc, CC, and gfortran, are recommended for use with PrgEnv-gnu to generate compilation and link lines. The following modulefiles support the GNU 6.1.0 based Programming Environment: craype 2.4.1 or later cray-mpich 7.4.1 or later cray-libsci 13.2.0 or later cray-tpsl 1.5.2 or later cray-petsc 3.6.1.0 or later cray-trilinos 11.12.1.5 or later cray-hdf5 1.8.14 or later cray-netcdf 4.3.3.1 or later cray-parallel-netcdf 1.6.1 or later cray-ga 5.3.0.3 or later fftw2 2.1.5.8 or later fftw 3.3.4.5 or later Installation instructions: -------------------------- rpm -ihv cray-set-gcc-libs-1.0.1-06.201507152117.dc1812b155d2e.x86_64.rpm rpm -ihv cray-gcc-6.1.0-323.201705300039.4877b433ceab0.sles12.x86_64.rpm To change the product version to default after installation: /opt/admin-pe/set_default_files/set_default_gcc_6.1.0 Certain components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. ################################################################################ cray-hdf5 ================= Purpose: -------- To release HDF5-1.10.0-patch1 (1.10.0.3) Product and OS Dependencies: ---------------------------- The HDF5 release is supported on on the following Cray systems: * Cray XE, XK and XC systems with CLE 5.2 and later. The HDF5 1.10.0.3 release requires the following software products: One or more compilers: CCE 8.6.0 or later GCC 4.9 GCC 5.1 Intel 16.0 or later PGI 15.3 or later For Cray XE, XK and XC systems: craype 2.1.2 or later Notes and Limitations: ---------------------- The HDF5 compiler scripts (h5cc, h5fc, h5c++) are not included in the release. The user should use the Cray compiler scripts (cc, ftn, CC) to include the HDF5 header files and link in the HDF5 libraries. Documentation: -------------- http://www.hdfgroup.org/HDF5/doc/index.html Modulefile: ----------- module load cray-hdf5 OR module load cray-hdf5-parallel Product description: -------------------- HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format. Installation: ------------- Installation instructions for CLE 6.0 and later: ================================================ rpm -ivh cray-hdf5-1.10.0.3-*.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_hdf5_1.10.0.3 Installation instructions for CLE 5.2: ============================================= rpm -ivh --prefix=/opt/cray cray-hdf5-1.10.0.3-*.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/admin-pe/set_default_files/set_default_hdf5_1.10.0.3 Certain components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. ################################################################################ LibSci 17.06.1 ============== Release Date: ------------- June 29, 2017 Purpose: -------- Cray LibSci 17.06.1 provides scientific libraries for Cray CS and XC Systems. Cray LibSci is supported on the host CPU but not on the accelerator of these systems. The Cray LibSci 17.06.1 release provides the following: * Added support for the Intel Xeon processor Scalable family * Additional optimizations for Intel Xeon Phi targets * Compiler support and misc bug fix updates Cray LibSci 17.06.1 includes the following versions of publicly available libraries: * LAPACK 3.6.1 - For further information, see http://www.netlib.org/lapack * ScaLAPACK 2.0.2 - (Scalable LAPACK) For further information, see http://www.netlib.org/scalapack. Product and OS Dependencies: ---------------------------- The Cray LibSci 17.06.1 release is supported on the following Cray systems: * Cray CS systems with a June 2017 PE supported OS and MVAPICH2 or Intel MPI * Cray XC systems with CLE 5.2 UP04 or CLE 6.0 The Cray LibSci 17.06.1 release requires the following software products: For Cray XC series systems: craype 2.5.11.5 or later MPT 7.6.0 or later One or more of the following major compiler versions: CCE 8.6 GCC 4.9 GCC 5.1 GCC 6.1 Intel 17.0 For Cray CS systems: CCE 8.6 craype for CS MVAPICH2 or Intel MPI Notes and Limitations: ---------------------- OMP threaded versions: The CrayPE 2.1.2 and later releases add support for link line generation for the multi-threaded versions of the libsci library based on the OpenMP flags the user specifies for each compiler: * CCE by default links to the OpenMP LibSci library. CrayPE will link in the serial version of LibSci when the CCE flag -hnoomp is used. * GNU by default links serial LibSci library. CrayPE will link in the OpenMP version of LibSci when the GNU flag -fopenmp is used. * INTEL by default links serial LibSci library. CrayPE will link in the OpenMP version of LibSci when the INTEL flag -qopenmp is used. Non-default dynamic linking: When using a non-default version of cray-libsci with dynamic linking users should set at runtime and at linktime after loading the desired cray-libsci module--or set the equivalent for their linux shell: export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH Documentation: -------------- See the intro_libsci man page for additional information. See the csmlversion man page for information to display version information on the currently loaded scientific libraries. Modulefile: ----------- module load cray-libsci Installation instructions: -------------------------- LibSci is now packaged into separate compiler specific RPMs to allow rpmbuild to correctly include compiler dependencies. On CLE 5.2: rpm -ihv --prefix=/opt/cray cray-libsci-17.06.1-*-1.201706201742.e8a91838cafc6.x86_64.rpm On CLE 6.0: rpm -ihv cray-libsci-17.06.1-*-1.201706201742.e8a91838cafc6.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version: On CLE 5.2, execute /opt/cray/admin-pe/set_default_files/set_default_libsci_17.06.1 On CLE 6.0, execute /opt/cray/pe/admin-pe/set_default_files/set_default_libsci_17.06.1 The csmlversion RPM is now packaged with the LibSci distribution. On CLE 5.2: rpm -ihv --prefix=/opt/cray csmlversion-1-3.x86_64.rpm On CLE 6.0: rpm -ihv csmlversion-1-3.x86_64.rpm csmlversion does not have a module file so the last installed version is the only version available. License: -------- Except for the third party modules and software licensed by Cray through proprietary agreements, components, files or programs contained within this package or product are Copyright 2001-2017 Cray Inc. All rights reserved. Attribution notices for open source licensed software contained in this package are detailed in the file: /opt/cray/libsci/17.06.1/ATTRIBUTIONS_libsci17.06.1.txt ################################################################################ Cray Message Passing Toolkit 7.6.0: ======================================= Release Date: ------------- July 06, 2017 Purpose: -------- The following features were added to MPT 7.6.0 over MPT 7.5.5: - Support for the CCE 8.6.0 compiler has been added. Note users who wish to use older versions of CCE will accordingly need to use older versions of MPT. - Support for a new member of the Intel Xeon Processor Scalable Family has been added. - Support has been added to improve the default Cray MPI one-sided performance on XC systems. Initial performance improvement over the previous default MPI one-sided version has been observed to be over 4X for both latency and bandwidth. The DMAPP based MPI one-sided optimized version is now deprecated, along with environment variables MPICH_RMA_OVER_DMAPP, MPICH_RMA_USE_NETWORK_AMO, and MPICH_RMA_DMAPP_MAX_WINDOWS_ON_NODE. A new environment variable, MPICH_RMA_FALLBACK, allows you to fall back to an implementation of MPI RMA over MPI point-to-point functions. A future release will extend the options for this the environment variable to allow users to fall back to the implementation of MPI RMA available in ANL MPICH. - Support has been added to optimize the Cray MPI memcpy for a new member of the Intel Xeon Processor Scalable Family. Additionally users can set the MPICH_OPTIMIZED_MEMCPY to 2 to use a highly optimized memcpy for this new member that may improve their performance. Since, in some cases, applications may run slower, the optimized memcpy is not enabled by default. - Improvements have been made to the Cray MPI async-progress algorithms. See the MPICH_NEMESIS_ASYNC_PROGRESS env variable in the intro_mpi man page for more information. - MPI_Reduce_scatter and MPI_Reduce_scatter_block has been modified to scale better on high process counts by using much less memory. See the MPICH_REDSCAT_MAX_COMMSIZE and the MPICH_REDSCAT_COMMUTATIVE_LONG_MSG_SIZE env variables in the intro_mpi man page for more information. - Cray MPI has been enhanced so that MPI-IO will recognize the new DataWarp Cache FileSystem feature. - A CRAY_MPICH_VERSION preprocessor macro has been added so that users can test version-specific cases for Cray MPI. It can be accessed by including “mpi.h”. For example for this release it will be set to “7.6.0”. - Cray SHMEM now supports an optimized broadcast algorithm for very large monolithic data sizes. Initial results show 2X bandwidth improvements for large monolithic broadcasts greater than 16MB. See the SHMEM_USE_OPT_MASSIVE_BCAST and SHMEM_MASSIVE_BCAST_CUTOFF env variables in the intro_shmem man page for more information. - The Cray SHMEM optimized shared memory based reduction algorithm for team-based reduction operations and the optimized reduction algorithm for large data sizes have now been enabled by default. See the SHMEM_TEAM_SMP_REDUCE and SHMEM_USE_LARGE_OPT_REDUCE env variables in the intro_shmem man page for more information. In addition, the following features were added in MPT updates since the MPT 7.5.0 release in November 2016: - Cray MPI now offers a new optimized thread synchronization mechanism to Improve the performance of multi-threaded MPI applications. This optimization implements lock/unlock operations between threads in a fast and fair manner and is closely related to the thread synchronization mechanism used to implement Thread Hot MPI-3 RMA operations in Cray MPI. Initial performance results show improvements as much as 40% over the previous implementation. See the MPICH_CRAY_OPT_THREAD_SYNC environment variable in the intro_mpi man page for more information. - Cray MPI shared-memory collective operations have been improved by using a "network buffer" design. This optimization is currently available for the MPI_Bcast collective operation. This feature is currently disabled by default but may be enabled by default in the future. Initial performance results show improvements up to 16% over the current default for small message MPI_Bcast operations. See the MPICH_NETWORK_BUFFER_COLL_OPT env variable for more information. - Cray MPI has been enhanced to include a new environment variable to provide additional information for MPI-IO operations. This information may guide the user in trying different I/O operations or different algorithms. See the MPICH_MPIIO_TIMERS environment variable in the intro_mpi man page for more information. - Cray SHMEM has been enhanced to support shmem_align for all align sizes as required by the OpenSHMEM standards. - Cray PMI has been enhanced to prefix each stdout and stderr line with MPI rank number. See the PMI_LABEL_ERROUT in the intro_mpi man page for more information and limitations. Note: PMI 5.0.11 is required for this feature. - Cray PMI has been modified so that users can modify the timeout value used during PMI startup when PMI_NO_FORK is set or when SLURM is used. See the PMI_MMAP_SYNC_WAIT_TIME environment variable in the intro_mpi man page for more information. In addition, the default value when PMI_NO_FORK or SLURM is used has been increased from 60 to 180 seconds to account for launching programs on very large scaled systems. - Improvements to Cray SHMEM by adding shared memory optimizations for team-based allreduce operations. Initial performance results show improvements up to 2X over the default team-based allreduce algorithm on fully packed nodes. See the SHMEM_TEAM_SMP_REDUCE environmental variable in the intro_shmem man page for more information. - Cray PMI now supports a new topology and placement-aware rank reordering option. This option determines an optimized rank placement based on the hardware resources available to the job at the time of job launching. Please note that this option has no effect on the resources selected by the workload manager. Use of this placement style requires setting the MPICH_RANK_REORDER_OPTS environment variable. See the intro_mpi man page for more information. Initial results have shown some applications have improved by as much as 35% using this option. - Cray MPI now supports a subset of Dynamic Process Management(DPM) from the MPI-2 and MPI-3 standards. This support is available as a separate version of the Cray MPICH library, invoked using the new “-craympich-dpm” compiler driver option. The new functions that are now supported are MPI_OPEN_PORT, MPI_CLOSE_PORT, MPI_COMM_ACCEPT, MPI_COMM_CONNECT, MPI_COMM_DISCONNECT, MPI_LOOKUP_NAME, MPI_PUBLISH_NAME and MPI_UNPUBLISH_NAME. See the Notes section of the MPI man page for information on DPM functions that are not supported. In addition a new env variable has been added MPICH_DPM_DIR to specify the DPM directory. See the intro_mpi man page and the Cray Programming Environment User’s Guide for more information. - Cray MPI now supports a larger MPI_TAG_UB value. This feature is linked to the DPM support due to implementation dependencies that would have reduced the max tag size to an unreasonable level. For that reason this support is only available using the new “-craympich-dpm” option. See the intro_mpi man page for more information. - Cray MPI now supports optimized message matching. Since this feature was needed for the feature to increase the max tag size it is also only available when using the new “-craympich-dpm” option. See the intro_mpi man page for more information on the following env variables: MPICH_USE_BINNING_MSG_MATCH, MPICH_NUM_POST_RECV_BINS and MPICH_NUM_UNEXPECTED_BINS. Initial results have shown improvements of as much as 16% in some micro-benchmarks, where the number of messages to be traversed before getting an actual match varies in the range of 60-80 comparisons. - Cray PMI has been enhanced such that PMI_LABEL_ERROUT now supports rank reordering and user-defined labels. The user-defined labels may be set using PMI_LABEL_ERROUT_FORMAT environment variable. See into_mpi man page for more information. The following bugs were fixed since the MPT 7.5.5 release: o 843363 - IMB-EXT Accumulate test hangs o 851127 - cray-mpich: High delay for MPI-RMA operations o 849235 - Two RMA tests from MPICH test suite fail with Cray MPI in "default" (non-DMAPP) mode o 849616 - KNL MPI_Init fails with MPICH_RMA_OVER_DMAPP o 849237 - rma/acc-pairtype from MPICH test suite running Cray MPI in "default" (non-DMAPP) mode o 837219 - Poor performance of MPI_Accumulate (of cray-mpich 7.3.1) on Cori-1 o 842808 - Weird Timings when Win_flush_local is used o 843419 - cray-mpich crashes with more than 1022 MPI shared-memory windows o 841296 - MPI_Win_lock/unlock interaction with MPI_Raccumulate fails in multi-threaded test o 838193 - Update MPI RMA's network AMO implementation to take MPI-3.1's accumulate ordering into account. o 851393 - Missing "stripe sized writes" output during MPI IO STATS with CRAY-MPICH v7.5.2 o 850944 - request more details about GNI Routing Modes o Misc Cray SHMEM fixes o Misc improvements to the intro_shmem and intro_mpi man pages In addition, the following bugs were fixed since the MPT 7.5.0 release in November 2016: o Misc MPI RMA fixes o Misc fixes to Cray SHMEM broadcast and shmem_align o Improved error messages for incorrect values to Cray SHMEM routines o 845356 - Three suggested post MPICH v3.2 patch fixes for Cray MPI o Misc fixes to Cray MPI-IO o 847690 - performance issues with MPI_Allgatherv with MPICH_RDMA_ENABLED_CUDA=1 o 850049 - Use of MPI and IOBUF results in multiple definitions of pread and pwrite o Misc Cray SHMEM fixes Product and OS Dependencies: ---------------------------- The Cray MPT 7.6.0 release is supported on the following Cray systems: * Cray XE and Cray XK systems running CLE 5.2 * Cray XC systems with CLE version 5.2 or later Product and OS Dependencies by network type: ------------------------------------------------------------------------------------+ | Gemini (XE) | Aries (XC30) | --------------------------+---------------------------+---------------------------+ craype | >= 2.4.0 | >= 2.4.0 | --------------------------+---------------------------+---------------------------+ pmi | >= 5.0.4 | >= 5.0.4 | --------------------------+---------------------------+---------------------------+ cray-libugni | >= 5.0 | >= 5.0 | --------------------------+---------------------------+---------------------------+ cray-libugni-devel | >= 5.0 | >= 5.0 | --------------------------+---------------------------+---------------------------+ cray-libudreg | >= 2.3.2 | >= 2.3.2 | --------------------------+---------------------------+---------------------------+ cray-libudreg-devel | >= 2.3.2 | >= 2.3.2 | --------------------------+---------------------------+---------------------------+ cray-libxpmem | >= 0.1 | >= 0.1 | --------------------------+---------------------------+---------------------------+ cray-libxpmem-devel | >= 0.1 | >= 0.1 | --------------------------+---------------------------+---------------------------+ cray-libdmapp | >= 4.0.1 | >= 7.0.1 | --------------------------+---------------------------+---------------------------+ cray-libdmapp-devel | >= 4.0.1 | >= 7.0.1 | --------------------------+---------------------------+---------------------------+ alps | default | default | --------------------------+---------------------------+---------------------------+ One or more compilers: * CCE 8.6 or later * PGI 15.3 or later * GNU 4.9 or later * Intel 16.0 or later Documentation: -------------- For more information see the intro_mpi and intro_shmem man pages. Modulefile: ----------- module load cray-mpich/7.6.0 Installation: ------------- On CLE 5.2: rpm -ivh --prefix=/opt/cray cray-mpt-7.6.0-*-201706211108.2f3677d55fabd-0.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/admin-pe/set_default_files/set_default_mpt_7.6.0 On CLE 6.0 or later: rpm -ivh cray-mpt-7.6.0-*-201706211108.2f3677d55fabd-0.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_mpt_7.6.0 Certain components, files or programs contained within this package or product are Copyright 2007-2015 Cray Inc. All rights reserved. ################################################################################ cray-netcdf ================= Purpose: -------- New netCDF-C Release Product and OS Dependencies: ---------------------------- The NetCDF release is supported on the following Cray systems: * Cray XE, XK and XC systems with CLE 5.2 or later The NetCDF 4.4.1.1.3 release requires the following software products: One or more compilers: CCE 8.6 or later GCC 4.9 GCC 5.1 Intel 16.0 or later PGI 15.3 or later For Cray XE, XC, XC systems: CrayPE 2.1.2 or later Notes and Limitations: --------------------- Unidata now packages Netcdf-4 and legacy netcdf-3 separately. Cray has decided not to continue supplying the legacy Netcdf-3 package. Due to CCE changes a version of netcdf built with "-sreal64" is neither needed nor provided. NetCDF is supported on the host CPU but not on the accelerator on Cray XC and XK systems. Documentation: -------------- http://www.unidata.ucar.edu/software/netcdf/docs Modulefile: ----------- module load cray-netcdf OR module load cray-netcdf-hdf5parallel Product description: -------------------- NetCDF (network Common Data Form) is a set of interfaces for array-oriented data access and a freely-distributed collection of data access libraries for C, Fortran, C++, Java, and other languages. The netCDF libraries support a machine-independent format for representing scientific data. Together, the interfaces, libraries, and format support the creation, access, and sharing of scientific data. Installation: ------------- Installation instructions for CLE 6.0 and later: ================================================ rpm -ivh cray-netcdf-4.4.1.1.3-*.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_netcdf_4.4.1.1.3 Installation instructions for CLE 5.2: ============================================= rpm -ivh --prefix=/opt/cray cray-netcdf-4.4.1.1.3-*.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/admin-pe/set_default_files/set_default_netcdf_4.4.1.1.3 Certain components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. ################################################################################ papi ================= Purpose: -------- New version of papi 5.5.1.2 Product and OS Dependencies: ---------------------------- The papi release is supported on on the following Cray systems: * Cray XE, XK and XC systems with CLE 5.2 or later. Documentation: -------------- Release notes: http://icl.cs.utk.edu/viewcvs/viewcvs.cgi/PAPI/papi/RELEASENOTES.txt Overview: http://icl.cs.utk.edu/projects/papi/wiki/Main_Page Modulefile: ----------- module load papi Product description: -------------------- PAPI aims to provide the tool designer and application engineer with a consistent interface and methodology for use of the performance counter hardware found in most major microprocessors. PAPI enables software engineers to see, in near real time, the relation between software performance and processor events. Installation instructions: -------------------------- Installation of PAPI for CLE 5.2: --------------------------------- rpm -ivh --prefix=/opt/cray cray-papi-5.5.1.2-0.x86_64.rpm To make this the default version, execute: /opt/cray/admin-pe/set_default_files/set_default_papi_5.5.1.2 Installation of PAPI for CLE 6 and later: ----------------------------------------- rpm -ivh cray-papi-5.5.1.2-0.x86_64.rpm To make this the default version, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_papi_5.5.1.2 Certain components, files or programs contained within this package or product are Copyright -2016 Cray Inc. All rights reserved. ################################################################################ Parallel NetCDF 1.8.1.3 ================= Purpose: -------- New version of Parallel NetCDF (1.8.1.3) Product and OS Dependencies: ---------------------------- This Parallel NetCDF release is supported on on the following Cray systems: * Cray XE and XK systems with CLE 5.2. * Cray XC systems with CLE 5.2 or later. The Parallel NetCDF 1.8.1.3 release requires the following software products: One or more compilers: CCE 8.6 or later GCC 4.9 GCC 5.1 Intel 16.0 or later PGI 15.3 or later For Cray XE and XK systems running CLE 5.2: CrayPE 2.1.2 or later For Cray XC systems: CrayPE 2.1.2 or later Notes: Shared libraries are not provided as the Third Party software vendor does not support shared libraries. Documentation: -------------- http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/Documentation Modulefile: ----------- module load cray-parallel-netcdf Product Description: -------------------- Parallel NetCDF is a library providing high-performance I/O while still maintaining file-format compatibility with Unidata's NetCDF. Installation: ------------- Installation instructions for CLE 6.0 or later: ================================================ rpm -ivh cray-parallel-netcdf-1.8.1.3-*.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_parallel-netcdf_1.8.1.3 Installation instructions for CLE 5.2: ============================================= rpm -ivh --prefix=/opt/cray cray-parallel-netcdf-*-1.8.1.3-*.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/admin-pe/set_default_files/set_default_parallel-netcdf_1.8.1.3 Certain components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. ################################################################################ CrayPE 2.5.11 ============== Release Date: -------------- June 29, 2017 Purpose: -------- - Add support for CCE 8.6. - Add support for a new member of the Intel Xeon Processor Scalable Family. Dependencies: --------------------------------------------- The CrayPE 2.5.11 release is supported on the following Cray systems: * Cray XE and XK systems with CLE 5.2. * Cray XC systems with CLE 5.1, CLE 5.2, and CLE 6.0. * Cray CS300 systems with CentOS 6.4 and Redhat 6.4. The CrayPE 2.5.11 release is dependent on .pc files in the following software products: ATP 1.6.3 or later FFTW 3.3.0.4 or later FFTW 2.1.5.6 or later Global Arrays 5.1.0.2 or later HDF5 1.8.11 or later iobuf 2.0.5 or later LibSci 12.1.01 or later MPT 6.0.2 or later NetCDF 4.3.0 or later Parallel-NetCDF 1.3.1.1 or later PMI 4.0.1 or later PETSc 3.4.2.0 or later Trilinos 11.4.1.0 or later TPSL 1.3.04 or later TotalView 8.12-totalview-support-1.1.5 or later Documentation: --------------- See manpages for cc, CC, ftn, intro_craype-api, intro_hugepages and pkg-config See section 2.6 Using Targeting Modules of the Cray Programming Environment User's Guide (S-2529-116) See http://www.freedesktop.org/wiki/Software/pkg-config for a pkg-config introduction. Installation instructions: -------------------------- CLE 6.0: rpm -ivh craype-2.5.11-5.201706271433.4944c8b757bfa.x86_64.rpm CLE 5.1, 5.2, CentOS 6.4, and Redhat 6.4: rpm -ivh -prefix /opt/cray craype-2.5.11-5.201706271433.4944c8b757bfa.x86_64.rpm To make this the default version, execute: CLE 6.0: /opt/cray/pe/admin-pe/set_default_files/set_default_craype_2.5.11 CLE 5.1, 5.2, CentOS 6.4, and Redhat 6.4: /opt/cray/admin-pe/set_default_files/set_default_craype_2.5.11 Product description: -------------------- CrayPE contains drivers, cc, CC, and ftn to compile for the CCE, PGI, GNU, and Intel Programming Environments. The craype-* targeting modules are also provided in this product. Certain components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. ################################################################################ Cray PE Installer 1.23.0 ======================================== Purpose: -------- Resolved the following RFE's: 815887 - sn2476 no haswell CRAY_CPU_TARGET in install-cdt.yaml Product and OS Dependencies: ---------------------------- The Cray PE Installer 1.23.0 release requires the following software products: * perl 5.8 or later * rpm 4.1.1 or later The Cray PE Installer is supported on the following Cray systems running Cray Linux Environment (CLE) operating system * Cray XE systems with CLE version 4.0 or later on boot nodes * Cray esLogins with CLE support package version 4.1 or later * Cray esMS with CLE support package version 4.1 or later * Cray XC systems with CLE version 5.0 or later on each boot node Documentation: --------------- For a summary of the command line arguments supported: craype-installer.pl --help For detailed information on using the installer see: S-2372-105 Cray Programming Environments Installation Guide Installation instructions: -------------------------- rpm -ivh craype-installer-1.23.0-3.x86_64.rpm To make this version default then run: /opt/cray/admin-pe/set_default_files/set_default_craype-installer_1.23.0 Product description: -------------------- The Cray PE Installer is used to simplify and automate the process of installing and updating the Cray Programming Environment software on Cray esLogin images on an esMS server, directly on esLogins and login nodes for the Cray XE/XK/XC systems. The Cray PE Installer uses a configuration file, set up once by the system administrator, to ensure a complete and consistent installation of the Cray Programming Environment software. Detailed information about the entire installation process is captured in log files and can optionally be seen on the console if the --verbose argument is specified. Setting the default versions of libraries, tools and compilers is supported by specifying the --set-default argument when running the installer. Certain components, files or programs contained within this package or product are Copyright 2012 - 2017 Cray Inc. All rights reserved. ################################################################################ PETSc 3.7.6.0 ============= Release Date: ------------- June 29, 2017 Purpose: -------- The Cray PETSc 3.7.6.0 release is supported on Cray XC series systems. Cray PETSc is supported on the host CPU but not on the accelerator of Cray XC series systems. The Cray PETSc 3.7.6.0 release is based on the official patch release of PETSc 3.7.6 by Argonne National Laboratory. For further information about the PETSc 3.7.6 release, see http://www.mcs.anl.gov/petsc/documentation/changes/37.html The Cray PETSc 3.7.6.0 release provides the following: * Source rebase onto PETSc 3.7.6 maintenance release * Support for cray-mpich 7.6.0 * Support for CCE 8.6 * Support for Intel Xeon processor Scalable family Product and OS Dependencies: ---------------------------- The Cray PETSc 3.7.6.0 release is supported on the following Cray systems: * Cray XC systems with CLE 5.2 or later. The Cray PETSc 3.7.6.0 release requires the following software products: For Cray XC series systems: CrayPE 2.1.2 or later Libsci 13.0.0 or later TPSL 17.06.1 MPT 7.6.0 or later HDF5 1.10.0.2 One or more compilers: CCE 8.6.0 or later GCC 4.9.0 or later GCC 5.1.0 or later Intel 16.0.3.210 or later Documentation: -------------- http://www.mcs.anl.gov/petsc/documentation/index.html See the intro_petsc man page for additional information. Modulefile: ----------- module load cray-petsc module load cray-petsc-complex module load cray-petsc-64 module load cray-petsc-complex-64 Installation instructions: -------------------------- On CLE 5.2: rpm -ivh --prefix=/opt/cray cray-petsc-3.7.6.0-*-.x86_64.rpm On CLE 6.0: rpm -ivh cray-petsc-3.7.6.0-*-.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, on CLE 5.2 execute /opt/cray/admin-pe/set_default_files/set_default_petsc_3.7.6.0 on CLE 6.0 execute /opt/cray/pe/admin-pe/set_default_files/set_default_petsc_3.7.6.0 License: -------- Except for the third party modules and software licensed by Cray through proprietary agreements, components, files or programs contained within this package or product are Copyright 2001-2016 Cray Inc. All rights reserved. Attribution notices for open source licensed software contained in this package are detailed in the file: /opt/cray/petsc/3.7.6.0/ATTRIBUTIONS_petsc3.7.6.0.txt ################################################################################ Cray Message Passing Toolkit 7.6.0 ================================== SNP Launcher 7.6.0 ================= Release Date: -------------- July 06, 2017 Purpose: -------- The Single Node Parallel (SNP) Launcher allows a user to launch single node MPI applications on Service MAMU nodes. Please see the CLE documentation for further information about Service MAMU nodes. Product and OS Dependencies: ---------------------------- The Cray SNP Launcher 7.6.0 release is supported on the following Cray systems: - Cray XC systems with CLE version 5.2 or later Product and OS Dependencies by network type: -------------------------------------------------------+ | Aries (XC30) | --------------------------+---------------------------+ craype | >= 2.4.0 | --------------------------+---------------------------+ pmi | >= 5.0.4 | --------------------------+---------------------------+ cray-libugni | >= 5.0 | --------------------------+---------------------------+ cray-libugni-devel | >= 5.0 | --------------------------+---------------------------+ cray-libudreg | >= 2.3.2 | --------------------------+---------------------------+ cray-libudreg-devel | >= 2.3.2 | --------------------------+---------------------------+ cray-libxpmem | >= 0.1 | --------------------------+---------------------------+ cray-libxpmem-devel | >= 0.1 | --------------------------+---------------------------+ cray-libdmapp | >= 7.0.1 | --------------------------+---------------------------+ cray-libdmapp-devel | >= 7.0.1 | --------------------------+---------------------------+ alps | default | --------------------------+---------------------------+ One or more compilers: CCE 8.6 or later PGI 15.3 or later GNU 4.9 or later Intel 16.0 or later Documentation: -------------- For more information see the mpiexec(1) man page after loading the cray-snplauncher module. Installation: ------------ rpm -ivh cray-snplauncher-7.6.0-.2f3677d55fabd-0.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, execute: /opt/cray/admin-pe/set_default_files/set_default_cray-snplauncher_7.6.0 Certain components, files or programs contained within this package or product are Copyright 2007-2015 Cray Inc. All rights reserved. =================================================================== To re-display cray-snplauncher/7.6.0 copyright information, type: less /opt/cray/snplauncher/7.6.0/COPYRIGHT To re-display cray-snplauncher/7.6.0 release information, type: less /opt/cray/snplauncher/7.6.0/release_info =================================================================== ################################################################################ STAT 3.0.1.0 ============= Release Date: ------------- July 06, 2017 Purpose: -------- Updated Cray support of STAT [THREE_DIGIT_VERSION] to support both ALPS and native SLURM. Launch support inside STAT on SLURM is not supported with this version. Attaching to an already running application is. This issue will be resolved in a future release. This release contains a newer version of STAT: 3.0.1 This release contains a newer version of dyninst: 9.3.2 Product and OS Dependencies: ---------------------------- The STAT 3.0.1.0 release is supported on Cray systems running CLE 5.2, and CLE 6.0 and greater operating system, and systems running SLURM 16. Documentation: -------------- https://computing.llnl.gov/code/STAT/ Notes: ------ Installation Instructions: ------------- Installation for CLE 5.2: ------------------------- rpm -ivh --prefix=/opt/cray cray-stat-3.0.1.0-201706191610.a47bd48b31512-1.x86_64.rpm To make this the default version on CLE 5.2, execute: /opt/cray/admin-pe/set_default_files/set_default_stat_3.0.1.0 Installation for CLE 6.0: ------------------------- rpm -ivh cray-stat-3.0.1.0-201706191610.a47bd48b31512-1.x86_64.rpm To make this the default version on CLE 6.0, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_stat_3.0.1.0 Certain components, files or programs contained within this package or product are Copyright 2009-2017 Cray Inc. All rights reserved. ################################################################################ Release Date: May 04, 2017 Purpose: -------- - This package includes the 2017.1.21 release of the Totalview Technologies debugger. - Add support for 'totalview-support' on CLE-5.2 and CLE-6.0 systems. Changes: -------- Installation: ------------- rpm -ivh cray-totalview-2017.1.21-totalview-support-1.2.0.14-1.x86_64.rpm To make this the default version, execute: /opt/admin-pe/set_default_files/set_default_totalview_2017.1.21 /opt/admin-pe/set_default_files/set_default_totalview-support_1.2.0.14 TotalView 2017.1.21 new features can be found at: /opt/totalview/2017.1.21/doc/pdf/TotalView_New_Features.pdf ################################################################################ Third Party Scientific Libraries 17.06.1 ====================================== Release Date: ------------- June 29, 2017 Purpose: -------- The TPSL 17.06.1 release is supported on Cray XC series systems. TPSL is supported on the host CPU but not on the accelerator of Cray systems. The TPSL 17.06.1 release provides the following: * MUMPS upgrade to version 5.1.1, with limited 64-bit integer support * HYPRE upgrade to version 5.11.2 * Support for cray-mpich 7.6.0 * Support for CCE 8.6 TPSL (Third Party Scientific Libraries) contains a collection of mathematical libraries that can be used with PETSc and Trilinos. This module increases the flexibility of PETSc and Trilinos by providing users with multiple options for solving problems in dense and sparse linear algebra. The cray-tpsl module is automatically loaded when PETSc or Trilinos is loaded. The libraries included are MUMPS, SuperLU, SuperLU_DIST, METIS, ParMETIS, HYPRE, SUNDIALS, Scotch, matio, and GLM. The libraries included in cray-tpsl-64 are SuperLU_DIST, METIS, ParMETIS, HYPRE, SUNDIALS, and Scotch. * MUMPS 5.1.1. MUMPS (MUltifrontal Massively Parallel sparse direct Solver) is a package of parallel, sparse, direct linear-system solvers based on a multifrontal algorithm. MUMPS can interface with SCOTCH as well. For further information, see http://graal.ens-lyon.fr/MUMPS/. * SuperLU 5.2.1. SuperLU is a sequential version of SuperLU_DIST and a sequential incomplete LU preconditioner that can accelerate the convergence of Krylov subspace iterative solvers. For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. * SuperLU_DIST 5.1.3. SuperLU_DIST is a package of parallel, sparse, direct linear-system solvers. For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. * METIS 5.1.0. METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices. For further information, see http://glaros.dtc.umn.edu/gkhome/views/metis * ParMETIS 4.0.2. ParMETIS (Parallel Graph Partitioning and Fill-reducing Matrix Ordering) is a library of routines that partition unstructured graphs and meshes and compute fill-reducing orderings of sparse matrices. For further information, see http://glaros.dtc.umn.edu/gkhome/views/metis. * HYPRE 2.11.2. HYPRE is a library of high-performance preconditioners that use parallel multigrid methods for both structured and unstructured grid problems (not included with petsc-complex). For further information, see http://www.llnl.gov/CASC/linear_solvers/. * SUNDIALS 2.7.0 (SUite of Nonlinear and DIfferential/Algebraic equation Solvers) consists of 6 solvers: ARKODE, CVODE, CVODES, IDA, IDAS, and KINSOL. In addition, SUNDIALS provides a MATLAB interface to CVODES, IDAS, and KINSOL that is called sundialsTB. For further information, see https://computation.llnl.gov/casc/sundials/main.html. * Scotch 6.0.4. Scotch is a software package and libraries for sequential and parallel graph partitioning, static mapping, sparse matrix block ordering, and sequential mesh and hypergraph partitioning. For further information, see http://www.labri.fr/perso/pelegrin/scotch. * matio 1.5.2. matio is a C library for reading and writing Matlab MAT files. NOTE: Matlab 7 (or later) is not supported by matio 1.5.2. For further information, see http://sourceforge.net/projects/matio/. * GLM 0.9.6.3: OpenGL Mathematics (GLM) is a header only C++ mathematics library for graphics software based on the OpenGL Shading Language (GLSL) specifications, which it extends with matrix transformations, quaternions, data packing, random numbers, noise, etc. For further information, see http://glm.g-truc.net/0.9.6/index.html Product and OS Dependencies: ---------------------------- The Cray TPSL 17.06.1 release is supported on the following Cray systems: * Cray XE and XK systems with CLE 5.2 or later. * Cray XC systems with CLE 5.2 or later. The Cray TPSL 17.06.1 release requires the following software products: For Cray XC series systems: craype 2.1.2 or later cray-libsci 13.0.0 or later MPT 7.6.0 or later One or more compilers: CCE 8.6.0 or later GCC 4.9.0 or later GCC 5.1.0 or later Intel 16.0.3.210 or later Documentation: -------------- http://graal.ens-lyon.fr/MUMPS/ http://crd.lbl.gov/~xiaoye/SuperLU/ http://glaros.dtc.umn.edu/gkhome/views/metis/ http://www.llnl.gov/CASC/linear_solvers/ https://computation.llnl.gov/casc/sundials/main.html http://www.labri.fr/perso/pelegrin/scotch/ http://sourceforge.net/projects/matio/ Modulefile: ----------- module load cray-tpsl module load cray-tpsl-64 Installation instructions: -------------------------- On CLE 5.2: rpm -ivh --prefix=/opt/cray cray-tpsl-17.06.1-*-.x86_64.rpm On CLE 6.0: rpm -ivh cray-tpsl-17.06.1-*-.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version, on CLE 5.2, execute /opt/cray/admin-pe/set_default_files/set_default_tpsl_17.06.1 on CLE 6.0, execute /opt/cray/pe/admin-pe/set_default_files/set_default_tpsl_17.06.1 License: -------- Attribution notices for open source licensed software contained in this package are detailed in the file: /opt/cray/tpsl/17.06.1/ATTRIBUTIONS_tpsl17.06.1.txt Certain components, files, or programs contained within this package or product are Copyright 2011-2017 Cray Inc. All rights reserved. ################################################################################ Trilinos 12.10.1.1 ================= Release Date: ------------- June 29, 2017 Purpose: -------- The Cray Trilinos 12.10.1.1 release is supported on Cray XC series systems. Cray Trilinos is supported on the host CPU but not on the accelerator of Cray XC series systems. The Cray Trilinos 12.10.1.1 is based on the official release of Trilinos 12.10.1 by Sandia National Laboratories. For further information about the Trilinos 12.10.1 release, see http://trilinos.org/oldsite/release_notes-12.10.html The Cray Trilinos 12.10.1.1 release provides the following: * Support for CCE 8.6 * Support for MPT 7.6.0 * Support for Intel Xeon processor Scalable family Product and OS Dependencies: ---------------------------- The Cray Trilinos 12.10.1.1 release is supported on the following Cray systems: * Cray XC systems with CLE 5.2 or later. The Cray Trilinos 12.10.1.1 release requires the following software products: For Cray XC series systems: CrayPE 2.1.2 or later cray-libsci 16.03.1 or later TPSL 17.06.1 MPT 7.6.0 or later NetCDF 4.4.1.1.3 HDF5 1.10.0.3 One or more of the following major compiler versions: CCE 8.6 or later GCC 4.9 or later Intel 16 or later Notes and Limitations: ---------------------- When using Cray Trilinos 12.10.1.1 with PrgEnv-intel, users must load a gcc module >= 4.9.0 for C++11 support. See Bug 836679. Documentation: -------------- For a list of Trilinos documentation, go to http://trilinos.org/about/documentation/ To see descriptions of each individual Trilinos package, go to http://trilinos.org/packages See the intro_trilinos man page for additional information. Modulefile: ----------- module load cray-trilinos Installation instructions: -------------------------- On CLE 5.2: rpm -ivh --prefix=/opt/cray cray-trilinos-12.10.1.1-*-.x86_64.rpm On CLE 6.0: rpm -ivh cray-trilinos-12.10.1.1-*-.x86_64.rpm The "*" in the install command represents compiler version combinations. To make this the default version: on CLE 5.2, execute /opt/cray/admin-pe/set_default_files/set_default_trilinos_12.10.1.1 on CLE 6.0, execute /opt/cray/pe/admin-pe/set_default_files/set_default_trilinos_12.10.1.1 License: ---------- Except for the third party modules and software licensed by Cray through proprietary agreements, components, files or programs contained within this package or product are Copyright 2001-2016 Cray Inc. All rights reserved. Attribution notices for open source licensed software contained in this package are detailed in the file: /opt/cray/trilinos/12.10.1.1/ATTRIBUTIONS_trilinos12.10.1.1.txt ################################################################################ Forge 7.0.4.1 ======== Release Date: ------------- June 01, 2017 Purpose: -------- -Release of Allinea Forge 7.0.4.1 Product and OS Dependencies: ---------------------------- The Cray Forge 7.0.4.1 release is supported on the following Cray systems running Cray Linux Environment(CLE) operating system - Cray XE and Cray XK systems running CLE 5.2 - Cray XC systems with CLE version 5.2 or 6.0 or later Documentation: -------------- http://www.allinea.com/products/ddt-support /opt/allinea/forge/7.0.4.1/doc/RELEASE-NOTES Installation for CLE 5.2: ------------------------- rpm -ivh forge-7.0.4.1-1.sles11.x86_64.rpm To make this the default version, execute: /opt/allinea/forge/7.0.4.1/set_default_forge_7.0.4.1 Certain components, files or programs contained within this package or product are Copyright 2008-2016 Cray Inc. All rights reserved. ################################################################################ Forge 7.0.4.1 ======== Release Date: ------------- June 01, 2017 Purpose: -------- -Release of Allinea Forge 7.0.4.1 Product and OS Dependencies: ---------------------------- The Cray Forge 7.0.4.1 release is supported on the following Cray systems running Cray Linux Environment(CLE) operating system - Cray XE and Cray XK systems running CLE 5.2 - Cray XC systems with CLE version 5.2 or 6.0 or later Documentation: -------------- http://www.allinea.com/products/ddt-support /opt/allinea/forge/7.0.4.1/doc/RELEASE-NOTES Installation for CLE 6.0: ------------------------- rpm -ivh forge-7.0.4.1-1.sles12.x86_64.rpm To make this the default version, execute: /opt/allinea/forge/7.0.4.1/set_default_forge_7.0.4.1 Certain components, files or programs contained within this package or product are Copyright 2008-2016 Cray Inc. All rights reserved. ################################################################################ Perftools 6.5.0 =============== Release Date: June 29, 2017 Purpose: ======== This is a major feature and bug fix release for Cray XC and CS/300/400/Storm systems. Key enhancements or changes from the previous release: ====================================================== o All files produced when collecting performance data using CrayPat will now be consolidated into a single, unique experiment data directory (see Notes section below for more details) o Reduced pat_report processing time, especially for perftools-lite* experiments where pat_report runs on the compute nodes o CUDA support in Apprentice2 (Overview, GPU timeline) o Intel(r) Xeon(r) processor Scalable family performance counters o perftools-base loaded by default starting with cdt-prgenv 6.0.4 (available in May) o perftools-lite enhancement available through environment variables CRAYPAT_LITE_BLACKLIST and PAT_LD_BLACKLIST which prevents instrumentation of test programs built through CMake and GNU Autotools build systems as part of application builds (see the pat_build man page for more information) o New environment variables added: PAT_RT_EXPDIR_BASE, PAT_RT_EXPDIR_CLEANUP, PAT_RT_EXPDIR_FSLOCK, PAT_RT_EXPDIR_NAME, PAT_RT_EXPDIR_REPLACE (see intro_craypat man page for more info) o Removed environment variables: PAT_RT_EXPFILE_APPEND, PAT_RT_EXPFILE_DIR, PAT_RT_EXPFILE_FSLOCK, PAT_RT_EXPFILE_NAME, PAT_RT_EXPFILE_REPLACE, PAT_RT_EXPFILE_SUFFIX (functionality replaced by PAT_RT_EXPDIR_* environment variables) o PAT_RT_EXPFILE_PES issues FATAL message if a subset of PEs is specified o Trace wrappers for Fortran 77 and Fortran 90 NetCDF functions o Upgrade to PAPI 5.5.1.2 o Upgrade to DWARF 16.8.0 Bugs Fixed: =========== o 842836 - app2: When zooming in on a traffic report the horizontal scroll goes to a random location o 848426 - app2: App2 does not show GPU usage for CUDA codes o 849023 - reveal/6.4.6: font resizing in reveal is not working o 849453 - CrayPAT6.4.6 error on cle 5.2 o 850073 - craypat instrumented zladiv_ from libsci_intel causes segfault o 850100 - Application crashes when PAT_RT_EXPFILE_PES is set o 850424 - app2: "Cannot find a pdf viewer" Notes: ====== o 841275 - mismatch between version of libc installed on login and compute nodes causes pat_report to abort on compute nodes for CS systems running RHEL/CentOS 7 - perftools software expects that libc versions match between the login and compute nodes o 841220 - Intel/16.x: -ipo prohibits dwarf generation for Fortran, C or C++ compilation o 851978 - app2: doing message matching requires more memory than is available o New experiment data directory: o Files included in new data directory: .xf, .apa, MPICH_RANK_ORDER, .rpt, .ap2 files o Directory name will follow the same naming convention currently used for a directory of .xf files: myprog+pat_PID-node…/ o Directory name can be passed to pat_report, reveal and app2. pat_report will know to process .xf files to produce .ap2 files, and will know how to run different reports if you pass it the data directory name. o pat_report produces a directory of .ap2 files, similar to the directory of .xf files produced when running larger jobs. This too lives inside the data directory. o Example directory from perftools-lite experiment: > ls vhone+23415-12s/ MPICH_RANK_ORDER.USER_Samp ap2-files index.ap2 rpt-files xf-files o Example usage: o View a saved lite report (.rpt file, which also goes to STDOUT as in the past): > cd vhone+23415-12s/rpt-files > cat RUNTIME.rpt o Generate full report from perftools-lite experiment: > pat_report vhone+23415-12s > my_rpt o Generate full report from perftools-lite experiment from within data directory: > cd vhone+23415-12s > pat_report . > my_rpt Dependencies: ============= o The 17.06 PE release is a compatibility release. All PE components of this release are dependent on each other. If you wish to use a previous version of a PE component such as cray-mpich, you should swap all PE modules to a previous PE release using the cdt module. o CLE 6.0UP04 (Intel Broadwell uncore performance counters) o cdt-prgenv 6.0.4 (perftools-base loaded by default) No longer supported: -------------------- Red Hat(R) Enterprise Linux(R) 6 and CentOS 6 on CS/300/400/Storm systems Cray XC systems running CLE 5.2 -------------------------------- o craype 2.5.11 o papi/5.5.1.2 o One or more compilers running these minimum versions: CCE 8.6.0 GCC 4.9, 5.3, 6.3 Intel 17 o Cray Scientific Libraries (LibSci) 17.06.1 o LibSci_ACC 17.03.1 for Cray XC systems with GPUs o Cray MPT 7.6.0 o CUDA 8.0 (including cudatoolkit) on Cray XC systems with GPUs o Chapel 1.15.0 Cray CS Systems: ---------------- o CCE 8.6.0 o papi/5.5.1.2 o libsci 17.06.1 o libsci_acc 17.03.1 o craype 2.5.11 o CUDA 8.0 (including cudatoolkit) on Cray CS systems with GPUs o mvapich 2.2rc.1.0.3 Non-Cray systems: ------------------ Apprentice2 is supported on non-Cray systems running: o Windows 7 (for the desktop version of Cray Apprentice2) o Mac OS X 10.7.5 or later Documentation: ============== Installation instructions: ========================== For Cray XC systems, see the "Cray Programming Environments Installation Guide" for more information. For Cray XE/XK systems, see the "Cray Performance Measurement and Analysis Tools Installation Guide" for more information. Installation of perftools for CLE 5.2 ------------------------------------- rpm -ivh --prefix=/opt/cray cray-dwarf-16.8.0-201704051817.45b9161c9249b-0.x86_64.rpm rpm -ivh --prefix=/opt/cray perftools-6.5.0-201706202039.a3f9a46869f6f-2.x86_64.rpm To make perftools/6.5.0 the default version of perftools, execute: /opt/cray/admin-pe/set_default_files/set_default_perftools_6.5.0 Installation of PAPI for CLE 5.2 -------------------------------- rpm -ivh --prefix=/opt/cray cray-papi-5.5.1.2-201704041539.c93cdc1bafdd4-0.x86_64.rpm To make PAPI 5.5.1.2 the default version of PAPI, execute: /opt/cray/admin-pe/set_default_files/set_default_papi_5.5.1.2 Installation of perftools for CLE 6 and later: ---------------------------------------------- rpm -ivh cray-dwarf-16.8.0-201704051817.45b9161c9249b-0.x86_64.rpm rpm -ivh perftools-6.5.0-201706202039.a3f9a46869f6f-2.x86_64.rpm To make perftools/6.5.0 the default version of perftools, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_perftools_6.5.0 Installation of PAPI for CLE 6 and later: ----------------------------------------- rpm -ivh cray-papi-5.5.1.2-201704041539.c93cdc1bafdd4-0.x86_64.rpm To make PAPI 5.5.1.2 the default version of PAPI, execute: /opt/cray/pe/admin-pe/set_default_files/set_default_papi_5.5.1.2 Installation of perftools for Cray CS/300/400/Storm --------------------------------------------------- See Installing the Cray Programming Environment for the Cray CS Systems for instructions. Installation of app2 remote client (+ server) on Mac systems: ------------------------------------------------------------- Apprentice2Installer-6.5.0-2.dmg The Cray Apprentice2 installer for Mac is included in the perftools-clients rpm, and placed in $CRAYPAT_ROOT/share/desktop_installers/ on a Cray login node when the perftools-clients rpm is installed. Download the Cray Apprentice2 installer onto a desktop or laptop running Mac OS. Double click on installer to begin installation. The installer will walk you through the process for your system. Installation of app2 remote client (+ server) on Windows 7 systems: ------------------------------------------------------------------- Apprentice2Installer-6.5.0-2.exe The Cray Apprentice2 installer for Windows is included in the perftools-clients rpm, and placed in $CRAYPAT_ROOT/share/desktop_installers/ on a Cray login node when the perftools-clients rpm is installed. Download the Cray Apprentice2 installer onto a desktop or laptop running Windows 7. Double click on installer to begin installation. The installer will walk you through the process for your system. Documentation: -------------- License: Except for the third party modules discussed below and software licensed by Cray through proprietary agreements, components, files or programs contained within this package or product are Copyright 2001-2017 Cray Inc. All rights reserved. Attribution notices for open source licensed software contained in this package are detailed in the file: $CRAYPAT_ROOT/ATTRIBUTIONS_perftools.txt ################################################################################ cray-python 17.06.1 =================== Release Date: ------------- June 29, 2017 Purpose: -------- The cray-python 17.06.1 release contains python 2.7.13 python 3.5.2 numpy 1.11.3 scipy 0.18.1 mpi4py 2.0.0 dask 0.13.0 The numpy and scipy packages are configured to call cray-libsci routines. The mpi4py package is configured to call cray-mpt routines. Product and OS Dependencies: ---------------------------- The cray-python 17.06.1 release is supported on Cray XC systems with CLE 6.0 and higher. Documentation: -------------- https://www.python.org/doc/ Modulefile: --------------------- module load cray-python/17.06.1 Installation: ------------- rpm -ihv cray-python-17.06.1-201706210417.8be3e5bf52459-02.sles12.x86_64.rpm To make this the default version, execute: /opt/admin-pe/set_default_files/set_default_python_17.06.1 Certain components, files or programs contained within this package or product are Copyright -2017 Cray Inc. All rights reserved. ################################################################################