NERSCPowering Scientific Discovery Since 1974

Roofline Training 11/8/2017 - 11/9/2017

November 8, 2017

In collaboration with Intel and LBNL's Computational Research Division, NERSC will host a Roofline Training at the CRT building (Bld 59, room 3101) on the LBNL campus from November 8 to 9. The meeting is intended for advanced users who are interested in learning about new optimization tools and techniques. We will focus on the roofline model and new roofline features in the Intel Advisor including DRAM roofline data collection and cache-simulator based roofline collections. The training will be mostly hands-on with roofline experts from Intel available on-site for advice and personal 1-1 assistance on your application. 

Please contact Charlene Yang (NERSC) or Zakhar Matveev (Intel) for more information.

Note: To participate in the hands-on sessions (most of the training), participants should Bring Your Own Codes (BYOC) to work on.


Register at the following link:

Remote Connection Information

Topic: 2nd Roofline Training Day 1 Meeting
Time: Nov 8, 2017 11:45 AM Pacific Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android:

Or iPhone one-tap :
US: +16699006833,,993441129# or +16468769923,,993441129#
Or Telephone:
Dial(for higher quality, dial a number based on your current location):
US: +1 669 900 6833 or +1 646 876 9923
Meeting ID: 993 441 129
International numbers available:

Topic: 2nd Roofline Training Day 2 Meeting
Time: Nov 9, 2017 8:45 AM Pacific Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android:

Or iPhone one-tap :
US: +16468769923,,533484868# or +16699006833,,533484868#
Or Telephone:
Dial(for higher quality, dial a number based on your current location):
US: +1 646 876 9923 or +1 669 900 6833
Meeting ID: 533 484 868
International numbers available:

Draft Agenda

Wednesday November 8

12:00 - 12:30 : Welcome (C. Yang, T. Koskela)

12:30 - 13:00 : Introduction to Roofline (S. Williams)

13:00 - 14:00 : Introduction to Advisor, Running on Cori (Z. Matveev, T. Koskela)

14:00 - 14:30 : Break

14:30 - 17:00 : Hands-on

Thursday November 9

09:00 - 09:30 : Roofline Usage at LANL (R. Green)

09:30 - 15:00 : Hands-on + Breaks/Lunch

15:00 - 16:00 : Reporting and Sync Up

Online Roofline Material

  1. Utilizing Roofline Analysis in the Intel® Advisor to Deliver Optimized Performance for Applications on Intel® Xeon Phi™ Processor (Code Named Knights Landing)
  2. Roofline Analysis in Intel Advisor 2017
  3. Getting started with Intel® Advisor Roofline feature
  4. Measuring Arithmetic Intensity
  5. A New Way to Visualize Performance Optimization Trade-Offs
  6. Roofing a House

Advisor How-To

  • Please compile codes with debug mode and link object files/libraries dynamically
  • Please run on $SCRATCH file system
  • To quickly get an interactive session, try 'salloc --qos=interactive -C haswell/knl,quad,cache -N 1 -t hh:mm:ss -A <your_account>' 
  • Two new versions of Intel Advisor are available: 'module load advisor/2018.up1' or 'module load advisor/2018.integrated_roofline'
  • To collect data for Roofline analysis, please run two data collections: survey and tripcounts
    • srun -n <num-of-ranks> -c <num_of_cores_per_rank> advixe-cl -v -collect survey -project-dir=<same_dir_name> -data-limit=0 <your_executable>
    • srun -n <num-of-ranks> -c <num_of_cores_per_rank> advixe-cl -v -collect tripcounts -flops-and-masks -project-dir=<same_dir_name> -data-limit=0 <your_executable>
    • Please add '-no-auto-finalization' for KNL when running survey data collection 
  • To view the results, you can either run NX on Cori or download the results to your laptop and view them offline
    • For NX:; please module load the correct version (you collected the data with) and run 'advice-gui'
    • For Mac OS viewers, please download:
      • /global/common/cori/software/intel/advisor_2018.1.1.534062/mac_os_viewer/
      • /global/common/cori/software/intel/advisor_2018.0.2.534031/mac_os_viewer/
      • Please note the two viewers are incompatible
  • To pack results and necessary source/binary files for download, run
    • advixe-cl --snapshot --project-dir <same_dir_name> --pack --cache-sources --cache-binaries -- <target_file_name>
  • If overhead from Advisor is too high, try adding these flags to the survey data collection:
    • -interval=40
    • -stackwalk-mode=online
    • -no-stack-stitching

Data Collection

  • Once you have collected roofline data with Advisor
    • Please fill in this spreadsheet 
    • Create a snapshot of the Advisor result and share it with NERSC user tkoskela with the give command
      • give -u tskoskela <file-or-directory>

Sample Data

  • You can find a sample integrated roofline data of the stencil code presented in Sam Williams' talk on Cori in /project/projectdirs/m1759/stencil_advisor_snapshot/stencil.snapshot.advisor.2018.integrated.roofline.advixeexpz
    • use module load advisor/2018.integrated_roofline to view this file