NERSCPowering Scientific Discovery Since 1974

Roofline Training 2/16/2017 - 2/17/2017

January 23, 2017

NERSC will host a Roofline Training at the CRT building (Bld 59, room 4102) on the LBNL campus from February 16 to 17. The meeting is intended for advanced users who are interested in learning about new optimization tools and techniques. We will focus on the roofline model and new roofline features in the Intel Advisor. The training will be mostly hands-on with roofline experts from Intel available on-site for advice and personal 1-1 assistance on your application. There will be one or two introductory talks and case studies presented.

The first day of the Roofline Training will run in parallel with the last day of the Cori KNL training on Thursday. We will make an effort to interleave talks so that those interested can attend the talks of both events.

Please contact Tuomas Koskela (NERSC) or Zakhar Matveev (Intel) for more information.

Note: To participate in the hands-on sessions (most of the training), participants should Bring Your Own Codes (BYOC) to work on.


Please fill in the registration form to attend the training. Note: If you are bringing a code to the hands-on BYOC sessions, fill in its name and a very brief description.

Remote Connection Information

Topic: Roofline Tutorial Day 1 Talks
Time: Feb 16, 2017 9:00 AM Pacific Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android:

Or iPhone one-tap (US Toll): +14086380968,282551618# or +16465588656,282551618#

Or Telephone:
Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll)
Meeting ID: 282 551 618
International numbers available:

Topic: Roofline Tutorial Day 2 Talks
Time: Feb 17, 2017 1:00 PM Pacific Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android:

Or iPhone one-tap (US Toll): +14086380968,699688930# or +16465588656,699688930#

Or Telephone:
Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll)
Meeting ID: 699 688 930
International numbers available:



Thursday February 16

Morning Session (9am - noon):

  • Practical Arrangements (Tuomas Koskela, NERSC postdoc)
  • Introduction to the Many Flavors of the Roofline Model (Tuomas Koskela)
  • Introduction and demo of Roofline Model automation (Zakhar Matveev, Intel product architect)
  • Contrast algorithmic (cache-aware) and platform (DRAM-focused) flavors of Roofline automation (Zakhar Matveev)

Working Lunch (noon - 1pm):

  • Setup for hands-on session

Afternoon Session (1pm - 5pm):

  • Hands on (BYOC)
  • Closing: Lessons learned and feedback

Friday February 17

Morning Session (9am - noon):

  • Hands on (BYOC)

Working Lunch (noon - 1pm):

  • Case studies of application optimization using roofline (Tuomas Koskela, NERSC)

Afternoon (1pm - 5pm):

  • Hands on (BYOC)
  • Closing: Lessons learned and feedback

Online Roofline Material

  1. Utilizing Roofline Analysis in the Intel® Advisor to Deliver Optimized Performance for Applications on Intel® Xeon Phi™ Processor (Code Named Knights Landing)
  2. Roofline Analysis in Intel Advisor 2017
  3. Getting started with Intel® Advisor Roofline feature
  4. Measuring Arithmetic Intensity
  5. A New Way to Visualize Performance Optimization Trade-Offs
  6. Roofing a House

Advisor HowTo

  • How to Compile and Run
  • Advisor Version 2017 Update 1 is Installed on Cori.
    • To add binaries to your path, simply use `module load advisor/2017.up1
    • To use the roofline feature in 2017 Up1 (in GUI and cl), set env variable `ADVIXE_EXPERIMENTAL=roofline`
  • Local Installations of Unreleased Versions Provided for This Training can be Found Here:
    • 2017 Update2 : /project/projectdirs/m1759/advisor_2017.1.2.501009
    • To add binaries to your path, use `source` found in the above directory
  • To collect data for roofline, do two collections: survey and tripcounts. Add –flops-and-masks flag to tripcounts.
    • srun -n <num-of-ranks> -c <num_of_cores_per_rank> advixe-cl -v -collect survey -project-dir=<same_dir_name> -data-limit=0 <your_executable>
    • srun -n <num-of-ranks> -c <num_of_cores_per_rank> advixe-cl -v -collect tripcounts -flops-and-masks -project-dir=<same_dir_name> -data-limit=0 <your_executable>
    • Running jobs on Cori
  • To use DRAM roofline, add --vtune flag to #SBATCH/salloc (sets perf_event_paranoid=0)
  • On Cori, run in $SCRATCH
  • GUI works reasonably well through NX, see
  • If overhead from Advisor is too big, workaround is to add these flags to survey command line
    • -interval=40
    • -stackwalk-mode=online
    • -no-stack-stitching