NERSCPowering Scientific Discovery Since 1974

Spark Distributed Analytic Framework

Description and Overview

Apache Spark™ is a fast and general engine for large-scale data processing.

How to Use Spark

Because of its high memory and I/O bandwidth requirements, we recommend you run your spark jobs on Cori. Also, starting with spark 2.1.1, we recommend that you run Spark inside of Shifter. This will improve performance and usability by utilizing Shifter's per-node file cache for shuffle files and temporary data files. Without this functionality, these files are written either to your scratch directory (which is not optimized for repeated accesses of many small files) or the RAM file system at /tmp (which removes memory from the node doing the calculation and can lead to the node crashing from lack of memory).

Follow the steps below to use spark, note that the order of the commands matters. DO NOT load the spark module until you are inside a batch job. 

Interactive mode

Submit an interactive batch job with at least 2 nodes:

salloc -N 2 -t 30 -C haswell --qos=interactive --image=lgerhardt/spark-2.1.1:v1 --volume="/global/cscratch1/sd/<user_name>/tmpfiles:/tmp:perNodeCache=size=200G"

This will request a job with the Spark 2.1.1 Shifter image. It also sets up an xfs file on each node as a per node cache, which will be accessed inside the Shifter image via /tmp. By default, Spark will use this as the directory it caches temporary files.

Wait for the job to start. Once it does you will be on a compute node and you will need to load the Spark module:

export EXEC_CLASSPATH=path_to_any_extra_needed_jars #Only required if you're using external libraries or jarfiles
module load spark/2.1.1

You can start Spark with this command:

start-all.sh

To connect to the Python Spark Shell, do:

shifter pyspark

To connect to the Scala Spark Shell, do:

shifter spark-shell

 to shutdown the Spark cluster, do:

stop-all.sh

Batch mode

Below are example batch scripts for Cori. You can change number of nodes/time/queue accordingly (so long as the number of nodes is greater than 1). On Cori you can use the debug queue for short, debugging jobs and the regular queue for long jobs.
Here's an example script for Cori called run.sl:

#!/bin/bash

#SBATCH -p regular #SBATCH -N 2
#SBATCH -C haswell #SBATCH -t 00:30:00 #SBATCH -e mysparkjob_%j.err #SBATCH -o mysparkjob_%j.out
#SBATCH --image=lgerhardt/spark-2.1.1:v1
#SBATCH --volume="/global/cscratch1/sd/<user_name>/tmpfiles:/tmp:perNodeCache=size=200G"
export EXEC_CLASSPATH=path_to_any_extra_needed_jars #Only required if you're using external libraries or jarfiles
module load spark/2.1.1
start-all.sh

shifter spark-submit $SPARK_EXAMPLES/python/pi.py
stop-all.sh

To submit the job:

sbatch run.sl

 

Instructions for Older Spark Versions Using Modules

Submit an interactive batch job with at least 2 nodes:

salloc -N 2 -t 30 -C haswell --qos=interactive

Wait for the job to start. Once it does you will be on a compute node and you will need to load the spark module:

module load spark/2.0.0

You can start spark with this command:

start-all.sh

To connect to the Python Spark Shell, do:

pyspark

To connect to the Scala Spark Shell, do:

spark-shell

 to shutdown the Spark cluster, do:

stop-all.sh

Batch mode

Below are example batch scripts for Cori. You can change number of nodes/time/queue accordingly (so long as the number of nodes is greater than 1). On Cori you can use the debug queue for short, debugging jobs and the regular queue for long jobs.
Here's an example script for Cori called run.sl:

#!/bin/bash

#SBATCH -p regular #SBATCH -N 2
#SBATCH -C haswell #SBATCH -t 00:30:00 #SBATCH -e mysparkjob_%j.err #SBATCH -o mysparkjob_%j.out
module load spark
start-all.sh
spark-submit $SPARK_EXAMPLES/python/pi.py
stop-all.sh

To submit the job:

sbatch run.sl

Monitoring Your Spark Application

Running the History Server 

The history server allow you to visualize the information provided by the event logs in a nice interactive web interface. Here are instructions to run it on Cori (note the history server is independent of running a spark job, so no need to start a spark job to run the server):

Run the following commands in a login node (do not run in a compute node):

module load spark/hist-server
run_history_server.sh

This command will return a url that will look something like this: http://120.44.234.30:18080

Go to the address returned in your browser on your local machine.

 

Alternatively, if you are in an NX or in an X11 forwarded ssh session, you can enter

firefox

 which will open a firefox browser from the login node. From there, enter "localhost:18080" as the url to see the history server.

 

Initially, the page will display "No completed applications found!" until all logs are processed. This processing can take anywhere from a minute to ten minutes depending on how many event logs you have accumulated

 

To get a quicker turnaround time, consider using an event logs directory with fewer event logs (if possible!)

 

Have fun!

Make sure to stop the server when you are done. To stop the history server:

run_history_server.sh --stop

Note you must be on the same login node, where you started the history server, in order to stop it. 

Trouble Shooting 

Module Load Errors

A successful module load spark has four steps and looks like this:

Creating Directory SPARK_WORKER_DIR /global/cscratch1/sd/racah/spark/1054167
Creating /global/cscratch1/sd/racah/spark/1054167/slaves file
Determining the master node name...
Master node is nid00092

Module load error outputs look like this:

spark/1.6.0(137):ERROR:102: Tcl command execution failed: if { [ module-info mode load ] } { 
puts stderr "Creating Directory SPARK_WORKER_DIR $env(SPARK_WORKER_DIR)" 
puts stderr "Creating $env(SPARK_WORKER_DIR)/slaves file" 
puts stderr "Determining the master node name..." 
set master [exec $root/myfindmaster.sh] 
puts stderr "Master node is $master" 

exec /bin/mkdir -p $env(SPARK_WORKER_DIR) 
exec $root/myfindslaves.sh $master $env(SPARK_WORKER_DIR)/slaves 
setenv SPARKURL spark://$master:7077 

setenv SPARKMASTER $master 

If the module load error comes after 

Master node is nid00092

OR

Determining the master node name...

then check your ~/.bashrc.ext and ~/.bash_profile.ext files to make sure they produce no errors upon login like:

ModuleCmd_Load.c(226):ERROR:105: Unable to locate a modulefile for

if you receive this error:

SLURM_JOBID not set, please run this module inside of a batch job

you have called module load either inside a login node or in one of the compute nodes that is not the initial one that you were placed on when your job started.

Memory Limits

Sometimes  jobs can start with incorrect user memory limits. You can add "ulimit -s unlimited" to your .bashrc.ext file to avoid this.

Further spark documentation is available from the Apache Spark Web Page. 

Availability

Spark is Available on Edison and Cori. However, we recommend you run on Cori because of its larger memory nodes and faster scratch connection.

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
Spark cori applications/ debugging 1.5.1 spark/1.5.1 2016-07-18
 Spark data analytic framework
Spark cori applications/ debugging 1.6.0 spark/1.6.0 2016-07-18
 Spark data analytic framework
Spark cori applications/ debugging 2.0.0 spark/2.0.0 2016-08-17
 Spark data analytic framework
Spark cori applications/ debugging 2.1.0 spark/2.1.0 2017-05-05
 Spark data analytic framework
Spark cori applications/ debugging 2.1.1 spark/2.1.1 2017-06-22
 Spark data analytic framework
Spark cori applications/ debugging hist-server spark/hist-server 2016-02-01
 Spark history server for monitoring jobs after the fact
Spark edison applications/ debugging 1.5.1 spark/1.5.1 2017-07-24
 Spark data analytic framework
Spark edison applications/ debugging 1.6.0 spark/1.6.0 2017-07-24
 Spark data analytic framework
Spark edison applications/ debugging 2.0.0 spark/2.0.0 2017-07-24 2017-07-24
 Spark data analytic framework
Spark edison applications/ debugging 2.1.1 spark/2.1.1 2017-07-24
 Spark data analytic framework
Spark edison applications/ debugging hist-server spark/hist-server 2017-07-24
 Spark history server for monitoring jobs after the fact