Skip to content

README.md

Latest commit

 

History

History
160 lines (122 loc) · 5 KB

README.md

File metadata and controls

160 lines (122 loc) · 5 KB
Raw

Graph Analytics - Spark Graph Processing Workload

This workload tests Spark graph processing performance using large Twitter datasets. Based on CloudSuite Graph Analytics benchmark, adapted to run in a single container environment.

Note: Large data files are stored in GitHub Releases and need to be downloaded manually. The workload uses standalone mode where the same container contains both Spark and the graph processing application.

Manual File Setup (Required)

Download these files from the GitHub release and place them in the correct locations:

  1. Download from release: https://github.com/Anjali05/platform-workloads/releases/tag/graphAnalytics-v1.0.0

  2. Place files in these locations:

    cloud-workloads/graphAnalytics/twitter-data/edges.csv.gz     # 310MB compressed Twitter graph data
cloud-workloads/graphAnalytics/host/spark-3.3.2-bin-hadoop3-scala2.13.tgz  # 293MB Spark binary

  3. Extract the compressed data (the Dockerfile will handle this):

    cd cloud-workloads/graphAnalytics/twitter-data
gunzip edges.csv.gz  # Creates the 1.3GB edges.csv file

For Docker (runc/runsc)

Use the provided Dockerfile to build the standalone graph analytics image.

# Build the image (handles data extraction automatically)
docker build -t mygraphanalytics .

# Start container 
docker run -id --rm --name worker \
  --cpuset-cpus="0-3" \
  --memory="24g" \
  -e WORKLOAD_NAME="pr" \
  -e NUM_ITER="1" \
  mygraphanalytics

Run Workload

Standard PageRank benchmark:

docker exec worker /root/run_benchmark.sh --driver-memory "8g" --executor-memory "8g"

With custom parameters:

# PageRank with 5 iterations
docker run -e WORKLOAD_NAME="pr" -e NUM_ITER="5" mygraphanalytics

# Connected Components
docker run -e WORKLOAD_NAME="cc" -e NUM_ITER="1" mygraphanalytics

# Triangle Counting  
docker run -e WORKLOAD_NAME="tc" -e NUM_ITER="1" mygraphanalytics

For Firecracker (fc)

Building the Firecracker Image

Resize the base image to 16GB, then boot the VM and install Spark 3.3.2 following commons/spark/3.3.2/Dockerfile. Spark should be installed at /opt/spark-3.3.2.

Copy the workload files into the VM:

# edges.csv is available from the GitHub release (see Manual File Setup above)
mkdir -p /root/graph-analytics
cp benchmark/graph-analytics-2.0.jar /root/graph-analytics/
cp <path-to>/edges.csv /root/graph-analytics/edges.csv

Set SPARK_LOCAL_IP=169.254.0.1 in /etc/environment or /etc/profile.d/spark.sh so it persists across SSH sessions.

1. Environment Setup

Set required Spark environment variable:

export SPARK_LOCAL_IP=169.254.0.1

2. Run Workload

PageRank (default):

/opt/spark-3.3.2/bin/spark-submit \
  --class GraphAnalytics \
  --driver-memory 8g \
  --executor-memory 8g \
  /root/graph-analytics/graph-analytics-2.0.jar \
  -file=/root/graph-analytics/edges.csv \
  -app=pr \
  -niter=1

Connected Components:

/opt/spark-3.3.2/bin/spark-submit \
  --class GraphAnalytics \
  --driver-memory 8g \
  --executor-memory 8g \
  /root/graph-analytics/graph-analytics-2.0.jar \
  -file=/root/graph-analytics/edges.csv \
  -app=cc \
  -niter=1

Triangle Counting:

/opt/spark-3.3.2/bin/spark-submit \
  --class GraphAnalytics \
  --driver-memory 8g \
  --executor-memory 8g \
  /root/graph-analytics/graph-analytics-2.0.jar \
  -file=/root/graph-analytics/edges.csv \
  -app=tc \
  -niter=1

Workload Parameters

Parameter Description Options
-app Graph algorithm to run pr (PageRank), cc (Connected Components), tc (Triangle Counting)
-niter Number of iterations Integer (e.g., 1, 5, 10)
-file Input graph file /root/graph-analytics/edges.csv
--driver-memory Spark driver memory e.g., 8g, 16g
--executor-memory Spark executor memory e.g., 8g, 16g

Environment Variables (Docker)

Variable Description Default
WORKLOAD_NAME Graph algorithm pr
NUM_ITER Number of iterations 1

Dataset Information

  • Source: Twitter social graph dataset
  • Format: Edge list (CSV)
  • Size: 1.3GB uncompressed, 310MB compressed
  • Nodes: ~41M nodes
  • Edges: ~1.47B edges

Performance Notes

  • Memory requirements scale with dataset size
  • PageRank typically requires multiple iterations for convergence
  • Connected Components and Triangle Counting are typically single-pass algorithms
  • Adjust Spark memory settings based on available system resources

Files

  • Dockerfile - Standalone container with Spark and graph analytics
  • twitter-data/ - Graph dataset (download from releases)
  • host/ - Host-specific files and Spark binaries
  • benchmark/ - Scala source code for graph algorithms