David Patterson:
A New Golden Age for Computer Architecture
Only performance path left is Domain Specific Architectures
Renaissance of hardware design
Afford us extreme amounts of capacity and flexibility.
Increasing complex, spawning industries to optimize cost.
Big cloud providers now sell unused capacity at a lower price (up to 90% savings).
(https://azureprice.net/: A VM with \(120C\)/\(456G\) RAM can cost only \(0.36\) USD/hr.)
To optimize performance/cost, one needs to consider:
This is unlike traditional scheduling problems, where the amount of resources is fixed.
And unlike cloud optimization solutions, we only focus on data analytics workloads.
A Spark workflow can be represented by a DAG,
in which vertices are RDDs and edges are operations
Source: Understanding your Apache Spark Application Through Visualization
val conf = new SparkConf().setAppName("wiki_test") // create a spark config object
val sc = new SparkContext(conf) // Create a spark context
val data = sc.textFile("/path/to/somedir") // Read files from "somedir" into an RDD of (filename, content) pairs.
val tokens = data.flatMap(_.split(" ")) // Split each file into a list of tokens (words).
val wordFreq = tokens.map((_, 1)).reduceByKey(_ + _) // Add a count of one to each token, then sum the counts per word type.
wordFreq.sortBy(s => -s._2).map(x => (x._2, x._1)).top(10) // Get the top 10 words. Swap word and count to sort by count.
Source: Understanding the working of Spark Driver and Executor
Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.
Can be used as cluster manager for Spark.
Spark 3+ supports accelerators such as GPUs as custom resources (i.e., accelerator-aware scheduling).
ResourceProfile: allows the user to specify executor and task requirements for an RDD that will get applied during a stage.
Dynamic allocation: dynamically scales cluster resources based on workload.
Previous work re. dynamic scaling:
Move data to different node before preemption.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.16"
}
}
required_version = ">= 1.2.0"
}
provider "aws" {
region = "us-west-2"
}
resource "aws_instance" "app_server" {
ami = "ami-830c94e3"
instance_type = "t2.micro"
tags = {
Name = "ExampleAppServerInstance"
}
}
digraph G {
compound = true
fontname = "Palatino Linotype"
bgcolor = "#00000000"
node [ fontname = "Palatino Linotype" ]
subgraph cluster_cloud {
label = "Cloud service / k8s cluster"
n11, n12 [ label = "highcpu" ]
{n11, n12} -> {n21, n22, n23}
n21, n22, n23 [ label = "highmem" ]
{n21, n22, n23} -> {n31, n32}
n31, n32 [ label = "gpu" ]
{n31, n32} -> n41
n41 [ label = "driver" ]
}
monitor [label = "Pricing monitor"]
monitor -> model
model [ label = "Estimator" ]
n22 -> model [ ltail = cluster_cloud ]
opt [ label = "Optimizer" ]
instance [ label = "Instance selection"]
sched [ label = "Spark / Cluster manager" ]
model -> opt -> instance -> sched
sched -> n22 [ lhead = cluster_cloud ]
n22 -> sched [ ltail = cluster_cloud ]
}
Since estimation and integration is complex,
we mostly deal with the optimization part here.
Recall that a Spark workflow can be represented as a DAG \((V, E)\).
Let \(I\) be the set of possible instance types.
Assume that we have the following estimators:
We then want to find a mapping \(\theta: E \to I\) that minimizes both
It seems that the objectives above can be optimized via a greedy algorithm.
This is because we have yet to consider network costs \(f_n\) and \(g_n\).
We can then proceed to optimize this via
the multi-objective genetic algorithm NSGA-II.
problem = Problem(
n_rdds=3,
n_instances=3,
edges=[
[0, 1, [[10, 1], [5, 3], [100, 1]]],
[1, 2, [[100, 1], [500, 3], [10, 5]]],
],
network_cost=[
[[0, 0], [1, 1], [1, 1]],
[[1, 1], [0, 0], [1, 1]],
[[1, 1], [1, 1], [0, 0]],
],
)
~/work/hpc via 🐍 v3.9.12 took 25s
✦ ❯ python -i main.py
=======================================================
n_gen | n_eval | n_nds | eps | indicator
=======================================================
1 | 9 | 2 | - | -
2 | 9 | 2 | 0.00000E+00 | f
[0 2] [20. 60.]
[1 2] [15. 65.]
Automatically provision workers on the cloud w/ accelerators and spot instances to ensure fast large-scale data processing at a low cost.