Analyzing Spark Performance on Spot Instances

Transcription

1 Analyzing Spark Performance on Spot Instances Presented by Jiannan Tian Commi/ee Members David Irwin, Russell Tessier, Lixin Gao August 8, defense Department of Electrical and Computer Engineering 1

2 thesis overview 2

3 overview: Infrastructure as a Service expecta<on on IaaS high performance high availability low cost market offers a plethora of instance choices cost-availability tradeoff a pragmatical combination of instances low-cost IaaS provisions Amazon EC2: Spot Instances Google Compute Engine: preemptible instance Sample Spot price saving for general-purpose instances Type Saving to On-demand price Spot 1hr-fixed 6hr-fixed m3.medium 86% 45% 30% m3.large 80% 45% 30% m3.xlarge 85% 45% 30% m3.2xlarge 86% 45% 30% m4.large 88% 45% 30% m4.xlarge 89% 45% 30% m4.2xlarge 89% 45% 30% m4.4xlarge 88% 45% 30% m4.10xlarge 86% 45% 30% Spot Instances Preemptible VMs 3

4 overview: Infrastructure as a Service Cost-availability Tradeoff Bid-cost Expecta<on Price cheap expensive Reserved On-demand Fixed-duration Spot Total Cost low high bidding strategy 1 expected sweet point bidding strategy 2 guaranteed, non-revocable not guaranteed, non-revocable Availability not guaranteed, revocable low Bidding Level high 4

5 overview: Infrastructure as a Service Bid-cost Expecta<on Bid-cost Expecta<on Rev.1 Total Cost low high bidding strategy 1 expected sweet point bidding strategy 2 Total Cost low high bidding strategy 1 bidding strategy 1 w/ fault-tolerance low Bidding Level high low Bidding Level high 5

6 overview: Spark general-purpose incorporates MapReduce methodology in-house apps apps interfaces defined according to application in-memory compu<ng access and interfaces Spark SQL Spark Streaming MLib (machine learning) GraphX (graph) holding volatile states in memory lazy evaluation, low-latency, high-performance fault-tolerant processing engine storage Spark Core HDFS, S3 states re-generation from resource virtualization Mesos Hadoop YARN still kept in memory hardware infrastructure compatible with checkpoint write 6

7 challenges impact of revoca<on on Spark performance framework is not designed for high failure rate multiple factors affect performance benefits from running Spark on market and savings price in Spot market is much lower however, revocation increases run time expect sweet point what to do before effec<vely op<mizing to balance lineage exploit and checkpoint write to analyze performance and cost for different bids 7

8 contribu<ons To balance performance and cost, we applied Spark the fault-tolerant compute framework to Spot Market. effec<veness experiment we design effectiveness experiment to prove Spark with simple lineage mechanism to get over the frequent revocation and multiple factors that affect performance are discussed. design combined fault-tolerance mechanism after discussing the effectiveness of simply utilizing lineage mechanisms, we introduce optimum checkpoint write interval to distributed computing, in order to balance the overhead of checkpoint write to low-speed storage and lineage regression. implementa<on and evalua<on we implemented simulator for fast conducting experiments analyzing relation between bid level and run time and total cost. 8

9 background 9

10 background: Spot Instance instances reserved for job 1 instances reserved for job 2 instances reserved for job 1 (factual) instances reserved for job 2 (factual) idle resource pool 1 idle resource pool 2 Scenario I Scenario II 10

11 background: Spot Instance non-revocable job #1 non-revocable job #2 Sample m3.large Spot price X spot job #1 revoked spot job #2 not on Sept Spot price User bid Price (USD per hour) X X X X X X X X X X X X X X X X X X X X X X X Time of day Scenario III 11

12 background: market vola<lity Market Volatility Measuring Mean Revocation Number per 24 hours When Mean Bidding revocation Mean number Spot Price every in 24 Past hours 90 days m3.xlarge c3.2xlarge low mean price high volatility c3.4xlarge low mean price medium-high volatility g2.2xlarge low mean price low-medium volatility m3.medium c3.2xlarge c3.4xlarge c3.8xlarge c3.large c3.xlarge d2.2xlarge d2.4xlarge d2.8xlarge d2.xlarge g2.2xlarge g2.8xlarge i2.2xlarge i2.4xlarge i2.8xlarge i2.xlarge m3.2xlarge m3.large m3.medium m3.xlarge r3.2xlarge r3.4xlarge r3.8xlarge r3.large r3.xlarge 200 g2.8xlarge c3.large high mean price low mean price low volatility low volatility Mean Ratio spot (0 to price 1.0x) divided of Spot by Price same-type to On-demand on-demandprice price 12

13 background: Spark applica<on input map shuffle reduce output RDD_0_0 RDD_1_* RDD_2_0 raw data RDD_0_1 RDD_1_* RDD_2_1 RDD_0_2 RDD_1_* RDD_2_2 repeatedly executed on Spark (iterations) 13

14 background: Spark applica<on - transforma<on RDD_0_0 RDD_1_* raw data raw data are loaded into the cluster of instances and split in multiple small partitions on Spark, partitions are RDDs RDD_0_1 RDD_1_* map function (in this example) are executed calling transformation API RDD_0_2 RDD_1_* 14

15 background: Spark applica<on - ac<on _1_* RDD_2_0 raw data are loaded into the cluster of instances and split in multiple small partitions on Spark, partitions are RDDs map function (in this example) are _1_* RDD_2_1 output executed calling transformation API reduce function (in this example) are executed calling action API _1_* RDD_2_2 15

16 background: fault-tolerance mechanisms lineage interruption s0 s1 s2 s3 s0 s1 s2 downtime checkpoint interruption s0 s1 s2 s3 s2 s3 s4 downtime Routine overhead from checkpoint write, and is in vain if there is no revocation. Exploiting lineage results in restart form the very beginning of the lineage. So what is the optimum of checkpoint write interval? s checkpoint write a finished stage downtime stage number 16

17 design and implementation 17

18 design: modeling design first based on node states and transitions modeling implementation verification experiments noticed that there are always leading and lagging nodes only lagging node determine running time for this stage ignoring node competitions, focusing on partition independent from but hosted on nodes its processing times comprise total running time directly 18

19 design: modeling stage 0 stage 1 stage 2 stage 1 stage 2 stage 3 Strict partition synchronization ion based based RDD_0_0 RDD_0_1 RDD_0_2 RDD_0_3 RDD_0_4 RDD_0_5 checkpoint write RDD_1_0 RDD_1_1 RDD_1_2 RDD_1_3 RDD_1_4 RDD_1_5 RDD_2_0 RDD_2_1 RDD_2_2 xrdd_2_3 RDD_2_4 RDD_2_5 RDD_1_4 RDD_1_5 RDD_2_4 RDD_2_5 node up RDD_3_0 RDD_3_1 RDD_3_2 RDD_3_3 RDD_3_4 RDD_3_5 stage 0 stage 1 stage 2 stage 3 stage 4 Non-strict partition synchronization based RDD_A_0_0 RDD_A_0_1 RDD_B_0_0 RDD_B_0_1 RDD_C_0_0 RDD_C_0_1 RDD_A_1_0 RDD_A_1_1 RDD_B_1_0 RDD_B_1_1 RDD_C_1_0 RDD_C_1_1 RDD_A_2_0 RDD_A_2_1 RDD_B_2_0 xrdd_b_2_1 RDD_C_2_0 RDD_C_2_1 RDD_A_3_0 RDD_A_3_1 RDD_B_3_0 RDD_B_3_1 RDD_C_1_0 RDD_C_1_1 RDD_A_4_0 RDD_A_4_1 RDD_B_4_0 RDD_B_4_1 RDD_C_2_0 RDD_C_2_1 checkpoint write node up 19

20 design: modeling checkpoint enabled latest checkpoint exception caught Presumedly during one job there is no repartitioning, and a partition is not annihilated when its hosted node is revoked. Partition life cycle checkpoint disabled launched start designated being processed sync-ed finished exit (stage+1) checkpoint enabled Simplified cluster life latest checkpoint checkpoint disabled exception caught interruption cycle launched start designate partitions commit changes process partitions try launching new nodes sync-ed finished exit (stage+1) 20

21 design: implementa<on effectiveness experiment implementation demo application implemented in Scala modeling implementa<on verification experiments control panel implemented in Python, shell script simulator implementation real cluster application are written in Scala simulator implemented in C++ later experiments are conducted in Python visualization is made in Python pseudo-code is given below 21

22 design: implementa<on initialization while not all partitions gone through processing: if time to interrupt: chosen victim nodes to remove from cluster hosted partitions roll back to checkpoint modeling implementa<on verification experiments done if iteration-based: select only lagging partitions to resume else: select all partitions to resume designate corresponding partitions to active nodes 1) overhead of recovery applied if any 2) bring back nodes if appropriate 3) process partitions if enabled and time-triggered: checkpoint write 22

23 design: verifica<on modeling implementation verifica<on experiments on single-node batch-job case that gives formula T w ( ) =T s + Ts 1 +[ + ] ( + ) n( ) +Rn( ). T s denotes solve time, n( ) interruption number, time to write a checkpoint file, ( + ) fraction of interruption averagely, and R time to restart. optimum proximation p opt = 2 M, wherem mean time to interrupt. on expectation that formula is extendable to parallel cases MTTR (mean time to revoke) is given by Poisson Distribution the optimum of checkpoint write interval is still effective 23

24 design: verifica<on exp. of running time mean running time median running time Single-node Batch-Job MTTR = 40, lineage = 1000 Batch job on single node w/ CKPT write: running time vs CKPT w i (MTTR = 40, lineage = 1000) Cluster with Interdependent Stages MTTR = 40, lineage = 1000 exp. of running time mean running time median running time Cluster losing one node w/ CKPT write: running time vs CKPT w i (MTTR = 40, lineage = 1000) Median running time (unit) Median running time (unit) Checkpoint write interval (unit) Checkpoint write interval (unit) 24

25 design: experiments working datasets recent price history records of 90 days from official modeling implementation verification experiments '16 March to '16 June covering wide range of instance types m3.large is mainly examined general-purpose, relatively 2-minute period time is considered we bid at multiple levels until we see job failure triggered by timeout ( bound) or we see qualitative change 25

26 evaluation 26

27 evalua<on: quan<ta<ve analysis 1100 Run Time Comparison 30- and 60-second-MTTR 1200 Run Time Comparison 10- and 20-partition 1400 Run Time of Comparison 10- and 20-iteration ± 2 30s 10 ± 2 60s 10 ± 4 30s 10 ± 4 60s 10 ± 6 30s 10 ± 6 60s ± 2 10p 10 ± 2 20p 10 ± 4 10p 10 ± 4 20p 10 ± 6 10p 10 ± 6 20p ± 2 10i 10 ± 2 20i 10 ± 4 10i 10 ± 4 20i 27

28 evalua<on 500 m3.large l50 m09 b0.19! Run time and total cost drop after checkpoint write interval increased to a certain time. We see drops to local minimums, among them there is global optimum for checkpoint write. Sensitive to rising bids: resulting in qualitative changes emerges when bidding rose to a certain level. In this qualitatively changed case, the cluster gets over revocations easily. And the only dominating overseas is from checkpoint write. Even more difficulty to determine optimum checkpoint write interval. Run Time Run Time run time w/ bid@0.19x run time w/ bid@0.18x run time w/ bid@0.17x Checkpoint Write Interval (40 seconds/unit) m3.large l50 m01 b0.20! 0.16 run time w/ bid@0.20x run time w/ bid@0.19x run time w/ bid@0.18x run time w/ bid@0.17x run time w/ bid@0.16x Checkpoint Write Interval (40 seconds/unit) Total Cost Total Cost run time w/ bid@0.19x run time w/ bid@0.18x run time w/ bid@0.17x Checkpoint Write Interval (40 seconds/unit) run time w/ bid@0.20x run time w/ bid@0.19x run time w/ bid@0.18x run time w/ bid@0.17x run time w/ bid@0.16x Checkpoint Write Interval (40 seconds/unit) 28

29 conclusion conclusion optimum checkpoint write interval tend to be small, i.e. overhead from checkpoint write is much smaller than that from rolling back with RDD recovery sometimes raising bid a bit may lead to qualitative change and lower bid does not always means lower cost 29

30 Thank you. 30