任务调度

V

LongArray

Partition 0 data

ResultStage 4

partitionId page number offset in page

map

CoGroupedRDD[12]、MapPartitionsRDD[13]、MapPartitionsRDD[14]Dep：OneToOneDependency、OneToOneDependency、OneToOneDependency

父Stage是否执行完毕

Do Nothing

MapPartitionsRDD[7]Dep：OneToOneDependency

ShuffleMapStage 0

Partition 1 data

rdd: MapPartitionsRDD[2]shuffleDep: ShuffleDependency[0]

parallelize

27 bit

13 bit

Partition 1 offset

rdd: MapPartitionsRDD[10]shuffleDep: ShuffleDependency[3]

no

dataFile

page1 page2 page3 page4

partitionId

ParallelCollectionRDD[6]Dep：Nil

Partition 2 data

24 bit

reduceByKey

Partition 2 offset

PartitionId 2 (pn + offset)

join

PartitionId 0 (pn + offset)

ShuffledRDD[3]Dep：ShuffleDependency[0]

输入waitingStages

ShuffleMapStage 1

PartitionId 1 (pn + offset)

ShuffleMapStage 2

父Stage

Partition 0 offset

ParallelCollectionRDD[0]Dep：Nil

page(MemoryBlock)

allocatedPages

rdd: MapPartitionsRDD[4]shuffleDep: ShuffleDependency[1]

tmp dataFile 3

MapPartitionsRDD[8]Dep：OneToOneDependency

MapPartitionsRDD[4]Dep：OneToOneDependency

ShuffleInMemorySorter

yes

4 bytes

Record length K + V

submitStage流程产生的waitingStages集合

PartitionId 1(pn + offset)

MapPartitionsRDD[10]Dep：OneToOneDependency

tmp dataFile 1

将Stage添加到集合waitingStages

ShuffledRDD[9]Dep：ShuffleDependency[2]

拷贝集合waitingStages到waitingStagesCopy清空waitingStages遍历waitingStagesCopy

rdd: MapPartitionsRDD[8]shuffleDep: ShuffleDependency[2]

ShuffleMapStage 3

currentPage

输入Stage

MapPartitionsRDD[1]Dep：OneToOneDependency

8 bytes

开始

submitStage流程

ShuffleExternalSorter

再次触发submitWaitingStages

ShuffledRDD[5]Dep：ShuffleDependency[1]

tmp dataFile 2

indexFile

Stage状态为running、waiting、failed

MapPartitionsRDD[2]Dep：OneToOneDependency

submitMissingTasks方法提交任务，并将Stage添加到集合runningStages

flatMap

rdd: MapPartitionsRDD[14]func:

ShuffledRDD[11]Dep：ShuffleDependency[3]

是否有父Stage

0L