spark submit 源码分析

rpcEnv

CoarseGrainedSchedulerBackend

spark-submit

3

name=driverClientmasterEndpoints：从传递参数中获取master的url

如果coresPerExecutor为Nil，则为每个Worker分配一个Executor

DAGScheduler

资源划分、启动Worker

CoarseGrainedScheduler

send

childMainClass: ClientApp

manager.start()

5C9G

usableWorkers

memoryPerExecutor

6C12G

driverDescriptionmainClass：org.apache.spark.deploy.worker.DriverWrapper

待分配核数：取还需分配核数和可用核数的最小值

ClientApp

每个Executor分配的内存

DriverRunner

启动我们写的方法，通过driverArgs.mainClass 调用我们写的new SparkContext(conf)方法

1

workers

SchedulerBackend

assignedCores

driver.start()

从waitingDrivers中遍历driver向Worker发送LaunchDriver类型的消息

向所有Master发送RegisterApplication类型的消息

spreadOutApps

org.apache.spark.deploy.worker.DriverWrapper

schedule()

SparkEnv

遍历masterEndpoints，向master ask消息，消息类型为：RequestSubmitDriver注册Driver

在LaunchExecutor之后向Application发送ExecutorAdded类型的消息

参数列表：--master MASTER_URL--deploy-mode DEPLOY_MODE--class CLASS_NAME--name NAME--jars JARS--driver-memory MEM --executor-memory MEM--executor-core    （standalone、yarn）--total-executor-cores  （standalone、mesos）--num-executors  （yarn）......

onStart

StandaloneSchedulerBackend

Worker: Driver

coresToAssign

ExecutorRunner

Application

start

runMain

_taskScheduler

assignedExecutors

AppClient

1、registerRpcEndpoint2、EndpointData3、Inboxinbox.synchronized {    messages.add(OnStart)  } 4、receivers.take(OnStart)5、endpoint.onStart()

DriverInfo

coresPerExecutor

9C8G

默认为true，一般不改true水平划分，遍历每台worker划分false垂直划分，先将一台worker分配满，再去下一台

TaskSchedulerImpl

org.apache.spark.executor.CoarseGrainedExecutorBackend

tryRegisterAllMasters

参数需求：--executor-memory 4g--executor-core 3--total-executor-cores 9

参数中不同的运行模式，创建不同的实现local：TaskSchedulerImpl、LocalSchedulerBackendspark：TaskSchedulerImpl、StandaloneSchedulerBackend

ask

Worker: Executor

Master

Nil

driverArgs.mainClass

minCoresPerExecutor

继承

每个executor上的core个数，从参数--executor-core获取

oneExecutorPerWorker

可用的worker数组：根据以下条件过滤1、worker.memoryFree >= app.desc.memoryPerExecutorMB2、worker.coresFree >= coresPerExecutor返回一个可用cores从大到小数组

SparkApplication

StandaloneAppClient

从waitingApps中遍历app向Worker发送LaunchExecutor类型的消息

prepareSubmitEnvironment: Tuple4

Client

基于spark跑在standalone cluster模式下的分析

SparkContext

client和cluster模式的区别在于：启动Driver进程是在Client端还是在集群的某一台机器上

默认最小值为1

分配的cores数组