Yarn 框架(spark on yarn cluster模式为例)

min-allocation-mb

⑩

reducetask[cpu+ram +jar](Excuter)

......

maptask[cpu+ram + jar](Executor)

on-heap

⑤

off-heap

WordCount.jar

⑪

root

⑨

ResourceManager

④

task3

②

Container

⑫

spark Executor内存模型(Capacity Scheduler调度器的场合)

NodeManager

(max-allocation-mb)

spark on  YARN Cluster模式

ExecutionMemory

UnifiedMemory

Queue

AppMaster[cpu+ram](driver)

Executor

task4

YARNRUNNER

①

YARN客户端

规整化因子

UserMemory

容器和重要配置参数(cluster模式driver和appmter同一容器)利用cgroup做cpu的隔离调度容器最小调度单元参数设值yarn.scheduler.minimum(max)-allocation-mbyarn.scheduler.minimum(max)-allocation-vcores

task5

maptask[cpu+ram +jar](Executor)

⑦

⑧

⑥

task1

③

StorageMemory

HDFS

Reserved Memory

task2

que1