Shuffle

Hadoop\t5Hbase\t4Hive\t\t4Spark\t2Sqoop\t2

reduce2

分区

排序

map task2

map task1

ruduce取得自己的数据

Hbase 1\treduce1Hbase 1\treduce1Hive 1 reduce2Hive 1 reduce2

Hive\t 1\treduce2Hive\t 1\treduce2

Hbase 1\treduce1Hbase 1\treduce1

Hadoop\t1\treduce1Hadoop\t1\treduce1Hadoop\t1\treduce1Hbase\t1\treduce1Hbase\t1\treduce1

环形缓冲区

Hadoop 1 reduce1Hadoop 1 reduce1Hadoop\t1 reduce1Spark 1 reduce2Sqoop 1 reduce2

Hive\t\t1\treduce2Hive\t\t1\treduce2Spark\t1\treduce2Sqoop\t1\treduce2

map数据

合并

Hadoop 1 reduce1Spark 1 reduce2Sqoop 1 reduce2

Hadoop 1 reduce1Hadoop 1 reduce1

溢写

Hadoop\t1\treduce1Hive\t\t1\treduce2Spark\t1\treduce2Sqoop\t1\treduce2Hadoop\t1\treduce1Hbase\t1\treduce1Hive\t\t1\treduce2Hbase\t1\treduce1Hadoop\t1\treduce1

reduce1

Hadoop 1 reduce1Hadoop 1 reduce1Spark 1 reduce2Sqoop 1 reduce2Hadoop\t1 reduce1

Hbase 1\treduce1Hbase 1\treduce1Hive 1\treduce2Hive 1\treduce2