Beam Translator

Beam代码：PCollection windowedWords = input.apply(Window.into(FixedWindows.of(Duration.standardMinutes(60))));

FlinkBatchPipelineTranslator

Flink API public static void main(String[] args) throws Exception {\tStreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();\tDataStreamWindowedValue source =\tenv.addSource( sourceFunction ).name(transform_name).returns(TypeInformation); \t\t

translation

ReshuffleTranslatorBatch

CombinePerKeyTranslator

PTransformTranslation.PAR_DO_TRANSFORM_URN

URN

input

BatchTransformTranslator

Beam API

urn1 —— ReadSourceTranslator urn2 —— ParDoStreamingTranslator urn3 —— WindowAssignTranslator urn4 —— CombinePerKeyTranslator urn5 —— GroupByKeyTranslator

CreateStreamingFlinkView.CREATE_STREAMING_FLINK_VIEW_URN

PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN

dataStreams

ParDoStreamingTranslator

WindowAssignTranslatorBatch

currentTransform

ReshuffleTranslatorStreaming

translator

FlattenPCollectionTranslator

GBKIntoKeyedWorkItemsTranslator

WindowAssignTranslator

boundedTranslator

PTransformTranslation.RESHUFFLE_URN

翻译

FlinkStreamingPipelineTranslator.StreamTransformTranslator

Transfrom 类

Beam 的API Pipeline p = Pipeline.create(options); PCollection lines = p.apply(\"ReadLines\

Translator

PipelineOptions defaultOptionsTransformHierarchy transforms

PTransformTranslation.READ_TRANSFORM_URN

CreateViewStreamingTranslator

八种

CombinePerKeyTranslatorBatch

TransformHierarchy

StreamExecutionEnvironment

GroupByKeyTranslatorBatch

GroupByKeyTranslator

Flink API StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();DataStreamWindowedValuesource=env.addSource(sourceFunction).name(transform_name).returns(TypeInformation); env.execute(\"jobName\"); \t\t

FlinkStreamingPipelineTranslator

PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN

ExecutionEnvironment

ReadSourceTranslator

FlinkStreamingTransformTranslators

ParDoTranslatorBatch

运行

SplittableParDo.SPLITTABLE_GBKIKWI_URN

PTransformTranslation.FLATTEN_TRANSFORM_URN

PTransformTranslation.CREATE_VIEW_TRANSFORM_URN

PipeLine

FlinkPipelineExecutionEnvironment

getTranslator(transform)

ParDoTranslationHelper

FlattenPCollectionTranslatorBatch

transforms算子树

fullName

十种

Primitive Node 叶子节点

Node

FlinkRunner

executePipeline方法

1、Pipeline p = Pipeline.create(options);

SPLITTABLE_PROCESS_URN

transfrom 对象

transform: new TransformHierarchy 树

PTransformTranslation.COMBINE_TRANSFORM_URN

translate方法

context

CreatePCollectionViewTranslatorBatch

SplittableProcessElementsStreamingTranslator

注册器接口

this.enclosingNode = null;this.transform = null;this.fullName = \"\";this.inputs =Map();this.outputs = Map();parts = new ArrayList()

ReadSourceTranslatorBatch

boolean isParallel

Transform

output

FlinkStreamingTranslationContext

FlinkBatchTransformTranslators

FlinkRunner的Translator