Flink uid和uidHash
2021-05-25 23:49:09 0 举报AI智能生成
flink 源码
作者其他创作
大纲/内容
uid
<b>设置org.apache.flink.api.dag.Transformation#setUid</b><br>
uidHash
<b>设置org.apache.flink.api.dag.Transformation#setUidHash</b>
<b>设置<br>org.apache.flink.streaming.api.graph.<br>StreamGraphGenerator#transform</b><br>
if (transform.getUid() != null) {<br> streamGraph.setTransformationUID(transform.getId(), transform.getUid());<br> }<br> if (transform.getUserProvidedNodeHash() != null) {<br> streamGraph.setTransformationUserHash(transform.getId(), transform.getUserProvidedNodeHash());<br> }
<b>org.apache.flink.streaming.api.graph.<br>StreamingJobGraphGenerator#createJobGraph()<br></b>setChaining(<b>hashes</b>, <b>legacyHashes</b>, chainedOperatorHashes);<br>
defaultStreamGraphHasher<br><b>生成hashes</b>
StreamGraphHasherV2
generateNodeHash<br>判断用户是否设置uid<br><b>generateDeterministicHash<br> or<br>generateUserSpecifiedHash</b><br>
* Generates a deterministic hash from node-local properties and input and output edges.<br>有可能重复
* Generates a hash from a user-specified ID.<br>
legacyStreamGraphHashers<br><b>生成legacyHashes</b>
StreamGraphUserHashHasher<br>使用用户提供的uid hash
<b>traverseStreamGraphAndGenerateHashes</b>
org.apache.flink.streaming.api.graph.<br>StreamingJobGraphGenerator#createJobVertex<br>
new JobVertexID(hash)
legacyJobVertexIds
JobVertex
JobVertexID
generateDeterministicHash ps 有可能重复，不可靠，所以建议设置uid
idAlternatives
org.apache.flink.runtime.checkpoint.CheckpointCoordinator#restoreSavepoint
org.apache.flink.runtime.checkpoint.Checkpoints#loadAndValidateCheckpoint
org.apache.flink.runtime.checkpoint.savepoint.SavepointV2#convertToOperatorStateSavepointV2
如果使用job vertex id 找不到就考虑<br>使用 legacy job vertex<br>
org.apache.flink.runtime.executiongraph.ExecutionJobVertex#includeLegacyJobVertexIDs
0 条评论
下一页
