Slipstream State架构
2016-08-18 16:52:58 0 举报
Slipstream State架构是一种用于实现高吞吐量、低延迟数据处理的系统架构。它通过将数据流划分为多个小批次,并在每个批次之间共享状态信息,从而实现了高效的数据处理。这种架构通常用于处理大量的实时数据,如传感器数据、网络流量等。 在Slipstream State架构中,每个批次都包含一个状态向量,该向量包含了前一个批次的状态信息。当新的批次到达时,系统会更新状态向量,并将其传递给下一个批次。这样,每个批次都可以在前一个批次的基础上进行计算,从而大大提高了处理速度。
作者其他创作
大纲/内容
AbstractStateBackend
ByteStreamStateHandle
ExactlyOnceState[Serializable]
ValueState
+ value(): T+ update(value: T)+ values(): Array[T]
StateHandle [T]
+ getState(): T
FsStateSnapshot
FsCheckpointStateOutputStream
+ closeAndGetHandle(): StreamStateHandle+ closeAndGetPath(): Path
序列化为字节数组
CheckpointStateOutputStream
+ closeAndGetHandle(): StreamStateHandle
DataInputViewHandle[DataInputView]
- stream: StreamStateHandle
KvState
Executor
StreamTaskStateHandle [StreamTaskState]
+ getState(): StreamTaskState
FileStreamStateHandle
FSKVState
- backend: AbstractStateBackend
StreamStateHandle [InputStream]
+ toSerializableHandle[R <: Serializable: ClassTag](): StateHandle[R]
Server
StateObject
+ discardState() + getStateSize(): Long
StreamTaskState
+ discardState()
CheckpointStateOutputView
- os: CheckpointStateOutputStream
+ close()+ closeAndGetHandle(): StateHandle[DataInputView]
MorphlingTaskState
AbstractFileStateHandle
- fs: FileSystem- path : Path
+ getFileSize(): Long+ getFileSystem(): FileSystem+ getFilePath(): Path
AbstractHeapState
State
+ clear()
MorphlingSubTaskState
- state: Array[Byte]- stateSize: Long- duration: Long
+ discard()
FsStateBackend
0 条评论
下一页