datanode

2015-12-24 11:42:56   4  举报





Datanode是Hadoop分布式文件系统HDFS中的一个关键组件，负责存储和管理数据块。在HDFS中，当客户端上传一个文件时，该文件会被分割成多个数据块，这些数据块会被分配给不同的datanode进行存储。Datanode会将这些数据块保存在其本地的文件系统中，并定期向Namenode报告其存储的数据块信息。此外，当客户端需要读取某个文件时，datanode会负责提供其所存储的数据块，从而帮助客户端完成数据的读写操作。总之，datanode在HDFS中起到了数据存储和备份的关键作用，确保了数据的可靠性和高可用性。

作者其他创作

大纲/内容

Runnable

HttpServer

- webServer- listener

+ start+ stop

DataBlockScanner

- dataset- blockPoolScannerMap

+ start+ run+ shutdown+ isInitialized+ addBlock+ deleteBlock+ removeBlockPool

ServicePlugin

DirectoryScanner

- diffs- stats

- reconcile- scan

BPOfferService

+ updateActorStatesFromHeartbeat+ processCommandFromActor- processCommandFromActive- processCommandFromStandby

DataTransferProtocol

+ readBlock+ writeBlock+ transferBlock+ replaceBlock+ copyBlock+ blockChecksum

BlockPoolSliceScanner

- blockPoolId- blockInfoSet- blockMap- processedBlocks

+ shutdown+ scanBlockPoolSlice+ verifyBlock+ addBlock+ deleteBlock- scan

DataNodeDiskChecker

+ checkDir

Receiver

- dataInputStream

DataNode

- clusterId- fsDatasetSpiData- datanodeID- dataDirs

+ startDataNode- initDataXceiver- startInfoServer- initIpcServer+ runDatanodeDaemon+ shutdown- shutdownPeriodicScanners- initPeriodicScanners

TcpPeerServer

BlockBalanceThrottler

- numThreads- estimateBlockSize

+ run+ acquire+ kill+ addPeer+ closePeer

InterDatanodeProtocol

+ initReplicaRecovery+ updateReplicaUnderRecovery

Storage

- storageType- List storageDirs

DataTransferThrottler

Sender

- dataOutputStream

BlockPoolManager

- bpByNameserviceId- bpByBlockPoolId- offerServices

- createBPOS+ refreshNamenodes+ startAll+ joinAll+ shutDownAll

PeerServer

+ setReceiveBufferSize+ accept+ getListeningString+ close

DomainPeerServer

ClientDatanodeProtocol

+ getReplicaVisibleLength+ refreshNamenodes+ deleteBlockPool+ getBlockLocalPathInfo+ getHdfsBlocksMetadata

DataXceiver

- peer- remoteAddress- socketIn- socketOut

DataStorage

- storageID- bpStorageMap

- initDataXceiver+ createStorageID+ doUpgrade+ doRollback+ doFinalize+ finalizeUpgrade+ recoverTransitionRead

BPServiceActor

- nnAddr- pendingIncrementalBR

+ connectToNNAndHandshake+ retrieveNamespaceInfo+ reportBadBlocks+ reportReceivedDeletedBlocks+ notifyNamenodeBlockImmediately+ notifyNamenodeDeletedBlock+ blockReport+ sendHeartBeat+ register- offerService- processCommand+ start+ run+ stop

BlockPoolSliceStorage

- blockpoolID

+ format+ recoverTransitionRead

StorageInfo

DataXceiverServer

- peers

+ run+ kill

ipcServer