大数据-kafka

写相同分区数据

5000-6000

kafka3

000000000.index

zookeeper 持有kafka 的分区的leader信息，本示例 replication 为2，假设 partition1 的leader 是 kafka3，partition2 的 leader 是 kafka2

zookeeper

kafka 消费者索引消息

发送数据

topic1partition0

topic1partition1

producer

topic1partition1leader

8000-9000

kafka2brokerID=1

返回brokerID=2

topic1partition0follower

000030235.index

offset=853

offset=567

kafka

zookeeper集群

topic2partition0leader

topic=topic1partition=2replication=2

topic2partition1leader

kafka1

根据分区获取brokerID

kafka2

kafka集群

topic1partition1follower

kafka 分区及复制

zookeeper包含kafka如下信息：/brokers/brokers/ids/brokers/topics/cluster/admin/isr_change_notification/controller_epoch/consumers/latest_producer_id_block/ config

kafka1brokerID=0

000000000.log

topic=topic2partition=3replication=1

567

853

000030235.log

controller

topic1partition0leader

topic1partition0partition1

kafka3brokerID=2

批量拉取数据

写数据

发送ACK表示成功

ACK表示同步成功

信息topic=topic1partition=2replication=2

consumer