Kafka Internals
Kafka uses Apache Zookeeper to maintain the list of brokers that are currently members of a cluster. Every broker has a unique identifier. Every time a broker process starts, it registers itself with its ID in Zookeeper by creating an ephemeral node. Different Kafka components subscribe to the /brokers/ids path in Zookeeper where brokers are registered so they get notified when brokers are added or removed.
The Controller
The 1st broker that starts in kafka cluster or we can say the broker which create ephemeral node very first with zookeeper is controller node. this broker is also reponsible for electing leader partition (master replica). At a time only one controller node can present in the cluster.
Replication
Each topic can have multiple partitions and a partition can have multiple replicas. those replicas are stored on different broker other then leader partition(master replica) broker.
Leader Replica: Each partition in a topic has a single replica designated as the leader. All produce and consume requests go through the leader, in order to guarantee consistency. the replicas send the leader Fetch requests to stay up to date.
Follower Replica: All replicas expect leader are followers. they don’t serve request. their only job is to replicate messages from leader.
If a leader replica goes down then one of the follower which is most in-sync with leader become new leader and server requests
If any follower replica did not send fetch request to leader till time which we define in config file replica.lag.time.max.ms. then that replica consider as out-of-sync. and that replica can’t become a leader.
Request Processing
Clients always initiate connections and send requests and the broker processes the requests and responds to them. Broker runs an acceptor thread that creates a connection and hands it over to a network thread(processor thread) for handling. The network threads are responsible for taking requests from client connections, placing them in a request queue, and picking up responses from a response queue and sending them back to clients.
Once requests are placed on the request queue, IO threads are responsible for picking them up and processing them.
Why Apache Kafka is so Fast?
Low latency I/O: There are two possible places which can be used for storing and caching the data: Random Access Memory (RAM) and Disk.
Kafka relies on the filesystem for the storage and caching of messages.
However, we can achieve low latency while delivering messages is to use the RAM. But the downside in this approach is it can be expensive to use the RAM when the data flowing through your system is around 10 to 500 GB per second or even more.
Kafka uses log data structure which is an append-only sequence of records to avoid seek-time on disk. kafka maintains head pointer to consumer messages for consumers and tail pointer for producers to append messages.
Zero Copy Principle: Sending data over a network requires multiple context switches between the Kernel mode and the User mode. Zero Copy Principle reduce this by requesting the kernel to move the data directly to the response socket. this will reduce consumption of memory bandwidth and CPU cycles.
File Management: Kafka does not keep data forever, neither it waits for consumers to consume messages before deleting it. all storage handle process is handled by retention period configuration. finding a message that needs to delete from a large file is time-consuming. so kafka split each partition into segments when segment size reaches its max limit, Kafka close that segment and opens new segment to start writing to it and old segment eligible for expire based on configuration.
Thanks for reading.
