Setting up a scalable and robust data streaming platform is crucial for modern applications. Deploying a three-node Kafka cluster using the KRaft (Kafka Raft metadata mode) consensus protocol represents a significant step forward, eliminating the traditional reliance on Zookeeper. This new architecture streamlines deployment and simplifies management.
At its core, a KRaft cluster involves nodes performing different roles: controllers, which manage cluster metadata, and brokers, which handle data serving. For simplicity and efficiency in smaller clusters like a three-node setup, nodes can often perform a combined controller and broker role. This is a common and recommended configuration.
The deployment process begins with preparing the necessary configuration for each node. Each node requires a unique identifier (node.id) and specific listener configurations to communicate with other nodes and clients. A critical step is specifying the directory where KRaft metadata will be stored (metadata.log.dir).
Before starting the nodes, an initial cluster ID must be generated. This unique identifier ties the nodes together into a single KRaft cluster. One of the nodes (or a separate utility) is used to perform this initialization step, writing the cluster ID into the metadata directory of each node.
Once the configuration is set, the nodes can be started sequentially. They will discover each other using the shared cluster ID and listener information, forming the distributed KRaft cluster. The controller quorum will establish itself, and the brokers will become ready to serve topic partitions.
Verifying the successful deployment involves checking logs for confirmation that nodes have joined the cluster and that the controller role is active. You can also use Kafka command-line tools to inspect the cluster state, list brokers, and check controller status.
This three-node KRaft cluster configuration provides a resilient foundation for data streaming workloads. The inherent fault tolerance of the Raft consensus ensures that the cluster remains available even if a node fails, making it an excellent choice for scalable and highly available streaming applications. Mastering this deployment is key to leveraging the full power of modern Kafka.
Source: https://kifarunix.com/setup-a-three-node-kafka-kraft-cluster/
