Setting up a robust Kafka environment is fundamental for modern data streaming architectures. While a single node is useful for development, production deployments demand a multi-node cluster to ensure high availability and fault tolerance. The shift to KRaft (Kafka Raft Metadata mode) simplifies this significantly by removing the dependency on ZooKeeper. Building a minimal multi-node setup, such as a three-node KRaft cluster, is a crucial step for production readiness.
In a KRaft cluster, nodes can function as controllers, responsible for managing the cluster metadata, or as brokers, handling client requests for producing and consuming data. Often, nodes act as both. For fault tolerance, a quorum of controller nodes is essential to maintain consistent metadata. A three-node setup is the smallest configuration that can tolerate the failure of one controller node while still maintaining a quorum (2 out of 3).
The process involves installing Kafka on each of the three servers. Each node requires a unique node ID within the cluster. A single, consistent quorum ID must be generated and used across all nodes for the KRaft quorum. Configuration files on each node need to be updated to reflect its specific node ID, the common quorum ID, network listeners, and the list of controller nodes in the quorum.
After configuration, the storage directories on each node must be formatted using the Kafka utilities, referencing the unique node ID and the shared quorum ID. Once formatted, the Kafka server process can be started on each node. It’s critical to ensure that firewall rules allow communication between the cluster nodes on the necessary ports for broker and controller communication.
Successfully starting all nodes brings the three-node KRaft cluster online, ready to handle data streaming workloads with inherent resilience against single node failures. This foundational setup provides a scalable and reliable base for deploying Kafka applications in production environments.
Source: https://kifarunix.com/setup-a-three-node-kafka-kraft-cluster/
