Home » Blog » Deploying a Three-Node Kafka KRaft Cluster for Data Streaming

Information

Deploying a Three-Node Kafka KRaft Cluster for Data Streaming

Benhcmark Administrator

21/06/2025

0 Views 0

SaveSavedRemoved 0

Deploying a Robust Three-Node Kafka KRaft Cluster

Building reliable data streaming platforms requires careful consideration of the underlying infrastructure. Apache Kafka has long been the de facto standard, and its evolution towards using KRaft (Kafka Raft) for consensus marks a significant step, eliminating the dependency on external systems like Zookeeper. This guide focuses on deploying a highly available three-node Kafka cluster leveraging the power of KRaft.

The transition to KRaft simplifies the Kafka architecture by integrating the metadata management directly into Kafka itself. This results in a single process per server handling both broker and controller roles (or just one role), leading to improved scalability, faster failovers, and reduced operational overhead compared to the traditional Zookeeper setup.

For a robust, fault-tolerant deployment, a minimum of three nodes is recommended for the KRaft quorum. This ensures that even if one node fails, the cluster can still elect a leader and maintain metadata consistency. Each node in our three-node setup will participate in the controller quorum, managing the cluster state, and also function as a broker, handling data topics and partitions.

Before starting, ensure you have Java installed and the Kafka binaries downloaded on each of your three servers.

The deployment process involves several key steps:

1. Generate the Cluster ID: The KRaft cluster requires a unique identifier. This is generated once using the kafka-storage.sh tool on one of the nodes. The command is typically bin/kafka-storage.sh random-uuid. Make sure to record this UUID; it will be the cluster ID for all nodes.

2. Configure Each Node: Each of the three servers needs a server.properties configuration file tailored for KRaft. Key properties to set include:

   • process.roles: Set this to broker,controller for all three nodes, indicating they serve both roles.
   • node.id: Assign a unique integer ID to each node (e.g., 1, 2, 3).
   • controller.quorum.voters: This property lists the node IDs and their host/port for all nodes participating in the controller quorum. For a three-node cluster with IDs 1, 2, and 3, this might look like 1@host1:9093,2@host2:9093,3@host3:9093. Use consistent communication ports (e.g., 9093 for controller communication).
   • listeners and advertised.listeners: Configure these for clients and inter-broker communication, typically using port 9092. Ensure advertised.listeners uses the correct hostname or IP address reachable by clients and other brokers.
   • log.dirs: Specify the directory (or directories) where Kafka will store topic partition data and KRaft metadata.
   • cluster.id: Set this to the UUID generated in step 1 on all nodes.

3. Format Storage Directories: On each node, initialize the storage directories specified in log.dirs using the kafka-storage.sh tool with the format command. You will need the cluster ID for this step: bin/kafka-storage.sh format -c server.properties --cluster-id <your_cluster_id>. Perform this on all three nodes.

4. Start the Kafka Servers: Start the Kafka process on each of the three nodes. This is done using the standard kafka-server-start.sh script, pointing to your configured server.properties file: bin/kafka-server-start.sh config/server.properties. It’s good practice to start them sequentially, but with KRaft, they should be able to discover each other and form the quorum.

5. Verify Cluster Health: Once all nodes are started, you can verify the cluster status. You can use the kafka-broker-api-versions.sh tool pointed at one of your broker addresses to see if it’s responsive, or observe the logs for messages indicating that the controller quorum has formed and the broker is ready. Look for logs confirming the election of a controller leader.

With these steps completed, you will have a functional three-node Kafka KRaft cluster ready for handling your data streaming workloads. This configuration provides the necessary fault tolerance and high availability for production environments, leveraging the streamlined architecture offered by KRaft.

Source: https://kifarunix.com/setup-a-three-node-kafka-kraft-cluster/