Setting up a highly available and scalable messaging system is crucial for modern distributed applications. Apache Kafka, particularly with the KRaft (Kafka Raft metadata) mode, offers a streamlined approach to achieving this without external dependencies like ZooKeeper. Building a three-node cluster provides redundancy and distributed metadata management right from the start.
The process begins by ensuring you have the necessary prerequisites, including compatible Java and a Kafka distribution. Network configuration is paramount; ensure your nodes can communicate freely on the designated Kafka ports (typically 9092 for clients and 9093 for inter-broker/controller communication) and that hostname resolution works correctly across the cluster.
The first step in setting up a KRaft cluster involves generating a unique cluster ID. This ID is shared by all nodes and serves as a fundamental identifier for the cluster. You then configure each node, designating roles. In a three-node setup, it’s common for all nodes to act as both controllers (managing cluster metadata) and brokers (handling data). This provides maximum resilience.
Each node requires a specific configuration file (server.properties). Key configurations include:
- The generated cluster ID.
- A unique broker ID for each node.
- Configuring listeners to specify which network interfaces and ports Kafka binds to for client traffic and inter-broker communication. This usually involves both a
PLAINTEXTlistener (orSSL/SASLfor production) and an internal listener for controller communication. - Specifying the metadata directory where KRaft stores its state.
- Listing the initial cluster controllers (all three nodes in this case, each pointing to the others).
Once configuration is complete on all nodes, you format the storage directory on each machine using the Kafka provided tools, which prepares the metadata directory for KRaft.
Finally, you start the Kafka server on each node. It’s often recommended to start the designated controllers first, though in a combined role setup, starting them sequentially works. The nodes will discover each other using the controller list and the shared cluster ID, elect a leader controller, and form the cluster.
After starting, verify the cluster state by checking logs for successful controller election and broker registration. You can then create topics and produce/consume messages to test functionality across the distributed nodes. This methodical approach ensures your three-node KRaft cluster is robust, efficient, and ready to handle your workload with high availability.
Source: https://kifarunix.com/setup-a-three-node-kafka-kraft-cluster/
