How to Set Up a Multi-Node Elasticsearch Cluster: A Step-by-Step Guide
Moving from a single-node Elasticsearch setup to a multi-node cluster is a critical step for any application that requires high availability, scalability, and fault tolerance. A single node represents a single point of failure, but a well-configured cluster ensures your data remains accessible and your search capabilities remain online, even if one node goes down.
This guide provides a clear, step-by-step process for configuring a resilient, multi-node Elasticsearch cluster, perfect for production environments.
Why a Multi-Node Cluster is Essential
Before diving into the configuration, it’s important to understand the benefits of a distributed setup:
- High Availability: If one node fails, the other nodes in the cluster can take over, ensuring minimal to no downtime.
- Scalability: As your data grows, you can easily add more nodes to the cluster to handle increased storage and query loads.
- Fault Tolerance: Data is replicated across multiple nodes in the form of replica shards. If a node holding a primary shard fails, a replica on another node is automatically promoted to primary.
Prerequisites
To begin, ensure you have the following on each server that will act as a node in your cluster:
- A compatible version of Java (JDK) installed.
- The same version of Elasticsearch downloaded and extracted.
Consistency across all nodes is key to a stable cluster. Using different Elasticsearch versions within the same cluster is not supported and will lead to errors.
Configuring Your Elasticsearch Nodes
The core of setting up a cluster involves editing the elasticsearch.yml configuration file, located in the config directory of your Elasticsearch installation. You will need to make these changes on every node.
1. Define a Unique Cluster Name
Every node must belong to the same cluster. This is defined by the cluster.name setting. Choose a descriptive name for your cluster and ensure it is identical on every node.
cluster.name: my-production-cluster
It is crucial that this name is identical across all nodes, as this is how nodes identify which cluster to join.
2. Assign a Unique Node Name
While nodes share a cluster name, each individual node must have a unique identifier. This is set using the node.name parameter. A common practice is to use a descriptive name like the server’s hostname.
On your first node:
node.name: es-node-01
On your second node:
node.name: es-node-02
And so on for each additional node.
3. Configure Network Host Settings
By default, Elasticsearch only listens for traffic from localhost. To allow nodes to communicate with each other over a network, you must change the network.host setting.
Setting it to 0.0.0.0 binds Elasticsearch to all available network interfaces. For better security, it is recommended to bind it to a specific private IP address that the other nodes can reach.
network.host: 0.0.0.0
Or, for a more secure setup:
network.host: 192.168.1.10
4. Set Up Cluster Discovery
Discovery is the process by which nodes find each other to form a cluster. The discovery.seed_hosts setting provides an initial list of other nodes in the cluster that a node can contact to join.
You should provide a list of the IP addresses (or hostnames) of a few master-eligible nodes in your cluster.
discovery.seed_hosts: ["192.168.1.10", "192.168.1.11", "192.168.1.12"]
You do not need to list every node in the cluster, just enough for a new node to find and join the existing cluster.
Bootstrapping the Cluster
For the very first time you start the cluster, you must specify which nodes are eligible to become the master node. This is a critical one-time step that prevents a “split-brain” scenario. This is done with the cluster.initial_master_nodes setting.
List the node.name of the nodes that you want to be master-eligible.
cluster.initial_master_nodes: ["es-node-01", "es-node-02", "es-node-03"]
This setting is only needed when forming the cluster for the first time. Once the cluster has successfully formed, you should remove this setting from your elasticsearch.yml files on all nodes to prevent accidental re-bootstrapping.
Launching and Verifying Your Cluster
After configuring the elasticsearch.yml file on each node, you can start them one by one.
- Navigate to your Elasticsearch directory on each server.
- Run the command:
./bin/elasticsearch
Once all nodes are running, you can verify that they have formed a cluster by sending a request to the _cat/nodes API endpoint from any of the nodes.
curl -X GET "localhost:9200/_cat/nodes?v"
You should see an output listing all the nodes you configured, indicating they have successfully joined the cluster.
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.1.10 25 95 5 0.00 0.01 0.05 mdi * es-node-01
192.168.1.11 30 92 3 0.01 0.02 0.05 mdi - es-node-02
192.168.1.12 28 93 4 0.00 0.01 0.05 mdi - es-node-03
The asterisk (*) next to es-node-01 indicates that it is the currently elected master node.
Essential Security Best Practices
Setting up a cluster that is open to the network requires careful security considerations.
- Configure a Firewall: Use a firewall on each server to restrict access to Elasticsearch ports (typically
9200for HTTP and9300for transport). Only allow access from trusted IP addresses, such as your application servers and other nodes in the cluster. - Enable Elasticsearch Security: For production environments, it is highly recommended to enable the built-in security features. This allows you to set up authentication, role-based access control (RBAC), and enable TLS/SSL encryption for all communication.
- Run as a Non-Root User: Never run Elasticsearch as the
rootuser. The official packages create a dedicatedelasticsearchuser for this purpose.
By following these steps, you have successfully transformed your single-instance setup into a robust, scalable, and resilient multi-node Elasticsearch cluster ready for production workloads.
Source: https://kifarunix.com/setup-multi-node-elasticsearch-cluster/
