Mastering Elasticsearch 8: Your Guide to Setting Up a Multinode Cluster
Moving beyond a single-node Elasticsearch setup is a critical step for any application demanding high availability and scalability. A single node represents a single point of failure, but a multinode cluster distributes data and workload, ensuring your system remains resilient and performant. This guide provides a comprehensive walkthrough for building a robust, secure, and production-ready Elasticsearch 8.x cluster.
While a single node is great for development and testing, a multinode cluster is essential for production environments. It provides the foundation for fault tolerance, load balancing, and horizontal scaling, allowing your search and analytics capabilities to grow with your data.
The Core Benefits of a Multinode Architecture
Deploying a multinode Elasticsearch cluster unlocks several key advantages that are impossible to achieve with a single instance:
- High Availability: If one node fails, the cluster continues to operate. Data is replicated across multiple nodes, so a hardware or software failure on one machine won’t bring down your entire service.
- Scalability: As your data volume or query load increases, you can simply add more nodes to the cluster. This horizontal scaling allows you to expand capacity seamlessly without downtime.
- Improved Performance: Workloads like data indexing and search queries can be distributed across all nodes in the cluster, leading to faster response times and higher throughput.
Understanding Elasticsearch Node Roles
In a cluster, not all nodes are created equal. Assigning specific roles to different nodes is a best practice for optimizing performance and stability, especially in larger deployments.
Here are the primary node roles you need to know:
- Master-eligible Node (
master): This node is responsible for managing the cluster’s state. It tracks which nodes are part of the cluster, manages the creation of new indices, and orchestrates the allocation of shards. A stable cluster requires at least three master-eligible nodes to avoid a “split-brain” scenario. - Data Node (
data): These nodes are the workhorses of the cluster. They store the data and handle all data-related operations, such as CRUD (Create, Read, Update, Delete), searching, and aggregations. - Ingest Node (
ingest): This node is used to pre-process documents before they are indexed. It can execute pre-defined pipelines to transform and enrich data as it enters the cluster. - Coordinating Only Node: While not a formal role (
node.rolesis set to an empty list[]), this is a common pattern where a node acts as a smart load balancer. It receives client requests, forwards them to the appropriate data nodes, and consolidates the results before sending them back to the client.
For a basic, small-scale cluster, a single node can perform all of these roles. However, for a production environment, it is highly recommended to have dedicated nodes for each role.
Step-by-Step Guide to Building Your Cluster
This guide assumes you have at least three servers (physical or virtual) with a compatible version of Java and Elasticsearch 8.x downloaded.
Step 1: Secure Your Cluster with TLS Certificates
Elasticsearch 8 enables security by default, which is a significant improvement over previous versions. You can no longer run a cluster in production without configuring security. The first step is generating a Certificate Authority (CA) for your cluster.
On one of your designated nodes, run the following command from your Elasticsearch home directory:
./bin/elasticsearch-certutil ca
This will generate a elastic-stack-ca.p12 file. Securely copy this file to the configuration directory of every node that will join the cluster.
Next, on each node, generate a certificate and private key signed by your new CA:
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
This will create a elastic-certificates.p12 file in your configuration directory.
Step 2: Configure the First Node (Bootstrap Node)
Now, you need to configure the elasticsearch.yml file, typically located in the config/ directory. This first node will bootstrap the cluster.
Your configuration should look similar to this:
# Give your cluster a unique name
cluster.name: my-secure-cluster
# Give this node a unique name
node.name: node-1
# Assign roles to this node
node.roles: [ "master", "data" ]
# Network settings: bind to a non-loopback address
network.host: 0.0.0.0
# Define the initial master-eligible nodes for the first-time bootstrap
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
# Security configuration pointing to your generated certificates
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
Key Configuration Points:
cluster.name: Must be identical on all nodes.node.name: Must be unique for each node.network.host: Using0.0.0.0allows Elasticsearch to bind to all available network interfaces.cluster.initial_master_nodes: This setting is crucial for the very first time the cluster is formed. It should list thenode.nameof all master-eligible nodes.
Step 3: Start the First Node and Generate an Enrollment Token
Start Elasticsearch on your first node:
./bin/elasticsearch
Once it’s running, you need to set passwords for the built-in users. This is an interactive process:
./bin/elasticsearch-setup-passwords auto
Save these passwords securely. Now, create an enrollment token that other nodes can use to securely join the cluster. This token is valid for a limited time.
./bin/elasticsearch-create-enrollment-token -s node
Copy the generated token. You will need it for the other nodes.
Step 4: Configure and Add Remaining Nodes
On your second and third nodes, update the elasticsearch.yml file with their respective node.name (e.g., node-2, node-3). The rest of the configuration can be the same as the first node.
Instead of starting Elasticsearch directly, start them using the enrollment token you just generated. This command will automatically configure the node to join the existing cluster and handle the security setup.
./bin/elasticsearch --enrollment-token <your_enrollment_token>
After the node successfully joins, stop it with Ctrl+C and then start it normally (./bin/elasticsearch). Repeat this process for all other nodes.
Step 5: Verify Your Cluster Health
Once all nodes are running, you can verify that they have successfully formed a cluster. From any machine that can reach your cluster, use the elastic user and password to run a cURL command against the _cat/nodes API:
curl -X GET "https://<your_node_ip>:9200/_cat/nodes?v" -u elastic:<your_password> --insecure
You should see a list of all your configured nodes, confirming that your multinode cluster is online and healthy.
Essential Post-Setup Security Tips
- Configure Your Firewall: Only allow traffic on ports
9200(HTTP) and9300(Transport) from trusted IP addresses. Do not expose your Elasticsearch cluster directly to the public internet. - Use Role-Based Access Control (RBAC): Create unique users and roles with the minimum necessary permissions for your applications. Avoid using the superuser
elasticaccount for daily operations. - Enable Auditing: Configure audit logging to track security-related events, such as failed login attempts and changes to user permissions.
- Regularly Backup Your Data: Use the Elasticsearch Snapshot API to create regular backups of your cluster’s data and state. Store these snapshots in a secure, remote repository like S3, GCS, or Azure Blob Storage.
Source: https://kifarunix.com/setup-multinode-elasticsearch-8-x-cluster/
