A Step-by-Step Guide to Installing and Configuring Logstash 7 on Ubuntu & Debian
In the world of data management and observability, centralized logging is no longer a luxury—it’s a necessity. Logstash is a powerful, open-source data processing pipeline that allows you to collect data from various sources, transform it, and send it to your preferred destination. As a core component of the Elastic Stack (ELK), it’s the engine that powers real-time analytics and log monitoring.
This guide provides a comprehensive walkthrough for installing and configuring Logstash 7 on Ubuntu 18.04 and Debian 9 systems.
Prerequisites
Before we begin, ensure you have the following:
- A server running Ubuntu 18.04 or Debian 9.
- Access to a user account with
sudoprivileges. - A basic understanding of the Linux command line.
Step 1: Install Java
Logstash is built on Java and requires the Java Virtual Machine (JVM) to run. We will install OpenJDK, a widely-used, open-source implementation of the Java Platform.
First, update your package index to ensure you have access to the latest versions.
sudo apt update
Next, install the OpenJDK 11 package. While Logstash can run on version 8, version 11 is a stable, long-term support (LTS) release recommended for modern applications.
sudo apt install openjdk-11-jdk
Once the installation is complete, you can verify that Java is correctly installed by checking its version.
java -version
You should see output confirming the OpenJDK version 11 is active.
Step 2: Add the Elastic Repository
To ensure we install the official version of Logstash and can easily update it in the future, we will use the official Elastic package repository. This process involves two key steps.
First, import the Elastic public GPG key. This key is used by your system’s package manager to verify that the packages you are downloading are authentic and have not been tampered with.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Next, add the Elastic repository to your system’s sources list. This tells the apt package manager where to find the Logstash package.
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
Security Tip: Always install software from official, trusted repositories. Adding the GPG key is a critical step that ensures package integrity.
Step 3: Install Logstash
With the repository in place, you can now install Logstash. First, update your package index one more time to include the packages from the newly added Elastic repository.
sudo apt update
Now, execute the installation command for Logstash.
sudo apt install logstash
Logstash is now installed on your system. However, it will not start automatically until you provide it with a configuration file.
Step 4: Create Your First Logstash Pipeline
The real power of Logstash lies in its configuration files, which define the data pipeline. A pipeline has three main stages: input, filter, and output.
- Input: Where the data comes from (e.g., a log file, a network port like Syslog, or a message queue).
- Filter: Where data is processed, parsed, and enriched (e.g., parsing unstructured log lines into fields, looking up geo-IP data, or removing sensitive information).
- Output: Where the processed data is sent (e.g., Elasticsearch, a database, or even just the console for testing).
Logstash configuration files are stored in the /etc/logstash/conf.d/ directory. Let’s create a basic configuration to test our setup. This pipeline will take input from the command line (stdin) and output it to the console (stdout).
Create a new configuration file:
sudo nano /etc/logstash/conf.d/01-test-pipeline.conf
Add the following content to the file:
input {
stdin { }
}
output {
stdout {
codec => rubydebug
}
}
This is the simplest possible pipeline. The codec => rubydebug option provides detailed, structured output, which is excellent for debugging. Save and close the file.
Step 5: Test Your Logstash Configuration
Before starting Logstash as a service, it’s a best practice to test your configuration file for syntax errors. A simple typo can prevent the service from starting, and this test will save you significant troubleshooting time.
Run the following command to check the syntax of your configuration:
sudo -u logstash /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/01-test-pipeline.conf
If everything is correct, you will see the message: Configuration OK. If there are errors, the output will provide details on the file and line number where the issue occurred.
Step 6: Start and Enable the Logstash Service
With a valid configuration in place, you are now ready to run Logstash. Use systemctl to start the service.
sudo systemctl start logstash
To ensure Logstash starts automatically whenever the server reboots, you should enable the service.
sudo systemctl enable logstash
You can check the status of the service to confirm it’s running correctly.
sudo systemctl status logstash
If the service is active and running, your Logstash installation is a success. You can now build more complex pipelines to suit your needs, such as collecting Syslog data and sending it to an Elasticsearch cluster.
Final Thoughts and Next Steps
You have successfully installed and configured Logstash on your server. This setup provides a robust foundation for building a powerful, centralized logging system. From here, you can explore the vast library of input, filter, and output plugins to ingest and process virtually any type of data.
For production environments, remember to review the JVM heap size settings in /etc/logstash/jvm.options to allocate sufficient memory for your workload and consult the official documentation for advanced configuration and performance tuning.
Source: https://kifarunix.com/install-and-configure-logstash-7-on-ubuntu-18-debian-9-8/
