A Comprehensive Guide to Installing and Configuring Filebeat on Ubuntu & Debian
In the world of modern IT infrastructure, effective log management is not just a best practice—it’s a necessity. Centralizing logs from various servers and applications allows for easier monitoring, faster troubleshooting, and robust security analysis. Filebeat, a key component of the Elastic Stack (formerly ELK Stack), is a lightweight, powerful tool designed specifically for this purpose: shipping log data efficiently from your servers to a central location.
This guide provides a step-by-step walkthrough for installing and configuring Filebeat 7 on Ubuntu 18.04 and Debian 9 systems. We’ll cover everything from initial setup to enabling modules and securing your data pipeline.
Prerequisites
Before we begin, ensure you have the following:
- A server running a recent version of Ubuntu or Debian.
sudoor root privileges.- An accessible Elasticsearch or Logstash instance to send your logs to.
Step 1: Add the Elastic Repository
To ensure you are installing the official and most up-to-date version of Filebeat, it’s crucial to add the Elastic repository to your system. This involves adding the repository’s GPG key to verify the package authenticity and then adding the repository source itself.
First, import the Elastic public GPG key using wget:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Next, you may need to install the apt-transport-https package to access repositories over HTTPS:
sudo apt-get install apt-transport-https
Finally, add the Elastic repository to your system’s source list:
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
This one-time setup allows you to easily install and update any component from the Elastic Stack using your system’s package manager.
Step 2: Install the Filebeat Package
With the repository configured, installing Filebeat is as simple as running a standard package installation command. First, update your package lists to include the new Elastic repository, and then install Filebeat.
sudo apt-get update
sudo apt-get install filebeat
This command downloads and installs the Filebeat binary, its default configuration file, and the necessary systemd service unit.
Step 3: Configure Filebeat to Ship Your Logs
Filebeat’s main configuration file is located at /etc/filebeat/filebeat.yml. This YAML file controls everything from which log files to monitor to where the data should be sent.
It is critical to use spaces, not tabs, for indentation in YAML files.
Open the configuration file with your preferred text editor:
sudo nano /etc/filebeat/filebeat.yml
You’ll need to configure two main sections: filebeat.inputs and output.
Configuring Inputs
The filebeat.inputs section defines which log files to track. By default, it’s often disabled. You must enable it and specify the paths to your logs.
For example, to monitor standard system logs like syslog and auth.log, find the filebeat.inputs section and modify it as follows:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/syslog
- /var/log/auth.log
Configuring Outputs
The output section determines the destination for your log data. The two most common destinations are Elasticsearch and Logstash. You should only enable one output at a time.
Option 1: Sending Directly to Elasticsearch
If you are sending logs directly to Elasticsearch, find the output.elasticsearch section. Comment out the Logstash output section by adding a # at the beginning of each line.
Configure the Elasticsearch output with your cluster’s details:
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["your_elasticsearch_host:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "your_username"
#password: "your_password"
Replace your_elasticsearch_host:9200 with the actual IP address or hostname of your Elasticsearch instance.
Option 2: Sending to Logstash for Further Processing
If you use Logstash to parse or enrich your logs before they reach Elasticsearch, comment out the output.elasticsearch section and configure the output.logstash section instead:
output.logstash:
# The Logstash hosts
hosts: ["your_logstash_host:5044"]
Replace your_logstash_host:5044 with the address of your Logstash server. Port 5044 is the default port for the Beats input plugin in Logstash.
Step 4: Enable and Configure Filebeat Modules
One of Filebeat’s most powerful features is its modules, which provide pre-built configurations for common applications like Nginx, Apache, MySQL, and more. These modules automatically parse logs, and many come with pre-built Kibana dashboards.
To see a list of available modules, run:
sudo filebeat modules list
To enable a specific module, such as the system module (for syslog and auth logs) or the nginx module, use the enable command:
sudo filebeat modules enable system nginx
After enabling modules, you need to run the setup command. This step is crucial as it loads assets like Kibana dashboards and Elasticsearch index templates for the enabled modules. This command may take a minute to complete.
sudo filebeat setup -e
Step 5: Start and Enable the Filebeat Service
Once your configuration is complete, you can start the Filebeat service and enable it to run automatically on boot.
Use systemctl to manage the service:
# Start the Filebeat service
sudo systemctl start filebeat
# Enable Filebeat to start on boot
sudo systemctl enable filebeat
To verify that Filebeat is running correctly, check its status:
sudo systemctl status filebeat
You can also check the Filebeat logs for any errors: journalctl -u filebeat.service
Security Tip: Secure Your Connection with TLS
For production environments, it is highly recommended to encrypt the communication between Filebeat and your Elasticsearch or Logstash instance using TLS/SSL.
To do this, you will need the CA certificate from your Elastic Stack. Modify your output configuration to include the SSL settings.
Example for output.elasticsearch:
output.elasticsearch:
hosts: ["your_elasticsearch_host:9200"]
protocol: "https"
ssl:
enabled: true
certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
Ensure you copy your CA certificate (ca.crt) to the specified path on your server and secure its file permissions. This ensures that log data transmitted over the network is encrypted and protected from eavesdropping.
Source: https://kifarunix.com/install-and-configure-filebeat-7-on-ubuntu-18-04-debian-9-8/
