Mastering Centralized Logging: A Guide to Shipping System Logs to the ELK Stack
In today’s complex IT environments, server logs are a goldmine of information. They are essential for troubleshooting application errors, monitoring system performance, and detecting security threats. However, when logs are scattered across dozens or even hundreds of servers, managing them becomes a monumental task. This is where centralized logging comes in, and the ELK Stack (Elasticsearch, Logstash, Kibana) stands out as a powerful, industry-leading solution.
This guide will walk you through the modern approach to shipping system logs to your ELK Stack using the Elastic Agent, a unified and efficient method for data collection.
What is the ELK Stack?
Before diving into the process, let’s quickly recap the components of the ELK Stack. It’s a trio of powerful open-source tools that work together to provide a comprehensive logging platform.
- Elasticsearch: A distributed search and analytics engine at the heart of the stack. It stores, indexes, and makes vast amounts of log data searchable in near real-time.
- Logstash: A server-side data processing pipeline that ingests data from multiple sources, transforms it, and then sends it to a “stash” like Elasticsearch. While still powerful, its role for simple log shipping is often replaced by the more lightweight Elastic Agent.
- Kibana: The visualization layer of the stack. Kibana is a web interface that allows you to explore, visualize, and create dashboards from the data stored in Elasticsearch. This is where you’ll see your logs come to life.
The Modern Approach: Elastic Agent and Fleet
In the past, you needed to install multiple data shippers (called “Beats,” like Filebeat for logs and Metricbeat for metrics) on your servers. Today, the process is streamlined with Elastic Agent, a single, unified agent that you can manage centrally using Elastic Fleet.
- Elastic Agent: One agent to install on your host systems (like a Linux server or Windows machine). It can collect logs, metrics, traces, and security data and ship it directly to Elasticsearch.
- Fleet: A powerful application within Kibana that acts as a central management console for all your Elastic Agents. With Fleet, you can deploy, configure, and update agents across your entire infrastructure without having to log into individual machines.
This combination dramatically simplifies deployment and ongoing management, making your observability and security efforts more efficient.
Step-by-Step: Shipping Logs with Elastic Agent
Here’s a high-level overview of the steps required to get your system logs flowing from a server into the ELK Stack.
Step 1: Set Up the Fleet Server
The Fleet Server is the control plane that connects your Elastic Agents to the ELK Stack. It acts as the central communication point for policy updates and agent monitoring. You’ll first need to set up and run a Fleet Server and configure Kibana to connect to it. This is a one-time setup that enables you to manage thousands of agents.
Step 2: Create an Agent Policy in Kibana
An agent policy is a collection of settings that defines what data an agent should collect. This is where you specify the integrations you want to use.
- Navigate to the Fleet section in Kibana.
- Create a new Agent policy. Give it a descriptive name, like “Linux Production Servers.”
- Add an integration to the policy. For collecting system logs, metrics, and more, the System integration is the perfect choice. It’s a comprehensive package that collects:
- System logs (e.g., from
/var/log/syslogorjournald). - System performance metrics (CPU, memory, disk, network).
- Audit logs for security purposes.
- System logs (e.g., from
Once you add and save the integration, this policy is ready to be assigned to your agents.
Step 3: Install and Enroll the Elastic Agent
Now it’s time to deploy the agent on the server you want to monitor. The beauty of Fleet is that it makes this incredibly simple.
- In your Agent policy within Fleet, click “Add agent.”
- Fleet will provide you with a single, customized command to run on your target server. This command handles downloading the Elastic Agent, installing it as a service, and enrolling it with the Fleet Server under the correct policy.
- Simply copy this command and run it with administrative privileges (e.g.,
sudo) on your Linux or Windows server.
The agent will automatically start, receive its configuration from the policy you created, and begin shipping data.
Step 4: Verify Your Data in Kibana
Within moments, you should see your agent appear as “Healthy” in the Fleet UI. More importantly, your log data will start arriving in Elasticsearch.
To see your new data, navigate to the Discover app in Kibana. You can filter for your host’s name to see the raw log events. Even better, the System integration comes with pre-built dashboards. Go to the Dashboard app and search for dashboards related to the “System” integration to get instant, rich visualizations of your server’s performance and logs.
Key Benefits of Centralizing Your Logs
Taking the time to set up a centralized logging pipeline provides immense value across your organization.
- Enhanced Security Posture: By centralizing security logs (like audit logs and firewall events), you can set up alerts for suspicious activity, perform forensic analysis after an incident, and meet compliance requirements like PCI DSS or GDPR.
- Faster Troubleshooting: When an application fails, you no longer need to SSH into multiple servers to hunt for clues. All relevant logs are in one place, correlated by timestamp, allowing developers and operations teams to diagnose and resolve issues in record time.
- Proactive Operational Visibility: Pre-built dashboards give you a real-time view of your systems’ health. You can spot performance degradation, identify resource bottlenecks, and proactively address problems before they impact users.
- Scalable and Unified Data Collection: The Elastic Agent and Fleet model is built for scale. Whether you have ten servers or ten thousand, you can manage your entire data collection strategy from a single interface, ensuring consistency and control.
Source: https://kifarunix.com/ship-system-logs-to-elk-stack-using-elastic-agents/
