Mastering Prometheus: Your Guide to Setting Up Email Alerts with Alertmanager
Proactive monitoring is the bedrock of a reliable system. While Prometheus is exceptional at collecting metrics and identifying issues, its true power is unlocked when it can notify you of problems the moment they happen. Without a robust alerting system, you’re left staring at dashboards, waiting for something to break.
This is where Alertmanager comes in. As the dedicated handler for alerts sent by Prometheus, Alertmanager deduplicates, groups, and routes them to the correct receivers, such as email, Slack, or PagerDuty. Setting up email alerts is a fundamental step in creating a powerful, automated monitoring pipeline.
Let’s dive into the step-by-step process of configuring Prometheus and Alertmanager to send critical email notifications.
Step 1: Define Alerting Rules in Prometheus
Before you can receive an alert, you need to tell Prometheus what constitutes a problem. This is done by creating alerting rules in a YAML file. These rules define conditions based on your metrics that, when met, will trigger an alert.
Create a file named alert.rules.yml (or similar) and define your rules. For example, let’s create a rule that fires an alert if an instance has been down for more than one minute.
groups:
- name: instance_alerts
rules:
- alert: InstanceDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."
Let’s break down this rule:
- alert: The name of the alert.
- expr: The Prometheus Query Language (PromQL) expression. The alert fires if this expression returns a value. Here,
up == 0means the instance is not reachable. - for: The duration for which the condition must be true before the alert is fired. This prevents alerts for brief, transient issues.
- labels: Attaches custom labels to the alert, which can be used for routing in Alertmanager.
severity: criticalis a common example. - annotations: Provides additional, descriptive information about the alert, like a summary and description, which will be used in the body of our email.
Step 2: Connect Prometheus to Alertmanager
Now that you have rules, you must configure Prometheus to send any triggered alerts to your Alertmanager instance. This is done in your main Prometheus configuration file, prometheus.yml.
You need to add two key sections: rule_files to load your new alert rules, and alerting to specify the location of your Alertmanager.
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "alert.rules.yml"
# Alerting specifies settings related to alert notifications.
alerting:
alertmanagers:
- static_configs:
- targets:
- 'alertmanager:9093' # Assumes Alertmanager is running on a host named 'alertmanager'
In this configuration:
rule_filespoints to the file containing your alerting rules.alerting.alertmanagers.static_configs.targetsprovides the address of your Alertmanager instance. The default port for Alertmanager is 9093.
After adding this, restart or reload your Prometheus configuration for the changes to take effect.
Step 3: Configure Alertmanager to Send Emails
This is the final and most crucial piece of the puzzle. You need to configure Alertmanager to properly handle incoming alerts and route them to an email receiver. This is all managed in the alertmanager.yml file.
Here is a complete configuration for sending email alerts via an SMTP server, such as Gmail.
global:
resolve_timeout: 5m
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: '[email protected]'
smtp_auth_username: '[email protected]'
smtp_auth_password: 'YOUR_APP_PASSWORD' # Important: Use an app-specific password
route:
receiver: 'default-email'
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receivers:
- name: 'default-email'
email_configs:
- to: '[email protected]'
send_resolved: true
Let’s examine the key sections of this configuration:
global: This section defines parameters that are shared across all configurations.smtp_smarthost: The address of your SMTP server and its port.smtp_from: The “from” address that will appear on the alert emails.smtp_auth_username: Your SMTP username, usually your full email address.smtp_auth_password: Your SMTP password. Security Tip: For services like Gmail, it’s highly recommended to use an App Password instead of your main account password. This is more secure and avoids two-factor authentication issues. Never hardcode credentials in production; use a secret management tool.
route: This is the top-level routing tree. It defines how alerts are grouped and where they are sent.receiver: 'default-email': All alerts, by default, will be sent to the receiver named ‘default-email’.group_by: Groups alerts with the same labels into a single notification. This prevents an “alert storm” if many similar alerts fire at once.group_wait,group_interval, andrepeat_intervalcontrol the timing of notifications to avoid spamming your inbox.
receivers: This is a list of all possible notification destinations.name: 'default-email': A unique name for the receiver, referenced in theroute.email_configs: Defines the email-specific settings.to:: The recipient’s email address.send_resolved: true: A crucial setting that sends a follow-up notification when an alert’s condition is no longer met, confirming the issue is resolved.
Step 4: Apply Changes and Verify
Once you have updated your alertmanager.yml file, you need to restart or reload your Alertmanager service.
To verify that everything is working:
- Check the Prometheus UI: Navigate to the “Alerts” tab in your Prometheus dashboard. You should see your
InstanceDownalert listed. If it is firing, its state will be “PENDING” (during theforduration) and then “FIRING.” - Check the Alertmanager UI: Access your Alertmanager dashboard (usually at
http://<your-alertmanager-host>:9093). You should see the active alert received from Prometheus. - Check Your Inbox: If the alert is firing, Alertmanager will route it according to your configuration, and you should receive an email at the address specified in
to:. The email will contain the summary and description you defined in your alert rule.
By integrating Prometheus with Alertmanager for email notifications, you transform your monitoring system from a passive data collector into an active, automated guardian of your infrastructure. This setup ensures that you are immediately informed of critical issues, enabling you to respond quickly and maintain system reliability.
Source: https://kifarunix.com/configure-prometheus-email-alerting-with-alertmanager/
