Proactive .NET API Monitoring: How to Detect and Alert on 5xx Errors with OpenTelemetry, Prometheus, and Grafana
Silent failures are the bane of any production application. While your .NET API might seem healthy on the surface, intermittent 5xx server errors can be eroding user trust and causing hidden problems. Relying on customer complaints to discover these issues is a reactive strategy that damages your reputation. A proactive approach to monitoring is essential for maintaining application health and stability.
This guide will walk you through setting up a powerful, open-source monitoring and alerting system for your .NET APIs. By leveraging the combined strengths of OpenTelemetry, Prometheus, and Grafana, you can automatically detect and receive alerts for 5xx server errors, allowing you to address problems before they escalate.
Why 5xx Errors Demand Your Immediate Attention
HTTP 5xx status codes (e.g., 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable) represent critical server-side failures. They indicate that your application received a valid request but was unable to process it due to an internal problem. These errors are not the user’s fault, but they directly impact their experience.
Unchecked 5xx errors can lead to:
- Poor User Experience: Users encounter broken functionality, leading to frustration and abandonment.
- Loss of Credibility: A service that frequently fails is seen as unreliable.
- Data Integrity Issues: Failed operations can leave your data in an inconsistent state.
- Business Impact: In e-commerce or financial applications, these errors can directly translate to lost revenue.
Monitoring the rate of these errors is a fundamental indicator of your application’s stability and overall health.
The Modern Observability Trio: A Powerful Combination
To build our monitoring system, we’ll use three best-in-class open-source tools that work together seamlessly.
OpenTelemetry (OTel): This is the instrumentation layer. OpenTelemetry provides a standardized, vendor-agnostic set of APIs and SDKs to collect telemetry data (metrics, traces, and logs) from your application. By adding it to your .NET project, you can effortlessly export detailed performance data, including HTTP request counts and status codes.
Prometheus: This is our time-series database and monitoring engine. Prometheus is designed to scrape (pull) metrics from configured endpoints at regular intervals. It stores this data efficiently and provides a powerful query language called PromQL to analyze it. Our .NET application, instrumented with OpenTelemetry, will expose a
/metricsendpoint that Prometheus can read from.Grafana: This is our visualization and alerting powerhouse. Grafana connects to data sources like Prometheus and allows you to transform raw data into beautiful, actionable dashboards and automated alerts. We will use Grafana to create a visual representation of our 5xx error rate and configure an alert rule to notify us when it exceeds a specific threshold.
Building Your 5xx Alerting System: A Step-by-Step Guide
Let’s break down the process of setting up this robust monitoring stack.
Step 1: Instrument Your .NET API with OpenTelemetry
The first step is to integrate OpenTelemetry into your ASP.NET Core application. This is primarily done by adding a few NuGet packages and configuring the services in your Program.cs file.
You’ll need packages like:
OpenTelemetry.Extensions.HostingOpenTelemetry.AspNetCore.InstrumentationOpenTelemetry.Exporter.Prometheus.AspNetCore
Then, in your application’s service configuration, you will add the OpenTelemetry services. This code instructs your application to automatically track incoming HTTP requests and make the metrics available on a Prometheus-compatible endpoint.
The key action here is integrating OpenTelemetry directly into your application’s startup code. This enables the collection of standard metrics like http_server_request_duration_seconds, which automatically includes labels for the HTTP method, route, and—most importantly—the status code.
Step 2: Configure Prometheus to Scrape Your API’s Metrics
Once your application is exporting metrics, you need to tell Prometheus where to find them. This is done in the Prometheus configuration file, typically named prometheus.yml.
You will add a new scrape_config job that points to your .NET application’s host and port, specifically targeting the /metrics endpoint that the OpenTelemetry exporter creates.
scrape_configs:
- job_name: 'my-dotnet-api'
static_configs:
- targets: ['localhost:5000'] # Replace with your application's host and port
This simple configuration is all it takes for Prometheus to begin periodically collecting and storing your application’s performance data.
Step 3: Visualize the 5xx Error Rate in Grafana
With data flowing into Prometheus, you can now visualize it in Grafana.
Add Prometheus as a Data Source: In the Grafana UI, navigate to the data sources section and add a new Prometheus data source, pointing it to your Prometheus server’s URL.
Create a New Dashboard Panel: Create a new panel and select your Prometheus data source.
Write the PromQL Query: To specifically see the count of 5xx errors, you can use the following PromQL query:
sum(rate(http_server_request_duration_seconds_count{code=~"5.*"}[5m]))
Let’s break this down:
http_server_request_duration_seconds_count: This is the metric that counts all HTTP requests.{code=~"5.*"}: This is a filter that selects only the requests where the status code starts with a “5”.rate(...[5m]): This calculates the per-second rate of increase of these errors over a 5-minute rolling window, smoothing out brief spikes.sum(...): This aggregates the results in case your metric has multiple dimensions (like different routes).
This query gives you a clear, real-time view of your application’s 5xx error rate.
Step 4: Create a Powerful Alert in Grafana
Visualizing data is good, but automating notifications is better. Grafana’s alerting engine allows you to act on this data.
Create a New Alert Rule: In Grafana’s “Alerting” section, create a new rule using the same PromQL query from the previous step.
Set the Condition: The most crucial part is defining the alert condition. A simple but effective condition is to trigger an alert when the 5xx error rate is greater than 0.
sum(rate(http_server_request_duration_seconds_count{code=~"5.*"}[5m])) > 0Configure Notifications: Connect the alert to a notification channel, such as email, Slack, PagerDuty, or Microsoft Teams. This ensures the right people are notified instantly when a problem arises.
For production environments, you may want to set the threshold slightly higher to avoid noise from a single, transient error. However, for many critical systems, any 5xx error rate is unacceptable and warrants immediate investigation.
Conclusion: From Reactive to Proactive
By implementing this monitoring and alerting stack, you transform your approach to application maintenance. You move from a reactive state—waiting for users to report problems—to a proactive one where you are the first to know about critical server-side issues.
This combination of OpenTelemetry, Prometheus, and Grafana provides deep visibility into your .NET API’s health without locking you into a proprietary APM solution. You gain the power to detect, diagnose, and resolve issues faster, ensuring your application remains resilient, reliable, and trustworthy. Take control of your application’s stability today by implementing a robust 5xx error monitoring strategy.
Source: https://www.fosstechnix.com/monitoring-net-api-5xx-alerts-with-opentelemetry-prometheus-and-grafana/
