/ Deliverability ops · Observability guide
KumoMTA observability with Prometheus and Grafana.
KumoMTA exposes more than a hundred native metrics over a Prometheus-compatible endpoint. This guide covers the observability mental model for an MTA, the setup from the HTTP listener through the scrape config to the official Grafana dashboard, the metrics that actually matter grouped by category, the Four Golden Signals applied to email delivery with real PromQL, and a troubleshooting playbook that maps symptoms to the metrics that explain them.
In short
KumoMTA exposes a Prometheus-format metrics endpoint on the HTTP listener you configure with start_http_listener, usually port 8000 at /metrics. Point Prometheus at it with a five-second scrape interval, add a Prometheus data source in Grafana, and import the official starter dashboard ID 21391. The metrics that matter group into four families: throughput, queue depth, delivery outcomes, and system health. Pair KumoMTA's native metrics with the node_exporter so you can tell an MTA problem from a host problem, and alert on symptoms — rising permanent-failure rate, queue not draining, disk filling — rather than on every transient deferral.
-
Port 8000
The HTTP listener serves both the admin API and the
/metricsendpoint Prometheus scrapes. One port per kumod instance. -
ID 21391
The official KumoMTA starter Grafana dashboard. Import by ID directly once a Prometheus data source is connected.
-
100+ metrics
Native KumoMTA metrics, expanding per queue, per domain, and per provider. Includes lua_event_latency, disk_free_percent, thread_pool_size.
-
5s scrape
The scrape interval KumoMTA documentation recommends — high enough resolution to catch transient spikes without overloading Prometheus.
/ 01 — Mental model
Why is observing an MTA different from observing a web service?
A web service has a simple success model: a request comes in, a response goes out, and the latency between the two is the headline number. An MTA is messier. A message accepted into the queue is not delivered yet; it may sit for seconds or hours; it may be deferred by the receiver and retried later; it may succeed on the fourth attempt after failing on the first three. Success is a process that unfolds over time, not an event that completes in milliseconds.
That difference shapes what you measure. For a web service you watch request rate, error rate, and latency. For an MTA you watch those too, but the signals that matter live in the queue: its depth, whether it's draining or growing, how long messages wait, and the deferred-versus-delivered-versus-failed split. A steadily growing queue is rising latency for an MTA — invisible if you only watch front-door acceptance.
The second difference is that an MTA's success depends on systems you do not control. A receiver can throttle you, block you, or change its rules overnight. Your metrics need to surface receiver behavior — deferral patterns, rejection codes, per-provider delivery rates — because a drop in delivery success often originates outside your infrastructure and your dashboard is the first place you will see the symptom.
There is a third difference worth naming early because it shapes how you build dashboards: cardinality. A web service has a bounded set of endpoints, so its metrics have predictable cardinality. An MTA sends to an unbounded set of destination domains, each of which can spawn its own queue, its own connection pool, and its own set of per-receiver metrics. A busy sender can have thousands of active queues at once. That means a naive dashboard panel that graphs every queue individually becomes unreadable, and a Prometheus instance that stores every per-domain series without thought can run into memory pressure. The craft is choosing which dimensions to aggregate on the top-line panels and which to leave available for drill-down when a specific queue misbehaves.
Three questions a good MTA dashboard answers
- Is mail flowing?
- Throughput metrics: connections open, messages delivered per minute, per receiver. The top-line health check.
- Is mail backing up?
- Queue depth and trend. A queue that grows faster than it drains is the earliest signal of trouble, often before delivery rates visibly drop.
- Is mail being rejected?
- Outcome metrics: permanent failures and transient deferrals by receiver. The signal that a reputation or block problem is developing.
/ 02 — Setup
How do you wire KumoMTA into Prometheus and Grafana?
Four steps: expose the metrics endpoint in your Lua policy, point Prometheus at it, add the data source in Grafana, and import the official dashboard. Each step is small; the whole thing is a thirty-minute job for someone who has run Prometheus before.
Step 1 · Expose the HTTP listener in init.lua
kumo.on("init", function()
-- The HTTP listener serves both admin API and /metrics
kumo.start_http_listener {
listen = "0.0.0.0:8000",
-- restrict to your monitoring network in production
trusted_hosts = { "10.0.0.0/8", "127.0.0.1" },
}
end)Step 2 · Verify the feed with curl
$ curl -s http://localhost:8000/metrics | head -20
# HELP connection_count number of active connections
# TYPE connection_count gauge
connection_count{service="smtp_client"} 342
connection_count{service="smtp_client:gmail.com"} 88
# TYPE scheduled_count gauge
scheduled_count{queue="gmail.com"} 1240Step 3 · Prometheus scrape config
# prometheus.yml — the config from the KumoMTA docs
scrape_configs:
- job_name: kumomta
scrape_interval: 5s
metrics_path: /metrics
static_configs:
- targets:
- 'kumomta-1:8000'
- 'kumomta-2:8000'
- job_name: node
static_configs:
- targets:
- 'kumomta-1:9100' # node_exporter
- 'kumomta-2:9100'Step 4 · Import the official dashboard
In Grafana, configure a Prometheus data source pointed at your Prometheus server, then go to Dashboards, Import, and enter dashboard ID 21391. Grafana fetches the official KumoMTA starter dashboard from grafana.com and wires it to your data source. The community KetchUP dashboard at ID 21586 is an alternative with a different panel layout. Start with 21391, learn which panels you actually look at, then build a derivative dashboard tuned to your sending.
/ 03 — Pipeline
The observability pipeline from kumod to dashboard.
The data flows in one direction: each kumod and each host exposes a metrics endpoint, Prometheus scrapes them on a schedule and stores the time series, Grafana queries Prometheus with PromQL and renders the panels. Alertmanager sits beside Prometheus and fires when rules match.
/ 04 — Metrics
Which of the hundred-plus metrics actually matter?
KumoMTA exposes more than a hundred metrics and the number grows per queue, per domain, and per provider. The table below groups the base metrics that earn a place on a daily dashboard into four families. Everything else is drill-down detail you reach for when one of these four families shows a problem.
| Family | Metric | What it tells you |
|---|---|---|
| Throughput | connection_count | Active SMTP connections, per service and per receiver. Drops to zero against one receiver mean a block. |
| total_messages_delivered | Cumulative deliveries. Rate of change is your real-time send rate per receiver. | |
| Queue | scheduled_count | Messages waiting in the scheduled queue, per queue. Growing monotonically means messages are not draining. |
| ready_count | Messages ready to dispatch, waiting on a connection or shaping permit. High values point to throttling. | |
| Outcomes | total_messages_fail | Permanent failures. A rising rate by receiver is the earliest sign of a reputation or block problem. |
| total_messages_transfail | Transient deferrals. Normal in moderation; a spike against one receiver signals throttling or a soft block. | |
| System | disk_free_percent | Free space on the spool volume. A floor breach threatens the queue and must page someone. |
| lua_event_latency | Time spent in Lua policy handlers. Rising latency points to slow I/O in a hot-path handler. | |
| thread_pool_size | Worker thread pool occupancy. Saturation means the MTA is CPU- or I/O-bound and cannot scale further on this host. |
Metric names follow the KumoMTA metrics feed; exact names and labels evolve between releases, so confirm against your own /metrics output. The families and what each one tells you remain stable even as names shift. New metrics in later releases almost always fall into one of these four families, so a dashboard built around them rarely needs restructuring.
/ 05 — Golden signals
How do Google's Four Golden Signals map to an MTA?
Google's SRE practice defines four signals to watch on any system: latency, traffic, errors, and saturation. The mapping to an MTA is clean once you accept that delivery is a process rather than an event. Real PromQL for each signal below.
Signal · Traffic
Send rate per receiver
# Deliveries per second, by queue
rate(total_messages_delivered[5m])The top-line "is mail flowing" number. Break it down by queue label to see per-receiver throughput.
Signal · Errors
Permanent failure ratio
# Permanent fails as fraction of attempts
rate(total_messages_fail[5m])
/ rate(total_messages_delivered[5m])The deliverability health ratio. A sustained climb against one receiver is a developing block or reputation hit.
Signal · Saturation
Queue depth trend
# Total scheduled, and is it growing?
sum(scheduled_count)
deriv(sum(scheduled_count)[15m:])A positive derivative sustained over the window means the queue is growing faster than it drains. The earliest saturation signal.
Signal · Latency
Policy handler latency
# 99th percentile Lua event latency
histogram_quantile(0.99,
rate(lua_event_latency_bucket[5m]))Rising p99 latency in policy handlers throttles throughput. Points at slow I/O in a hot-path event handler.
/ Interactive — the alert tuner
Where would you set the line?
Golden signal
\u2014
\u2014
Illustrative ranges; your real normal-versus-danger lines depend on volume, receivers, and reputation. Establish them from your own baseline, then place alerts between them.
/ 06 — Alerting
The alerts worth defining, and the ones that cause fatigue.
The discipline of alerting is to page a human only when a human needs to act. Alert on symptoms users would notice, not on every metric that moves. The four rules below cover the conditions that warrant waking someone; everything else belongs on a dashboard, not in a pager. The cost of getting this wrong is not abstract: a team that pages on noise learns to ignore the pager, and then misses the one alert that mattered. Alert fatigue is the failure mode that quietly defeats otherwise excellent monitoring setups, so treat every new alert rule as a claim that this specific condition is always worth a human's interruption.
# alerting_rules.yml — symptom-based, not cause-based
groups:
- name: kumomta
rules:
- alert: HighPermanentFailureRate
expr: rate(total_messages_fail[10m])
/ rate(total_messages_delivered[10m]) > 0.05
for: 15m
labels: { severity: critical }
annotations:
summary: "Permanent failure rate above 5% for 15m"
- alert: QueueNotDraining
expr: deriv(sum(scheduled_count)[30m:]) > 0
for: 30m
labels: { severity: warning }
annotations:
summary: "Scheduled queue growing for 30m straight"
- alert: SpoolDiskLow
expr: disk_free_percent < 15
for: 5m
labels: { severity: critical }
annotations:
summary: "Spool volume below 15% free"
- alert: ReceiverDeliveryStalled
expr: rate(total_messages_delivered{queue=~".+"}[10m]) == 0
and scheduled_count > 100
for: 20m
labels: { severity: critical }
annotations:
summary: "A receiver has zero deliveries with mail queued"Alert on these
- Permanent failure ratio crossing a deliverability threshold
- Queue growing monotonically beyond a time window
- Spool disk approaching full
- A receiver going to zero deliveries with mail still queued
Do not alert on these
- Individual transient deferrals — normal MTA behavior
- Brief connection count dips that recover on their own
- Queue depth above an absolute number without a trend
- Any single scrape-interval spike that self-resolves
/ 07 — Troubleshooting
A troubleshooting playbook: symptom to metric to diagnosis.
When something looks wrong, the metrics tell you where to look. The playbook below maps the symptoms you notice first to the metrics that explain them and the diagnosis they point toward. The general method is the same each time: confirm whether the problem is localized to one receiver or spread across all of them, then narrow from there. A localized problem almost always lives at the receiver — a block, a throttle, a reputation event. A uniform problem almost always lives on your side — a host constraint, a policy regression, a configuration change. That single fork resolves most incidents to the right half of the system within minutes.
Symptom · Queue growing against one receiver
Look at: scheduled_count and total_messages_transfail for that queue. Diagnosis: if transient failures are climbing alongside the queue, the receiver is throttling or soft-blocking you. Check the deferral reason in the logs, slow your shaping for that receiver, and watch whether the queue starts draining.
Symptom · Throughput dropped across all receivers
Look at: thread_pool_size, lua_event_latency, and node_exporter CPU and network. Diagnosis: a uniform drop points to a host or MTA constraint rather than a receiver. Saturated thread pools mean CPU or I/O bound; high Lua latency means a slow policy handler; a network ceiling shows in node_exporter, not in KumoMTA metrics.
Symptom · Permanent failures spiking on Gmail
Look at: total_messages_fail for the Gmail queue, correlated with Postmaster Tools reputation. Diagnosis: a sharp climb in permanent failures against one major receiver, with no change in your sending, points to a reputation event. Cross-reference Postmaster Tools and check whether a specific IP or domain is the source.
Symptom · Connections to one receiver at zero
Look at: connection_count for that receiver and the queue's scheduled_count. Diagnosis: zero connections with a non-empty queue means you cannot establish a connection — a hard block, a DNS failure, or a firewall change. Check connectivity from the sending host and look for a block notification from the receiver.
/ 08 — Anti-patterns
Observability anti-patterns we have seen.
Anti-pattern 1 · Monitoring the MTA without the host
KumoMTA metrics alone cannot tell you the host ran out of network bandwidth or memory. Without node_exporter data on the same dashboard, you waste time looking inside the MTA for a problem that lives on the machine. Run both exporters and correlate them.
Anti-pattern 2 · Alerting on absolute queue depth
A queue of fifty thousand is fine if it is draining and a disaster if it is growing. Alerting on the absolute number produces false pages during normal sends and misses real problems that start from a low base. Alert on the trend, not the level.
Anti-pattern 3 · Ignoring per-receiver breakdown
An aggregate delivery rate that looks healthy can hide a complete block against one receiver if that receiver is a small fraction of volume. Always keep per-queue and per-receiver panels alongside the aggregates so a localized failure is visible.
Anti-pattern 4 · Scrape interval too coarse
A sixty-second scrape interval smooths over the transient spikes that precede a problem. The KumoMTA documentation recommends five seconds for a reason: MTA conditions change fast, and a coarse interval blinds you to the early signals.
/ 09 — FAQ
Questions operators ask when wiring up monitoring.
What metrics endpoint does KumoMTA expose for Prometheus?
KumoMTA exposes metrics on the HTTP listener you configure with start_http_listener, typically on port 8000, at the /metrics path. A simple curl to http://localhost:8000/metrics returns the full Prometheus-format feed. The same listener serves the admin API and the metrics endpoint, so you only open one HTTP port per kumod instance. Point your Prometheus scrape config at that port with metrics_path set to /metrics and a five-second scrape interval.
Is there an official KumoMTA Grafana dashboard?
Yes. The official starter dashboard is published on grafana.com with dashboard ID 21391. You import it directly in the Grafana UI by entering that ID, provided you have already configured a Prometheus data source. There is also a community-contributed dashboard, KetchUP KumoMTA, with ID 21586, which uses a different visual approach. Most operators start with 21391 and customize it as they learn which panels matter for their sending.
How many metrics does KumoMTA expose and which ones matter?
KumoMTA exposes more than a hundred native metrics, and the number changes with each release. The ones that matter for daily operations fall into four groups: throughput metrics like connection_count and message rates, queue metrics like scheduled_count and ready_count, delivery outcome metrics like delivered, transient fail, and permanent fail counts, and system health metrics like disk_free_percent, thread_pool_size, and lua_event_latency. The hundred-plus total includes per-queue, per-domain, and per-provider expansions of these base metrics.
What do the service labels like smtp_client mean?
KumoMTA labels metrics with a service identifier that lets you slice by scope. The bare smtp_client label represents the aggregate across all outbound SMTP delivery. A label like smtp_client:queue-name represents the count for one specific queue. This dual labeling means you can build one PromQL query that shows total outbound connections and another that breaks the same metric down per queue. The aggregate labels go on top-line panels; the scoped labels are what you drill into.
Should I also run the Prometheus node_exporter?
Yes. KumoMTA's native metrics tell you about the MTA's internal state, but not about the host it runs on. The node_exporter adds CPU, memory, network, and disk metrics for the underlying machine, which you need to distinguish an MTA problem from a host problem. When throughput drops, node_exporter data tells you whether the cause is inside the MTA, like a queue backup, or outside it, like the host hitting a network ceiling. Run both and correlate them on the same Grafana dashboard.
What alerting rules should I define?
Start with symptom-based alerts rather than cause-based ones. Alert when the permanent failure rate crosses a deliverability threshold, when queue depth grows monotonically beyond a window, when disk_free_percent falls below a floor, and when a sending IP stops delivering with mail still queued. Avoid alerting on every transient deferral because deferrals are normal MTA behavior. The discipline is to alert on conditions that require a human to act.
/ Next step
A dashboard tuned to your sending beats a generic starter.
The deliverability team at Email Delivery Platform operates KumoMTA observability across multiple tenants, with per-receiver panels, symptom-based alerting, and the node_exporter correlation that turns a metrics feed into an early-warning system. If you want a monitoring setup designed around your sending patterns, the discovery call begins with a review of what you measure today.