Talkr AI Documentation

Prometheus metrics and 5 pre-built Grafana dashboards for capacity planning and call-quality monitoring

Talkr ships with an optional Prometheus + Grafana stack: call-latency histograms, concurrency and queue-depth gauges, telephony failure counters, and container/database/Redis metrics, all wired into 5 pre-built Grafana dashboards. It's off by default — enable it when you want real historical metrics and dashboards instead of just the built-in health checks.

This is a deeper layer than a basic uptime check — a good complement to keep running alongside it even once this stack is up, since it's the one thing that can still alert you if the monitoring stack itself goes down.

What's included

Component	Purpose
Prometheus	Scrapes and stores every metric below. Internal only — never exposed publicly, no authentication of its own.
Grafana	The dashboards and alerting UI. Has its own login (admin password you set); this is the one thing you'd expose publicly if you want to check it from anywhere.
cAdvisor	Per-container CPU, memory, and network metrics.
postgres_exporter	Connection counts, cache-hit ratio, database health.
redis_exporter	Memory usage, ops/sec, hit rate, connected clients.

Talkr's own API and background workers export metrics directly — no separate agent needed:

HTTP request rate, latency, and error rate
Per-call STT/LLM/TTS time-to-first-byte and end-to-end response latency
Active call concurrency (fleet-wide and per-worker) and concurrency-limit events
Background job (ARQ) queue depth, throughput, and failure rate
Classified failures by source, type, provider, and whether they're retryable
Database/Redis reachability and event-loop lag

Enabling it

docker compose --profile monitoring up -d

Set these first, alongside your other deployment secrets in .env:

GRAFANA_ADMIN_PASSWORD=   # required — Grafana's admin login
GRAFANA_ROOT_URL=         # the URL you'll reach Grafana at (defaults to http://localhost:3030)
PROMETHEUS_RETENTION=30d  # optional, how long metrics are kept

Grafana comes up with all 5 dashboards and a Prometheus datasource already configured — nothing to click through manually. Sign in at GRAFANA_ROOT_URL (or http://localhost:3030 if you haven't put it behind your own domain yet) with the admin user and the password you set.

Prometheus, cAdvisor, and the two exporters are bound to 127.0.0.1 only and have no authentication of their own — by design, they're never meant to be reachable from outside the host. Only Grafana (which has a real login) is safe to expose publicly, e.g. behind your own reverse proxy with TLS.

The dashboards

Platform Overview — golden signals: request rate, error rate, latency percentiles by route, active calls, event-loop lag.
Call Pipeline Latency — STT/LLM/TTS time-to-first-byte and end-to-end bot response latency, the numbers that matter most for how a call actually feels.
Capacity & Concurrency — active calls against your configured limits, and background job queue depth/throughput — the numbers to watch before and during a load test.
Infrastructure — container CPU/RAM/network, database connections and cache-hit ratio, Redis memory and hit rate.
Telephony Reliability — call failures broken down by source, type, and provider, plus call volume and duration trends.

Built-in alerting (in Grafana, no separate service to run) covers a high HTTP error rate, a call-latency SLO breach, a sustained background-job backlog, the database or Redis going unreachable, calls completing with no caller speech ever heard, and calls completing with no bot response ever heard — delivered either by email or to a webhook. The last two catch total silent failures (e.g. an audio pipeline dependency breaking) that a "calls are completing fine" view alone would miss.

Where alerts go

Alerts default to email, using the same SendGrid credentials as your other Talkr notifications. On any real deployment, send them to a webhook instead:

ALERT_WEBHOOK_URL=https://hooks.slack.com/services/...   # or Discord, Telegram, ntfy, a pager
ALERT_RECEIVER=talkr-alerts-webhook                       # switches Grafana over

Setting ALERT_WEBHOOK_URL switches the standalone health monitor automatically; ALERT_RECEIVER switches Grafana's alert rules. The payload is plain JSON with a Slack-compatible text field, so most endpoints accept it as-is.

Alerts and user-facing email (invites, password resets) share one email quota. A busy alerting day can use it up, and the next real invite is then rejected — usually looking like a credentials problem rather than a quota one. Sending alerts to a webhook costs nothing, has no daily limit, and keeps your users' email working no matter how noisy monitoring gets.

Point the webhook somewhere outside this deployment. An alert saying "the API is down" shouldn't depend on the API to deliver it.

Synthetic call monitoring (the platform canary)

Real call volume tells you a lot, but during quiet hours — or right after a deploy, before any real call has happened yet — there's nothing to alert on. The eval canary places a real, fully synthesized call through your actual pipeline (real STT/LLM/TTS, no mocking) every 30 minutes and records pass/fail, so a broken deploy or a dead dependency is caught even with zero organic traffic.

It's off by default and never guesses at a workflow to use — point it at a small, dedicated workflow you control (not a live customer flow, since every synthetic call runs through the same post-call pipeline it would for a real caller):

EVAL_CANARY_WORKFLOW_ID=
EVAL_CANARY_ORGANIZATION_ID=
EVAL_CANARY_USER_ID=
PLATFORM_ALERT_EMAIL=   # where failure alerts go

scripts/seed_platform_canary.py creates that dedicated organization, user, and two-node workflow for you and prints the IDs to set above. Its TTS side is always free (a self-hosted, in-process model, no external API or key) — its LLM/STT side reuses whatever TALKR_PLATFORM_LLM_*/TALKR_PLATFORM_STT_* credentials this install already has configured for the trial-managed AI tier; if this install doesn't have those set, either set them or point the script at a different provider (it fails with a clear error naming exactly what to change). Run it once, set the four variables above, and restart the API. Failures alert on the first miss and then roughly hourly for as long as the outage continues, rather than paging once and going quiet.

Disabling it

docker compose --profile monitoring down

Metrics history is kept in a Docker volume, so starting the profile back up later picks up where it left off (until you also remove the volumes).

Observability