Scaling

By default, the Talkr API container runs a single uvicorn worker. For production traffic — especially with many concurrent voice calls (long-lived WebSockets) — you'll want multiple workers. Talkr ships with built-in support for this: nginx load-balances across N independent uvicorn processes using a least_conn strategy.

This page covers how the multi-worker setup works, how to choose a worker count at install time, and how to change it on a running stack.

How it works

The API container starts FASTAPI_WORKERS separate uvicorn processes, each bound to its own port (8000, 8001, 8002, …). nginx exposes a single upstream talkr_api that includes all worker ports and routes new requests to whichever worker currently has the fewest active connections.

                       ┌───────────────────────────────────┐
                       │ api container                     │
                       │  uvicorn worker 0  → :8000        │
 browser ──► nginx ──► │  uvicorn worker 1  → :8001        │
   (443)    (least_conn)  uvicorn worker 2  → :8002        │
                       │  uvicorn worker 3  → :8003        │
                       └───────────────────────────────────┘

The ari_manager and campaign_orchestrator processes inside the API container stay as single processes regardless of FASTAPI_WORKERS — they coordinate global state (Asterisk channels, campaign scheduling) and should not be duplicated within a server. ARQ background workers are controlled separately via ARQ_WORKERS.

Across multiple servers, ari_manager coordinates automatically — see How SIP telephony is shared between servers.

Choosing a worker count

A safe starting point is one worker per available vCPU, capped at 8 unless you've profiled your workload. The Remote Server Deployment prerequisites ask for a minimum of 4 vCPUs, so:

Each worker holds its own Python process and memory — budget roughly 300–500 MB RAM per worker in addition to the postgres/redis/minio overhead. If you're near the 8 GB RAM minimum and see OOMs, drop the worker count before adding more.

How many concurrent calls one server handles

Measured on a 4 vCPU / 23 GB server running the full stack, with SIP calls arriving through Asterisk ARI:

A 4 vCPU server handles about 6 simultaneous conversations with noise suppression on, or about 12 with it off. Past that, callers start waiting noticeably longer for replies. Plan on roughly 0.3 of a CPU core per conversing call, or 0.15 with noise suppression disabled.

Noise suppression is the biggest single lever

Talkr applies RNNoise to inbound audio by default. It is also, by a wide margin, the most expensive thing on the audio path — profiling a loaded server attributes 78% of call-path CPU to it, because it processes every audio frame whether or not anyone is speaking, and 8 kHz phone audio must be upsampled to 48 kHz to run through it.

Turning it off roughly doubles how many conversations a server handles:

TALKR_NOISE_FILTER=0

In our testing it also improved transcription accuracy, which is the opposite of what noise suppression is meant to do. Comparing the same utterances with and without the filter, at several noise levels, word error rate was worse with suppression at every level — and the gap widened as audio got noisier:

The likely reason is the resampling: RNNoise runs only at 48 kHz, so phone audio is stretched up to meet it and squeezed back afterwards, and the filter appears to strip consonant detail along with the noise. Errors clustered in names and spelled-out confirmations — "Sasha" heard as "Tasha", "through" as "to you".

It still ships on by default, pending confirmation on real-world calls.

Idle calls cost much less

Calls that are connected but not conversing — on hold, listening to a long prompt, waiting in a queue — cost far less. The same server held 287 connected calls when little conversation was happening.

Note that this is genuinely idle, not merely "the caller pauses between sentences". Callers who speak in short bursts with multi-second gaps cost almost as much as callers who talk continuously, because the per-frame audio work continues through the silence. Size for the number of calls in active conversation, not for average talk time.

Memory is rarely the limit. Usage stayed near 6–7 GB of 23 GB throughout, barely moving with call count. Size for CPU first.

Raising the concurrency limits

Two settings cap concurrent calls:

These are admission ceilings, not capacity estimates. On a 4 vCPU server the default of 30 sits above what the box can serve conversationally (~6–12 per the measurements above), so it will not protect response times on its own — it exists to stop runaway traffic, and it binds mainly on mostly-idle calls.

Set them from your own measured capacity rather than raising them to a target number, and watch p95 response time as you do — that degrades before anything else does.

GLOBAL_CONCURRENT_CALL_LIMIT=120
DEFAULT_ORG_CONCURRENCY_LIMIT=60

Scaling beyond one server

Because the constraint is CPU, bigger servers scale as well as more of them — and usually cost less to operate. Using ~0.3 cores per conversing call (0.15 with noise suppression off), with roughly 75% of cores usable before latency suffers:

So 500 concurrent conversations is roughly 21 × 16 vCPU servers, or 11 with noise suppression disabled — not 50 small ones either way. Size against your own traffic, and settle the noise-suppression question first: it moves the server count by about half.

Postgres, Redis and MinIO are shared across servers and need their own sizing; they do not scale per call the same way.

How SIP telephony is shared between servers

Run the same Talkr stack on every server — there is no "designated ARI host" to configure.

Asterisk allows only one connection per Stasis application: a second connection using the same application name disconnects the first. So Talkr servers cannot simply all connect to the same Asterisk — they would take turns kicking each other off.

Instead, each server claims the telephony configurations it serves, using a short-lived lease held in Redis:

• A configuration is served by exactly one server at a time, so Asterisk only ever sees one connection per application.
• Configurations are spread across whichever servers are running — every server does useful work, rather than one leading and the rest standing by.
• If a server stops or fails, its lease expires and another server picks up its configurations within about 30 seconds. No manual failover, no configuration change.
• On a single-server install nothing changes: that server claims everything.

To scale SIP capacity beyond one server, add Asterisk instances as well as Talkr servers: each Asterisk is claimed by a Talkr server, and the load spreads across both tiers. Adding Talkr servers alone raises resilience and the number of concurrent conversations, but a single Asterisk still has one connection.

Setting the worker count at install time

setup_remote.sh prompts for the worker count alongside the other configuration:

Number of FastAPI workers (uvicorn processes nginx will load-balance):
[4]:

Press Enter for the default (4) or enter a different positive integer. Non-interactive callers (cloud-init, CI, Terraform) can set the value via environment variable instead:

sudo SERVER_IP=... TURN_SECRET=... FASTAPI_WORKERS=8 ./setup_remote.sh

The script stores the value in .env (FASTAPI_WORKERS=N). The supported startup path (./remote_up.sh) preflights the talkr-init render from that value before every remote start, so nginx and the API worker count stay aligned.

Changing the worker count on a running stack

Once Talkr is running, increasing or decreasing the worker count is a one-file edit plus a restart. Change .env, then start through ./remote_up.sh so talkr-init regenerates nginx runtime config before Docker starts the stack.

Steps

All commands run from your talkr/ directory (the one with docker-compose.yaml).

1. Edit .env and change the FASTAPI_WORKERS line:

# Before
FASTAPI_WORKERS=4

# After
FASTAPI_WORKERS=8

2. Recreate the stack through the validated wrapper. The simplest path — brief downtime, no surprises:

./remote_up.sh

If you want to avoid downtime and your stack is healthy, you can recreate only the api and nginx containers:

./remote_up.sh -- api nginx

remote_up.sh validates .env, runs the same talkr-init render that Compose will use at startup, runs docker compose config -q, and then starts the requested services.

3. Verify. Confirm the right number of uvicorn processes are running. The API image is slim and doesn't include ps, so use Docker's host-side view instead:

sudo docker compose --profile remote top api | grep uvicorn

You should see one line per worker. To confirm the bound ports, check the startup logs — each worker logs an Uvicorn running on http://0.0.0.0:800X line on boot:

sudo docker compose --profile remote logs api | grep "Uvicorn running"

Then hit the API through nginx to confirm requests still flow:

curl -k https://YOUR_SERVER_IP/api/v1/health

Why not just re-run setup_remote.sh?

setup_remote.sh refuses to overwrite an existing install by design — re-running it would regenerate OSS_JWT_SECRET (logging everyone out), reset the TURN shared secret (breaking WebRTC auth on connected clients), and regenerate SSL certificates. The two-file edit above is the supported way to change worker count after install.

If you genuinely want a clean reinstall, see the TALKR_FORCE_OVERWRITE=1 escape hatch documented in the script.

What this does not scale

Multi-worker mode scales the HTTP/WebSocket API surface. It does not scale:

• ARQ background workers — controlled by ARQ_WORKERS (defaults to 1). Increase this in the API container's environment if your background job queue backs up.
• ari_manager / campaign_orchestrator — one process per server by design; they don't benefit from extra processes. ari_manager does coordinate across servers, but through leases rather than extra processes (see above).
• Postgres, Redis, MinIO — each runs as a single container in the stack. For production-scale Postgres you'd run a managed service and point DATABASE_URL at it; the same applies to Redis and S3-compatible storage.

For multi-machine horizontal scaling (separate API containers across hosts), see the Custom Domain guide for the load-balancer-in-front-of-multiple-hosts pattern — it's the same idea as the in-container least_conn upstream, just one layer higher.