Emotion Detection

Emotion Detection reads the caller's emotional state on every turn of a live call and lets the agent adapt — more patient with a frustrated caller, more concise with a confused one — without slowing the conversation down or ever mentioning to the caller that it's happening.

This is a live-call signal, distinct from the QA node's overall_sentiment, which is generated once, after the call ends, from the full transcript. Emotion Detection runs turn-by-turn, during the call itself.

Enabling it

Open an agent's Settings and toggle Emotion Detection on. It's off by default for every agent — nothing changes until you enable it per agent.

How it works

On, an instruction is added to the agent's system prompt asking the model to silently report the caller's emotion once per turn, in parallel with its normal spoken reply — the same mechanism the builder already uses for node-transition tool calls, so a spoken response and a tool call can come out of one completion together.

This is what makes it zero added latency: detection rides along with the reply the agent is already generating, on a silent side channel that never triggers a second LLM completion. It never adds a round trip, and the caller never hears or sees any indication it's running.

Emotion taxonomy

A small, actionable set — not fine-grained sentiment analysis, since the point is tone adaptation and routing, not nuance:

Where you see it

The detected emotion appears as a small badge next to the caller's message — on the live call view while the call is in progress, and on the Run Detail page's conversation panel afterward, since both share the same transcript rendering. It's attached to the caller's own turn, not the agent's reply.

Live sentiment

While a call is live, the conversation panel also shows a rolled-up Sentiment badge — Positive, Neutral, or Negative — next to the Live indicator, so you can read how the call is going at a glance instead of tracking each turn's emotion individually. It updates as the call progresses, based on the caller's most recent emotions:

This is a live-only view of the same signal described above — it disappears once the call ends, since the completed Run Detail page already shows QA's own overall_sentiment, generated once from the full transcript after the call.

Escalation flag

If a caller shows frustrated or angry emotion for 2 consecutive turns, the run is internally marked as an emotion escalation. Two separate things happen from there:

• Live, in the call view — a "Needs Attention" banner appears in the conversation panel the moment the escalation happens, visible to anyone watching the call live.
• After the call — this feeds into QA Alerts: turn on Alert on live emotion escalation there to have this flag a call and notify your organization's admins, the same way a low QA score does.

There's still no automatic transfer — Emotion Detection gives you the signal, not an action; see Common mistakes below.

Adaptive Voice Tone

Emotion Detection on its own only changes what the model knows — Adaptive Voice Tone is what makes it audible. Enable it below the Emotion Detection toggle (it's disabled until Emotion Detection is on) and the agent's own vocal delivery shifts, turn to turn, in direct response to the caller's detected emotion. What actually changes depends on your TTS provider, since each one exposes different delivery controls:

This is deliberately a response to the caller's emotion, not a mirror of it — the agent de-escalates frustration and anger rather than matching it.

Requires a supported voice provider, in standard (non-Realtime) mode -- Realtime's speech-to-speech pipeline has no separate TTS step for this to reach. On any unsupported provider, or in Realtime mode, the toggle is disabled and the settings page explains why rather than silently doing nothing. On Cartesia specifically, you can also set a baseline emotion independent of this feature, under the agent's TTS provider settings (e.g. always speak with a "warm" or "excited" default delivery) — that applies regardless of whether Adaptive Voice Tone or Emotion Detection are on.

Same zero-added-latency guarantee as Emotion Detection itself: this is a settings update pushed alongside the existing detection call, not a second LLM round trip, so it never slows down the agent's reply.

Common mistakes

Expecting it to change the agent's behavior on its own. Emotion Detection only makes the caller's emotional state visible to the model as context for its next turn — the agent adapting its tone is a natural consequence of the model reading that context, not a separate rule engine. If your agent isn't softening its tone appropriately, adjust the agent's prompt to explicitly reference the caller's emotional state.

Expecting a hard auto-transfer on anger. There's no built-in "angry caller → transfer to human" trigger. If you need that, add a transfer tool and prompt instructions telling the agent when to use it — Emotion Detection gives the model the signal to make that call, but doesn't force the action itself.

Next Steps