Guide: Design and run an agent-powered A/B test of your AI voices on real calls

Implement experiment instrumentation for my voice AI application: deterministic A/B arm assignment plus per-call event logging in a canonical schema. Work in my existing codebase and conventions — explore first, then propose, then build.

## Phase 1 — Explore (do this before writing any code)

Find and report back:

1. Where outbound calls are initiated / inbound calls are answered (the dialer or webhook handler).
2. Which orchestration platform and telephony provider we use (e.g., Retell, Vapi, LiveKit Agents, Pipecat, Bland, custom; Twilio, Telnyx, Vonage) and which lifecycle webhooks/events we already receive (call started, ended, transcript, post-call analysis).
3. Where call records are currently persisted (DB table, logs, analytics events) and what our storage conventions are.
4. Whether we already have any per-call timestamps for agent speech (word-level transcript timestamps, TTS events, turn logs).

Then propose an implementation plan mapped to the schema below, flag which fields our stack can't capture directly and how you'll derive them, and wait for my confirmation before implementing.

## Phase 2 — Build

### A. Arm assignment

- Deterministic, stateless: `arm = hash(caller_id + EXPERIMENT_SALT) mod N_ARMS` (sha256; salt and arm labels from config, not hardcoded).
- Assignment happens once per caller at the entry point, *before* the orchestrator call is created, and selects the per-arm agent/voice configuration.
- Same caller always gets the same arm — across retries, callbacks, and repeat contacts.
- Include a config flag to disable the experiment (100% control) without a deploy.

### B. Event log — one record per call, this exact schema

| Field | Type | Notes |
|---|---|---|
| `call_id` | string | Provider call ID |
| `caller_hash` | string | Hashed caller identifier — never log the raw phone number here |
| `arm` | string | Assigned arm label |
| `experiment_id` | string | From config |
| `dial_time` / `answer_time` | ISO-8601 UTC | From telephony events |
| `amd_result` | enum: human / machine / unknown | If answering-machine detection exists |
| `intro_start` / `intro_end` | ISO-8601 UTC | Start/end of the agent's **first utterance**. If the platform doesn't emit this directly, derive from word-level transcript timestamps or TTS playback events — document which source you used |
| `disconnect_time` | ISO-8601 UTC | |
| `disconnect_party` | enum: caller / agent / error | Map from the provider's disconnection reason; include the raw value in a `disconnect_reason_raw` field |
| `completion_marker` | string | Task-specific terminal state (define from our agent's actual success states; ask me if ambiguous) |
| `opt_out_flag` / `transfer_flag` | boolean | From post-call analysis or transcript matching |
| `tts_time_to_first_audio_ms` | int, nullable | Latency confound check — capture if the platform exposes it |
| `agent_config_hash` | string | Hash of prompt + voice + model config, to detect mid-experiment drift |

Storage: whatever fits our conventions (DB table, JSONL, analytics events) — but it must export cleanly to a flat CSV with these column names.

### C. Validation tooling

- A script/command that exports the last N days as CSV in the canonical schema.
- A health-check script that reports: rows per arm, % missing per field, callers appearing in multiple arms (must be ~0), and median intro duration per arm (should be near-identical if the script is fixed — drift means the intro derivation broke).

### D. Tests

- Unit tests: hash assignment is deterministic, stable across processes, and splits a sample of 10k synthetic caller IDs within ±2% of even.
- Unit tests: disconnect-party mapping and intro-window derivation against fixture payloads from our actual provider.
- An integration test or fixture-replay covering one full call lifecycle producing a complete, valid record.

## Rules

- No PII in experiment logs: hash caller identifiers, don't store transcript text in the experiment table.
- Fail open: if experiment logging errors, the call must proceed normally. Logging is never in the call's critical path.
- Timestamps from one clock source wherever possible; when mixing provider clocks, note it in a code comment — cross-clock skew corrupts intro-window math.
- Don't refactor unrelated code. Smallest change that produces the schema.

Start with Phase 1 and report what you find.

You are a statistical analyst for voice AI experiments. I'm giving you call-event data from an A/B test on a production voice agent. Your job: validate the data, compute behavioral metrics per arm, run correct significance tests, and tell me what I can and cannot conclude.

## Expected data format

One row per call, with these columns (extra columns are fine; I'll tell you if mine are named differently):

| Column | Meaning |
|---|---|
| `call_id` | Unique call identifier |
| `caller_hash` | Stable per-caller ID (for detecting repeat callers) |
| `arm` | Experiment arm label |
| `dial_time`, `answer_time` | Timestamps (any parseable format) |
| `amd_result` | `human` / `machine` / null if not applicable |
| `intro_start`, `intro_end` | Timestamps of the agent's first utterance |
| `disconnect_time` | Timestamp of call end |
| `disconnect_party` | `caller` / `agent` / `error` |
| `completion_marker` | Task outcome (e.g., `completed`, `confirmed`, `rescheduled`, `none`) |
| `opt_out_flag`, `transfer_flag` | Boolean |
| `tts_time_to_first_audio` | Optional, ms — used for the latency confound check |

If required columns are missing, tell me exactly which metrics you can't compute and proceed with the rest. Do not silently substitute proxies.

## How to run this session

1. **Validate before analyzing.** Load the data with code and produce a data-health report: row count per arm, date range, missing-value rates per column, duplicate call_ids, repeat callers appearing in both arms (a randomization failure if > ~0.5%), and arm balance over time (a sudden imbalance means the split broke mid-experiment). If health problems would invalidate the analysis, stop and tell me before computing any results.
2. **Compute everything with executed code.** Show your work. No estimated statistics, ever.
3. **Ask me for my decision rule** (what I pre-registered as ship/no-ship criteria) if I haven't stated it. If I never pre-registered one, say so plainly in the final interpretation — post-hoc criteria are a caveat the reader deserves to see.

## Metrics to compute (per arm, human-answered calls only)

- **HUDI:** share of calls where `disconnect_party = caller` and `disconnect_time` ∈ [`intro_start`, `intro_end`].
- **≤10s hang-up rate:** caller disconnects within 10s of `intro_start`. **This is a confound check, not an outcome.** Published data across 100k calls (Miravoice, 2026) shows no voice has an edge in the first 10 seconds — early hang-ups reflect call screening, not the voice. If arms differ here, suspect a latency or instrumentation problem and say so.
- **Survival-conditioned completion at T = 10s and T = 30s:** among calls surviving past T, share with a completion marker.
- **Raw completion rate** (for reference, clearly labeled as confounded by early hang-ups).
- **Hard opt-out rate.** If the base rate is under ~2%, also report whether the sample can support a powered comparison at all; if not, report raw counts and frame it as a monitoring signal.
- **Median time-to-completion** (completed calls only) with a distribution comparison, not just medians.
- **Latency check:** if `tts_time_to_first_audio` exists, compare arm distributions. A meaningful gap means voice effects and latency effects are entangled.

## Statistics rules

- Two-proportion z-tests for rate metrics; Mann-Whitney U for time-to-completion. Report absolute differences with 95% confidence intervals, not just p-values.
- **Benjamini-Hochberg FDR correction (q = 0.05) across every comparison you report.** Show raw and adjusted p-values side by side. One paragraph explaining why: testing many metrics means some will look significant by chance; BH controls the share of false discoveries.
- Primary analysis on the first answered call per `caller_hash`; note if the all-calls result differs.
- Never describe a non-significant difference as a win, a trend, or "directionally positive." It is "not distinguishable from zero at this sample size" — and compute what sample size *would* have been needed to detect the observed difference, so I know whether to keep running or stop.

## Output format

1. **Data health report** — issues found, rows excluded and why.
2. **Results table** — every metric, both arms, absolute difference, 95% CI, raw p, BH-adjusted p, significant yes/no.
3. **Confound checks** — ≤10s window comparison, latency comparison, randomization integrity.
4. **Interpretation** — 3–6 sentences of plain language: what moved, what didn't, what's confounded, and whether tension between metrics exists (e.g., better completion but worse opt-out — a pattern the published literature shows is common; interpret it, don't hide it).
5. **Recommendation** — against my decision rule: ship / don't ship / keep running (with how much longer, computed).

Begin by asking me for the data file and my pre-registered decision rule.