Kinescope — a flight recorder for AI agents (deterministic replay, counterfactual forks)

Debugging an agent is archaeology through logs, made worse by nondeterminism: sampled completions, flaky tools, clock and RNG drift. You can't reliably reproduce the failure, so you can't reliably fix it. Kinescope is rr for AI agents — it records the nondeterministic frontier of a run (LLM calls, tool calls, the clock, RNG/UUIDs), replays the run deterministically from the trace, and forks it at any step: what if the model had chosen differently here?

The problem

The frontier is small — a handful of places where the same code can produce different results — but it's enough that a failure seen once may never be seen again. Logs tell you what happened; they don't let you re-run what happened, and they can't answer the counterfactual that matters when an agent makes one wrong decision mid-run.

Approach — record outputs, replay them, branch from them

• Interception at the httpx transport — the SDK's public http_client= seam, not method monkeypatching. Captures sync, async, and SSE streaming, and is provider-agnostic by construction: the same engine drives Anthropic, OpenAI, and Gemini (a genuinely different wire shape — model in the URL).
• The rest of the frontier. A @kinescope.tool decorator makes tool calls atomic recorded boundaries; an opt-in scoped patch of time/random/uuid catches the stdlib. Everything funnels into one global sequence.
• Output-authoritative replay. On replay a boundary never calls out — the model is never re-sampled, tools never re-execute, time.time() returns the recorded float. Replay provably never touches the network.
• Counterfactual forks. fork(run_id, at=k, override=...) replays to step k, swaps that one event's output, then switches to live execution for the tail — a linked child run that is itself replayable and forkable.
• Local-first, tiny core. Traces are SQLite plus content-addressed blobs in .kinescope/; the engine depends on httpx only. A Textual timeline TUI scrubs steps, shows per-step state diffs, and forks with one key.

Why it's trustworthy — divergence is surfaced, never hidden

• Input-verified matching. Every boundary's input is canonicalized and hashed at record time; on replay a mismatch — wrong input, wrong order, wrong count — is reported as a divergence (strict raises, warn continues), never silently papered over.
• Stress-tested adversarially. 25 randomly-structured agents replay faithfully; a 10k-event run reproduces every value exactly; deliberately reordered boundaries and hidden nondeterminism are flagged, as documented.
• Reality contact. A genuine recorded Anthropic run is committed as a bundle and replays offline, bit-for-bit, in CI — with credential headers redacted by default before anything hits disk.

The demo — fork-and-fix on a recorded run

An agent misreads a sensor and calls the weather "cold." Open the recorded run in the timeline, scrub to the sensor step, press f, override the reading — the downstream classification re-runs live and flips to "warm" — the whole loop reproducible from the recording. Fork-and-fix on a recorded run is the pitch.

Status & limits

Feature-complete: full-frontier capture, snapshots with per-step diffs, forking, the timeline TUI, a CLI runner (record/replay/fork an agent script), three providers, OpenTelemetry gen_ai.* export, a MongoDB backend, and shareable trace bundles — 62 offline tests, lint/typecheck clean. The limits are stated, not hidden: determinism holds at recorded boundaries; worker-thread boundaries outside the recording context aren't captured (single-thread or asyncio is the supported model); concurrent ordering under asyncio.gather is identical-or-flagged, never silently wrong; streaming replays batched, by content rather than re-timed; and inline events record at ~20k–190k/s while tool/LLM events are bound by blob writes.