Crucible — turning test-time compute into measured accuracy (or an honest negative)

Reasoning models are the moment, but most people only consume them. Crucible builds the machinery underneath: a frozen small open model proposes reasoning, a verifier judges it, and search spends inference-time compute on the traces the verifier trusts. The deliverable is not a leaderboard number — it is a curve: accuracy as a function of test-time compute, verifier compute included.

The problem

"Sample more and pick the best" is easy to claim and easy to fake: comparisons run at mismatched compute, the verifier's forward passes go uncounted, one lucky seed gets reported. The question worth answering is quantitative — how much accuracy does a unit of extra inference compute actually buy, and through which search? Most hobby versions of this die at the measurement.

Approach — one interface, the whole search ladder

Four strategies drive one policy/verifier/search interface, simplest first:

1. Majority@N — verifier-free self-consistency, the control line.
2. Best-of-N — sample N traces, pick by outcome verifier or PRM score.
3. PRM-guided beam — expand promising partial reasoning, prune step by step.
4. MCTS over steps — PUCT with the PRM as value; the most expressive and expensive.

Two verifier families back them: programmatic — symbolic math equivalence (1/2 ≡ 0.5, not string match) and sandboxed code execution against unit tests, off by default — and learned, an open process-reward model scoring each step. Compute accounting is a first-class value type: every sampled token and every PRM forward pass lands on the x-axis, with Wilson CIs over 3 seeds.

The measurement — negatives included

On real MATH-500 (3 seeds, a frozen 1.5B instruct policy plus a real 1.5B PRM):

• Search converts compute into accuracy: pass@1 38% → 70% with an oracle verifier at N=8.
• The learned PRM beats majority voting at every N on identical samples (N=4: 53% vs 45%) — and its ~2× scoring compute makes that roughly a wash at matched tokens. Both facts are in the report.
• The negatives, kept: a 7B baseline (67.5% pass@1) is more compute-efficient than 1.5B + search — small-beats-big does not hold on this stack — and real beam/MCTS on the hardest problems are the most expensive and don't win: a non-reasoning policy restarts rather than continues a partial trace. Those roadmap boxes stay open.

Why it's trustworthy — curves you can re-run

• Every headline number replays offline. Real GPU runs are recorded to cassettes (traces, PRM scores, correctness); CI regenerates the headline curve with no GPU, and a replay miss raises instead of silently zeroing.
• The compute axis is honest. The verifier's tokens are charged to its own line, so the PRM's curve sits visibly to the right of majority's.
• Adversarially hardened. A multi-agent audit found 8 real bugs (3 high) that only manifest on the real-model path — beam dropping completed answers, a sandbox clean-exit reward hack — all fixed with regression tests; 162 tests green.
• A milestone is checked only against a real captured artifact — synthetic runs are demoted to mechanism validation.

Status & what's next

M0–M7 built; the real 3-seed lift curves are captured. Deliberately out of scope: no training or fine-tuning (frozen policies, off-the-shelf PRMs), and the code sandbox is a guardrail, not a jail (isolated subprocess, not a container). Open and stated as such: beam/MCTS winning on real data needs a reasoning policy, and the small-beats-big headline needs a stronger, family-matched PRM.