[✳] CADOOpen source ↗
FOR THE SCIENCE. AND THE COMPUTE BILL.

cado

Good runs shouldn’t
depend on good luck.

Catch the expensive mistakes ↘
Open instruments for
reinforcement learning.Scroll to explore ↓
Bold, grainy avocado and clover forms overlap in lime, cream and dusty blue against deep olive
CADO / FIELD STUDY 001LESS NOISE. MORE SIGNAL.
[01] / PhilosophyA simple conviction.

Keep the signal.
Question the noise.

Don't get your hopes up or loose money on mid-way OOMs and false reward signals. We build open instruments that separate what your model did from what its environment brokes and lets you know ASAP programatically.

Keep the science honest ↗
[02] / Time, cost & proofThe expensive part is not knowing.

Bad feedback.
Real money.

A timeout can look like a wrong answer. A broken verifier can look like a bad policy. By the time the reward curve tells you, you’ve already paid for the steps.

Sandbox failures~16%

of audited early trajectories contained a sandbox-caused failure in KAT-Coder’s training.

Timeouts alone6–7%

of rollouts were invalidated by timeouts before the team fixed image management.

After infrastructure fixes<2%

sandbox feedback error rate, down from roughly 16%. The underlying problems were fixable.

One team’s reported rollout statistics, not an industry-wide percentage of failed runs. The improvements came from the KAT team’s fixes, not Cado. KAT-Coder-V2.5 report, §4.2.2 ↗

Notice. Investigate. Intervene.

Catch the change
in your terminal.

Cado’s tp integrity reads recorded outcomes and flags changes in failure rate, reward, and rollout–trainer mismatch. Run it periodically during training; it exits with code 1 when an alarm fires.

1 stepmedian alarm delay after typed failures rise to 20%, in 50 synthetic runs.

Synthetic validation, not a production guarantee. At a 5% failure rate, 43 of 50 runs had no alarm within 150 steps. See the full sensitivity results ↗

tp integrityActual output · synthetic fixture
$ tp integrity runs/demo
# Alarm excerpt; window table omitted

failure_rate  swe  u-04895  P=1.0000
infra or grader failures 8 of 32 units in step 152;
healthy rate 0.02, broken 0.16

$ echo $?
1

Fault injected at step 150 → alarm at step 152 in this example. Alarm line wrapped for readability.

Your next move

Inspect the failing source. Fix the environment, apply masking in your trainer, or pause and resume from your own checkpoint.

What earlier intervention could be worth

What could an
earlier alert save?

Estimate avoidable GPU spend for one incident, assuming the whole allocation would otherwise keep doing unusable work. Change the inputs to match your run.

Scenario estimate · not measured savings
Potential GPU spend avoided / incident$1,120

64 GPUs × $2.50/hour × 7 hours of avoided waste

Without an alert8 h
Alert + recovery1 h

Calculation: GPUs × hourly rate × max(0, hours without alert − hours to alert − recovery hours). The $2.50 rate and timing defaults are editable assumptions, not a cloud quote or measured Cado performance. This is gross compute value, excluding labor, storage, monitoring overhead, and earlier lost work. Realized cash savings require avoiding billed time; prepaid capacity may instead recover useful compute. Cado does not automatically restore checkpoints or guarantee a rescued run.

THE COMPARISON / RFT-FAULTBENCHMeasured, with the limits included.

Show the work.

Whole-run fault detection on RFT-FaultBench.
Measured results. Including the awkward parts.

Average F1 / 5 fault families
What this benchmark does — and doesn’t — tell us

This measures faults present from the first step, not faults that appear mid-run. Protocol A uses the paper’s labels. Protocol B treats 224 un-injected controls as normal; tplane scores 85.58 hard F1 with 4 / 235 false alarms.

Results use outer five-fold cross-validation on 779 runs. Thresholds are selected within each training fold, but prior feature exploration used the same outer folds and introduces selection bias. This is early evidence, not a production reliability claim.

Source: local evaluation dated September 24, 2026 ↗
[03] / How to useSmall enough to start today.

One run.
A clearer picture.

Start locally with tplane. Add a record around your rollout, then inspect what happened.

  1. 01

    Install from source

    Python 3.11–3.12. No service to deploy.

  2. 02

    Wrap a unit of work

    Record a reward, or a failure with a cause.

  3. 03

    Inspect your run

    Read the record. Apply the decision in your trainer.

Explore the SDK documentation
01 / Terminal
git clone https://github.com/ilovehhhyn/cado.git
cd cado/tplane
python -m pip install -e .
02 / example.py
from pathlib import Path
import tplane

run = tplane.Run(
    run_id="first-run",
    store=tplane.FileRecordStore(Path("runs/first")),
    # Demo only: resource sampling disabled.
    read_host=lambda: 0,
    read_gpu=lambda: (),
    gpu_total_bytes=(),
)

def rollout(unit):
    # Replace with your generation and grading.
    with unit.stage(tplane.Stage.ROLLOUT):
        unit.tokens(tokens_in=12, tokens_out=48)
    with unit.stage(tplane.Stage.GRADE):
        unit.reward(1.0)

result = run.attempt(
    "trajectory-001",
    kind=tplane.UnitKind.TRAJECTORY,
    body=rollout,
)
print(result.record.decision.action)
03 / Terminal
python example.py
tp show runs/first

Use a fresh run directory when repeating this example.
Already-recorded attempts are intentionally never overwritten.