From Black-Box Agent Traces to Distillation Data

Cheng Luo — August 2026

What do you do when a strong teacher API can solve software-engineering tasks and expose its tool calls, but does not return chain-of-thought? We built a two-model data pipeline: Claude Fable on Amazon Bedrock executes the actual agent trajectory, then Kimi-K3 writes a short, causal explanation for each already-recorded assistant action. The result is a provenance-rich SFT dataset.

1. Architecture

The two models have deliberately separate jobs. Fable is the behavior teacher: its tool calls, edits, observations and final answer are the trajectory we want to imitate. Kimi is only a rationale annotator. It must never change an action or invent a new one.

2. Collecting the observable trajectory

2.1 Keep credentials out of code and logs

The Bedrock bearer key lives in a private file outside the repository, with mode 0600. The bridge reads the file at startup; the harness itself receives only a dummy local key.

install -m 700 -d ~/.config/agent-distill
read -rsp 'Bedrock key: ' KEY_INPUT; echo
printf '%s' "$KEY_INPUT" > ~/.config/agent-distill/bedrock.key
unset KEY_INPUT
chmod 600 ~/.config/agent-distill/bedrock.key

export AWS_BEARER_TOKEN_BEDROCK_FILE=~/.config/agent-distill/bedrock.key
export BEDROCK_MODEL_ID='us.anthropic.claude-fable-5'
export BEDROCK_ENDPOINT='https://bedrock-runtime.<region>.amazonaws.com'

The local bridge translates the OpenAI chat/tool schema used by the evaluation harness into Bedrock Converse blocks. In particular, assistant tool calls become toolUse, tool observations become toolResult, and the response is converted back without dropping call IDs. A health endpoint validates configuration without spending tokens, while a stats endpoint reports request, retry and token counts without logging prompts or secrets.

2.2 Calibrate before the long run

Agent traces are expensive and heavy-tailed. We used three stages:

1. 2 tasks, concurrency 1: verify tool-call round trips and grader output.
2. 20 tasks, concurrency 2: estimate success rate, latency, token use and disk growth.
3. Timed campaign: only after the first two stages are clean.

uv run eval scaleswe \
  -m claude-fable-5 \
  --env.agent.runtime.type docker \
  --client.base-url http://127.0.0.1:8790/v1 \
  --client.api-key-var LOCAL_DUMMY_KEY \
  --env.agent.max-turns 30 \
  --env.timeout.episode 1800 \
  --sampling.max-tokens 8192 \
  --no-push -n 20 -c 2 -o /data/fable-calibration

We initially discovered extreme context replay: a single agent can resend a very long history dozens of times. A 30-turn cap and a 15–30 minute episode timeout bound the tail without imposing an arbitrary cap on the total campaign size. For the long run we kept Bedrock concurrency at 2; higher concurrency was not useful enough to justify harder rate-limit and cost behavior.

2.3 Preserve attempts, not just successes

Each JSONL line is an envelope with an ID, ok, errors and zero or more traces. A trace contains the message graph, tool definitions, calls, task identity, stop condition and grader reward. We keep all attempt envelopes so infrastructure errors remain auditable. The canonical data snapshot excludes only records with no sampled assistant turn.

3. Adding causal synthetic rationales

For each sampled assistant node, Kimi receives exactly two pieces of information:

1. history_before_action: only the ancestors of that node in the message graph;
2. chosen_action: the assistant content and/or tool call that Fable actually emitted.

It does not receive the subsequent tool result, later assistant actions, hidden test outcome, grader feedback or final answer. That prevents direct future leakage. The prompt also treats repository and tool text as untrusted data, not instructions, and asks for 1–4 compact, decision-relevant sentences.

uv run recover-rationales \
  raw/traces.snapshot.jsonl \
  enriched/traces.with-kimi-rationales.jsonl \
  --include-unsolved \
  --resume \
  --concurrency 8 \
  --max-tokens 500 \
  --timeout-seconds 300 \
  --retries 12

--resume is important: completed envelope IDs are skipped, so the recovery loop is idempotent. Transient 408/429/5xx failures use bounded exponential backoff. Incomplete envelopes go to a separate .failed.jsonl sidecar rather than contaminating the clean output.

4. Integrity and leakage validation

Generating plausible prose is easy; proving that the observable trace was not altered is the real data-engineering work. We apply the following gates:

The future-overlap detector flags long token sequences shared with later observable text. A flagged draft is regenerated without showing Kimi the matching future text; after repeated failure, the trace is rejected by default. We also recommend manually reviewing at least 50 random rows for invented observations, premature knowledge of tests, copied patches and prompt-injection obedience.

5. Deduplication and SFT export

Every minute, the snapshot builder scans raw collection directories, skips malformed trailing lines from an in-progress append, and deduplicates by (task_name, trace_id). It writes to a temporary file and atomically replaces the canonical snapshot, so a reader never sees a partial dataset.

Branching message graphs are expanded root-to-leaf. Each output row has:

{
  "messages":  [... assistant messages include reasoning_content ...],
  "tools":     "[...]",
  "provenance": "{\"episode_id\": ..., \"trace_id\": ...,
                   \"branch_index\": 0, \"reward\": 1.0,
                   \"teacher_model\": \"claude-fable-5\",
                   \"rationale_source\":
                     \"synthetic_rationale_not_teacher_cot\",
                   \"rationale_model\": \"kimi-k3\",
                   \"observable_sha256\": \"...\"}"
}

We retain reward-zero traces in the canonical enriched corpus because they are useful for error analysis and controlled ablations. Export policy is explicit: --min-reward 1 yields positive-only SFT, while --min-reward 0 yields an all-rewards dataset. Never let a filename silently decide this policy.

6. Running for 24 hours without babysitting

A reliable long run needs a wall-clock deadline, not “collect N” alone. Our wrapper persists the start and deadline timestamps once, resumes the same output directory after interruption, and uses a foreground timeout with a short TERM grace period. A separate recovery worker polls every 60 seconds, rebuilds the canonical snapshot, and annotates only new envelope IDs.

collector (Bedrock concurrency 2)
  -> append raw trace
  -> update attempts/success/error counters

every 60 seconds:
  -> atomic deduplicated snapshot
  -> recover unseen traces (Kimi concurrency 8)
  -> validate and append clean enriched rows
  -> record failures separately

at deadline:
  -> terminate collector gracefully
  -> one final snapshot/recovery pass
  -> validate full corpus
  -> export JSONL + Parquet + manifest

The monitor checks more than “is the service active?” Temporary systemd units can disappear while child processes keep running. We cross-check fixed PIDs, file modification times, result counts, recent rollout logs, active Docker containers, network activity, disk space and restart counters.

At one live checkpoint during the campaign, the collector had written 1,072 attempt envelopes: 757 completed successfully and 315 did not. The global canonical corpus contained 956 unique traces with assistant turns, and the recovery worker had annotated all of them with zero failed envelopes. These are operational checkpoint numbers, not a final benchmark claim; collection continued until the persisted deadline.

7. What mattered most

1. Separate behavior from explanation. Fable's actions are authoritative; Kimi's rationale is annotation.
2. Never call synthetic rationale “teacher CoT.” Provenance must survive every export.
3. Make recovery causal by construction. Prompt wording alone is not a leakage guarantee.
4. Keep raw data immutable. Enrichment always writes a new file.
5. Count attempts, valid traces and canonical rows separately. Conflating them hides failures and duplicates.
6. Resume at every layer. Collector, snapshot builder and rationale worker should all be idempotent.
7. Bound the tail. Turns, episode time, generation size, retries and concurrency all need explicit caps.
8. Validate semantics, not just JSON. Tool closure and future leakage matter more than parse success.
9. Freeze datasets for training. A live corpus can keep growing; a model run must point to an immutable snapshot.
10. Publish enough provenance to audit the claim. Model names, reward, IDs, branch and observable hash belong in every row.

This design does not recover private reasoning. It does something more defensible: it turns a black-box, tool-using teacher's observable behavior into a reproducible, auditable training corpus, while making the synthetic part impossible to mistake for the original.

Implementation: bedrock-kimi-distill on GitHub.