How far can you push a small model as an agentic coder by learning from a giant's grader-verified traces? We distilled Kimi K3's software-engineering rollouts into Qwen3-8B, watched the gain scale cleanly with data — and hit both a data plateau and a genuinely hostile compute cluster along the way.
Scale-SWE-Verified) as a bash+edit agent. Student = Qwen3-8B. Method = offline SFT on only the grader-verified (reward==1) traces. On held-out, contamination-controlled tasks the distilled model beats base ~13–36× in-distribution (11.8% vs 0.9%; and up to 25.4% vs 0.7% on a fully self-collected run), scaling monotonically with data and saturating around ~360 traces. The direction holds on an independent benchmark (swebench_verified), and reproduces with zero dependence on external data.
K3 is a black-box API — no logits — and at ~1.4 TB it can't be self-hosted on the GPUs we had, so white-box on-policy distillation is out. The offline route is simple and robust:
bash + edit tools), producing full multi-turn rollouts with its chain-of-thought.<think> via the Qwen3 renderer).Teacher and student share the exact system prompt and tool schema, so there is no harness-mismatch confound — the student learns to imitate K3 in the same interface it's later judged in.
eval scaleswe_v1 -m kimi-k3 through the proxy, docker runtime, episode-timeout + max_tokens guards. Filter reward==1 → a messages+tools parquet.sft, Qwen3-8B, FSDP across 8× H100, seq_len 49k (agentic SWE traces are long — median ~24k tokens), full activation checkpointing.The full loop, run at four data scales. In-distribution pass@1 (held-out, dedup'd against trained tasks):
| # traces | distilled vs base (scaleswe held-out) | verdict |
|---|---|---|
| 31 | ~0% vs ~4% | no effect — pure noise |
| 163 | 5.0% vs 0.7% | ~7× |
| 360 | 11.8% (13/110) vs 0.9% (1/115) | ~13×, p<0.001 |
| 718 | ~8–10% vs ~2% | plateau |
| 454 (self-collected only) | 25.4% (36/142) vs 0.7% (1/141) | ~36×, large sample |
The 360-trace run mixed in traces from a parallel collector, so as a clean-provenance check we re-ran the whole loop on 454 traces we collected entirely ourselves (all verified kimi-k3; the two sources are statistically indistinguishable in reasoning density, trace length, and tool-call count). On a full n=142 held-out sample it gave the strongest, cleanest result of the project: 36 solved vs the base model's 1. The conclusion holds — if anything more strongly — with zero dependence on external data.
Two things make the 360 result trustworthy rather than a lucky benchmark:
swebench_verified — a separate benchmark with no task overlap and no contamination — the distilled model still leads (7.9% vs 4.3% at 163; 4.3% vs 2.1% at 360). Same direction, independent data.And then it plateaus. Pushing to 718 traces (407 distinct tasks, ~2× the data) did not clearly beat 360 across three eval attempts. More agentic traces beyond a few hundred bought no further generalization here.
bash, read output, edit, iterate — and that transfers to a real, significant capability gain. Beyond that, the bottleneck stops being "has it seen enough good trajectories" and becomes the 8B model's own ceiling.
Honestly, the model work was the easy part. The hard part was the environment.
Kimi's upstream is diurnal. Single calls always worked, but heavy concurrent agentic requests (20k-token prompts, long reasoning) hit gateway 503/504s during Moonshot's night. Our first high-concurrency run was a disaster: 68% of rollouts errored, ~$293 for 76 usable traces. The clean fix was a self-gating auto-collector that probes K3 health every 15 minutes and only collects during healthy windows, pausing itself (and the spend) when the upstream degrades.
The GPU cluster reclaims aggressively. Long training and serving jobs got SIGTERM'd within tens of minutes, seemingly at random — corrupting checkpoints mid-write (AssertionError: metadata is None). We lost a perfect 3.5-hour run to a reclaim during the final save. The fix that finally worked:
[ckpt]
interval = 45 # checkpoint frequently
weights_only = true # skip the heavy optimizer-state gather -> fast, atomic-ish writesDistilling Kimi K3's grader-verified agentic SWE traces makes Qwen3-8B a meaningfully, statistically-significantly better software-engineering agent — about 13× in-distribution pass@1 over the base model — with the gain scaling cleanly from a handful of traces up to ~360, then saturating. The teacher's API and the compute cluster fought us the whole way; the winning move was building infrastructure resilient enough to let the (surprisingly data-efficient) science come through.