K3 → Qwen3-8B Distillation: The Complete Runbook

Cheng Luo — August 2026

The companion results post covers what we found (base ~1% → distilled ~25% on held-out agentic SWE, saturating at ~450 traces). This page is the how: the end-to-end pipeline, every config that mattered, and the 12 failure modes that cost us real time and money — so you can reproduce the result without re-paying the tuition.

Contents
0. Architecture overview · 1. Node setup · 2. Trace collection · 3. Filtering · 4. SFT training · 5. Serving & evaluation · 6. Results & scaling law · 7. The 12 pitfalls

0Architecture overview

[OpenRouter keys ×3] ↓ or_proxy.py :8787 ---- round-robin / retry upstream 5xx / cooldown 429 / drop dead keys ↓ eval scaleswe_v1 -m kimi-k3 ---- K3 solves real GitHub PRs as an agent in docker (auto_collect.sh gating) bash+edit tools, multi-turn, grader-scored → traces.jsonl ↓ keep reward==1 only conv_traces_to_sft.py ---- {messages, tools} parquet, reasoning_content preserved ↓ uv run sft (prime-rl) ---- Qwen3-8B full-param FSDP, seq 49k, assistant-only loss ↓ [ckpt] interval=45 weights_only=true ← survives GPU reclaims vllm serve ×2 (distilled + base) ---- hermes tool parser + qwen3 reasoning parser ↓ eval (scaleswe deduped held-out + swebench cross-benchmark) → analyze_eval_hash.py ↓ HF dataset release

Stack: Prime Intellect prime-rl + research-environments; teacher = moonshotai/kimi-k3 (OpenRouter, $3/M in, $15/M out); student = Qwen/Qwen3-8B; tasks = PrimeIntellect/Scale-SWE-Verified (17,202 real GitHub PRs, each shipping a docker image + hidden fail-to-pass tests).

1Node setup

# prime-rl source (36MB, skip .venv/.git)
rsync -a --exclude=.venv --exclude=.git <src>:~/kimi-distill/prime-rl/ ~/kimi-distill/prime-rl/

uv self update                                # needs uv >= 0.11.1
cd ~/kimi-distill/prime-rl
uv sync --extra flash-attn                    # prebuilt wheel; NEVER a source build
.venv/bin/python patch_tf_shim.py             # transformers-5.x / ring_flash_attn fix
.venv/bin/python -c "from huggingface_hub import snapshot_download; \
    snapshot_download('Qwen/Qwen3-8B', ignore_patterns=['original/*'])"   # 16G
.venv/bin/python -c "from prime_rl.trainer.model import pre_download_model; print('OK')"

The collection node additionally needs docker, the scaleswe_v1 env package (uv pip install -e environments/swe/scaleswe_v1, plus the verifiers[harbor] extra), and a lot of disk — task images are ~1.3 GB each and pile up into the terabytes.

2Trace collection

2.1 The three-key proxy (start this first)

cd ~/kimi-distill/or-proxy        # keys.txt: one sk-or-v1-... per line, chmod 600
setsid nohup uv run or_proxy.py --port 8787 > proxy.log 2>&1 </dev/null & disown
curl http://127.0.0.1:8787/healthz    # {"ok":true,"live":3}
curl http://127.0.0.1:8787/stats      # per-key spend / errors

What it does: rotate keys per request, remap kimi-k3 → moonshotai/kimi-k3, mark 401/402 keys dead, cool down 429s, and — critically — retry upstream 5xx and "HTTP 200 but the body is an error" across keys with exponential backoff. Moonshot's gateway degrades nightly under heavy concurrent agentic load; without this retry layer a whole run dies of 503s.

2.2 The collection command

cd ~/kimi-distill/research-environments
export OR_PROXY_KEY=dummy             # proxy injects real keys
uv run --no-sync eval scaleswe_v1 \
  -m kimi-k3 \
  --env.agent.runtime.type docker \   # REQUIRED: default subprocess can't run task images
  --client.base-url http://127.0.0.1:8787/v1 \
  --client.api-key-var OR_PROXY_KEY \
  --env.timeout.episode 900 \         # 15-min cap: a hung rollout stops burning money
  --sampling.max-tokens 8192 \
  -s True -n 3000 -c 12 \             # shuffled sampling; keep concurrency ≤ 12-16
  --no-push -o ~/kimi-distill/traces/k3-run1

Interrupted? eval --resume <dir> is idempotent — it re-runs only failed or unattempted tasks. (It takes no other flags; to change concurrency, edit max_concurrent in the saved config.toml.)

2.3 Unattended: the self-gating auto-collector

setsid nohup bash auto_collect.sh >/dev/null 2>&1 & disown
tail -f ~/kimi-distill/auto_collect.log
# [08-09 05:48] healthy 6/6 $834 usable=834 (running)
# [08-06 02:20] degraded 2/6 $615 usable=232 -> PAUSED

Every 15 minutes it fires 6 concurrent heavy probes at K3. ≥5/6 succeed → collect; fewer → pause (and stop spending). It also stops entirely when the key budget drops below a floor. Paired with disk_guard.sh (auto-docker image prune when free disk < 300G), this ran overnight unattended and nearly doubled the dataset.

2.4 Collection economics (measured)

Metric	Value
K3 pass@1 on scaleswe	~64%
Cost per usable (reward==1) trace	~$1.5–1.7
Per trace	median ~25k tokens, ~30 assistant turns, ~40 tool calls
Full 993-trace dataset	~26.6M tokens (~0.03B)

3Filter → SFT dataset

uv run --no-sync python conv_traces_to_sft.py \
  ~/kimi-distill/traces/k3-run1 --min-reward 1.0 \
  -o ~/kimi-distill/sft_data/k3_ours_993

Keeps only rewards.solved.score==1, completed, non-errored traces; emits a {messages, tools} parquet with K3's reasoning_content and tool_calls preserved. The grader is a free, perfect quality filter — every surviving trace is a verified-correct solution.

4SFT training

max_steps = 372          # ceil(993/8) * 3 epochs

[deployment]
num_gpus = 4             # batch_size must divide num_gpus (8%4=0 OK; 8%6 crashes)

[ckpt]
interval = 45            # save often...
weights_only = true      # ...and light: survives mid-write GPU reclaims

[model]
name = "Qwen/Qwen3-8B"
seq_len = 49152          # traces are long: median 25k, max 290k tokens
attn = "flash_attention_2"

[model.ac]
mode = "full"            # full activation checkpointing for the long context

[data]
name = ".../sft_data/k3_ours_993"
seq_len = 49152
batch_size = 8
micro_batch_size = 1

[renderer]
name = "qwen3"
enable_thinking = true   # render reasoning_content as <think> -- the CoT is trained

[optim]
lr = 1e-5                # AdamW, constant schedule

setsid nohup env CUDA_VISIBLE_DEVICES=2,3,4,5 \
  uv run --no-sync sft @ sft_ours993.toml --output-dir sft-out/ours993 \
  > sft.log 2>&1 </dev/null & disown

Full-parameter fine-tune (not LoRA), loss on assistant turns only. Measured: 4×H100, ~36s/step, ~4h for 3 epochs over 993 traces; loss 1.25 → 0.38. Output (weights/step_372/) is a directly-servable HF checkpoint.

The checkpoint config is the survival kit. Our shared cluster SIGTERM-reclaims GPUs at random; a full checkpoint takes minutes to gather and one reclaim mid-write corrupted a finished 3.5-hour run (metadata is None, unrecoverable). Frequent weights_only saves are fast and atomic-ish — a reclaim now costs at most 45 steps.

5Serving & evaluation

# distilled on GPUs 2,3 / base on 4,5 -- one command each
vllm serve sft-out/ours993/weights/step_372 \
  --served-model-name qwen3-ours993 --port 8100 --tensor-parallel-size 2 \
  --max-model-len 40960 \
  --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3

# same first-N tasks for both models (-s False), LOW concurrency
bash eval_student.sh scaleswe_v1          qwen3-base    http://<node>:8101/v1 ev-scale-base 200 6
bash eval_student.sh scaleswe_v1          qwen3-ours993 http://<node>:8100/v1 ev-scale-dist 200 6
bash eval_student.sh swebench_verified_v1 qwen3-base    http://<node>:8101/v1 ev-sweb-base  100 6
bash eval_student.sh swebench_verified_v1 qwen3-ours993 http://<node>:8100/v1 ev-sweb-dist  100 6

Decontamination: hash every training task's problem statement (md5 of the first user message), and exclude any eval task whose hash matches — analyze_eval_hash.py reports held-out-only pass@1. For strict model-vs-model comparisons, score on the intersection of tasks both evals completed (the base model's 1/137 = 0.7% was identical across runs — a good sanity check that the harness is stable).

Concurrency 6, not 12: two dozen concurrent 40k-token rollouts overflow the KV cache and the server starts returning 500s, silently poisoning the eval.

6Results & scaling law

# traces	scaleswe held-out (distilled vs base)	swebench_verified	note
31	~0% — no effect	—	pipeline validation only
163	5.0% vs 0.7% (~7×)	7.9% vs 4.3%
360	11.8% vs 0.9% (~13×, p<0.001)	4.3% vs 2.1%	mixed sources
454	25.4% vs 0.7%	—	self-collected
993	25.0% (45/180) vs 1.1% (2/186) — ~23×	9.2% vs 4.3%	self-collected, final

Saturation at ~450 traces (~0.012B tokens). On the 137 held-out tasks common to both evals: base 0.7%, distilled-454 25.0%, distilled-993 27.6% — statistically flat. ~0.03B tokens of grader-verified agentic traces lift an 8B model from ~1% to ~25%; beyond a few hundred traces the bottleneck is the student's capacity, not the data.

7The 12 pitfalls, ranked by damage

#	Pitfall	Symptom	Fix
1	Disk filled by docker images	evals mass-error / crawl; `No space left on device`	`disk_guard.sh`: auto-prune below 300G free
2	Cluster reclaims GPUs (SIGTERM)	random job death; checkpoint corrupted mid-write	`interval=45 weights_only=true`; serve-watchdog loops
3	Moonshot upstream degrades nightly	503/504 storms under concurrent agentic load (c=40 → 68% errors, $293 wasted)	proxy retries + c≤16 + health-gated auto-collector
4	KV-cache OOM while serving	vLLM 500s at eval c=12	eval concurrency c=6
5	transformers 5.6.2 vs ring_flash_attn 0.1.8	fake `No module named flash_attn`	`patch_tf_shim.py` re-adds the removed symbol
6	Model defaults to FlashAttention-3	`FA3 ... not installed` at setup	`attn="flash_attention_2"`
7	Batch divisibility	`batch_size*cp (8) must be divisible by world_size (6)`	num_gpus ∈ {1,2,4,8}
8	Default subprocess runtime	`task requires image ... use docker runtime`	`--env.agent.runtime.type docker`
9	pkill self-match	kill your own SSH session instead of the target	`[e]val`-style patterns, or kill by nvidia-smi PID
10	uv venv races	flash_attn "disappears" after concurrent uv runs	one uv command per venv at a time
11	zsh word-splitting	multi-node for-loop runs once on the whole string	wrap in `bash -c '...'`
12	`--resume` takes no flags	`--resume re-runs a saved config verbatim`	edit the saved `config.toml` instead

Artifacts

Dataset: Linz99/kimi-k3-scaleswe-distill-traces — sft-ours-993/ (final), sft-ours-454/, sft-718/, sft/ (360), raw/
Results post: Distilling Kimi-K3 into Qwen3-8B
Scripts: or_proxy.py, auto_collect.sh, disk_guard.sh, conv_traces_to_sft.py, eval_student.sh, analyze_eval_hash.py, patch_tf_shim.py