Distilling Agentic Software Engineering into Qwen3-8B

Two trace pipelines, one end-to-end study: what transfers from Kimi-K3's grader-verified reasoning-and-action trajectories, what changes when Claude Fable supplies only observable behavior, and how we made the whole data → SFT → evaluation loop reproducible.

Cheng Luo — August 2026

1The question and the two tracks

Software-engineering agents do not answer once. They inspect a repository, form a plan, call bash, read the output, edit files, run tests, revise the patch and only then produce a final answer. Distilling this behavior is therefore different from distilling a static instruction-response dataset. The object being transferred is a long, tool-coupled policy.

We asked how much of that policy could be transferred into Qwen3-8B with ordinary offline supervised fine-tuning. Because the strongest teachers were black-box APIs, logits and on-policy white-box distillation were unavailable. Full trajectories were the practical interface.

2End-to-end architecture

Both tracks converge on the same training interface: message sequences, tool definitions and assistant reasoning/action turns. That common interface is useful, but it can hide a semantic difference. Track A's reasoning belongs to the model that chose the action. Track B's rationale is an annotation of an action chosen by a different model. Preserving that distinction in every exported row is a first-class data requirement.

3Track A: direct Kimi-K3 traces

3.1 Collection and filtering

Kimi-K3 runs as a bash+edit agent over real GitHub pull-request tasks. The task harness executes its patch inside the task image and hidden tests produce a binary solved reward. The retained data is therefore selected by the final software outcome rather than by a separate reward model.

uv run --no-sync eval scaleswe_v1 \
  -m kimi-k3 \
  --env.agent.runtime.type docker \
  --client.base-url http://127.0.0.1:8787/v1 \
  --client.api-key-var OR_PROXY_KEY \
  --env.timeout.episode 900 \
  --sampling.max-tokens 8192 \
  -s True -n 3000 -c 12 \
  --no-push -o traces/k3-run1

uv run --no-sync python conv_traces_to_sft.py \
  traces/k3-run1 --min-reward 1.0 \
  -o sft_data/k3_ours_993

The conversion keeps completed, non-error traces with rewards.solved.score == 1, preserves tool calls and renders the returned reasoning_content through Qwen3's thinking format. The final 993-trace dataset contains about 26.6M tokens. A typical trace was long: roughly 25k tokens, 30 assistant turns and 40 tool calls at the median.

3.2 The scaling curve

The 454-trace run matters because it removed a provenance concern in an earlier mixed-source dataset. Every retained trajectory in that run was collected by the same pipeline and verified as Kimi-K3. Its 36 solved tasks against the base model's 1 show that the result did not depend on external trace contributions.

4Track B: Claude Fable behavior plus synthetic rationale

4.1 Bridging the agent harness to Bedrock

The evaluation harness speaks an OpenAI-compatible message and tool schema. A local bridge converts assistant tool calls to Bedrock toolUse blocks, observations to toolResult blocks, and converts Fable's response back without losing call IDs. The behavior trace—actions, edits, observations and final answer—is appended to immutable JSONL before any rationale is added.

scripts/start_bedrock_proxy.sh

uv run eval scaleswe \
  -m claude-fable-5 \
  --env.agent.runtime.type docker \
  --client.base-url http://127.0.0.1:8790/v1 \
  --client.api-key-var LOCAL_DUMMY_KEY \
  --env.agent.max-turns 30 \
  --env.timeout.episode 1800 \
  --sampling.max-tokens 8192 \
  --no-push -n 20 -c 2 -o runs/fable-calibration

We calibrated at 2 tasks with concurrency 1, then 20 tasks with concurrency 2, before a timed campaign. The turn and episode caps are important: each agent turn replays a growing context, so an apparently ordinary trajectory can develop a very expensive token tail.

4.2 Reconstructing a rationale without looking into the future

For assistant node i, the annotator receives only:

1. the root-to-node ancestor history before node i;
2. the action at node i that Fable already chose;
3. the available tool specifications.

It does not receive later tool output, later actions, the final answer, hidden test results or grader feedback. The request asks for one to four decision-relevant sentences explaining why the recorded action is a reasonable next move. The annotation cannot change that action.

uv run recover-rationales \
  raw/traces.snapshot.jsonl \
  enriched/traces.with-kimi-rationales.jsonl \
  --include-unsolved --resume \
  --concurrency 8 --max-tokens 500 \
  --timeout-seconds 300 --retries 12

uv run validate-distill-traces \
  enriched/traces.with-kimi-rationales.jsonl \
  --source raw/traces.snapshot.jsonl \
  --report enriched/validation-report.json \
  --include-unsolved

uv run export-distill-sft \
  enriched/traces.with-kimi-rationales.jsonl \
  dataset/sft-all-rewards --parquet --include-unsolved

The frozen Kimi-annotated snapshot contained 829 deduplicated, fully recovered rows: 689 reward-positive and 140 reward-zero. Unlike Track A, the final Track B SFT run included both outcomes. This is useful for studying observable behavior, but it also means that a direct numerical comparison between tracks mixes rationale provenance with training-data selection.

4.3 The result: in-distribution transfer, weak generalization

A plausible interpretation is that the post-hoc rationales taught longer or more brittle deliberation patterns aligned with Scale-SWE, increasing rollout length without reliably improving cross-repository generalization. That is an inference from the pass rates and timeout counts, not a proven causal mechanism. A controlled ablation would hold behavior traces, reward policy, token budget and training exposure fixed while changing only the rationale source.

5Causal integrity and provenance

Plausible prose is easy to generate. Proving that the new prose did not rewrite the behavior trace or leak the future is the substantive data-engineering problem.

If a rationale overlaps later text, it is regenerated with a more abstract instruction that still does not reveal the matching future content. After repeated failure, the trace is rejected. Raw traces remain immutable; enrichment always writes a new file. Snapshot construction writes atomically and deduplicates by task and trace identity, so an interrupted append cannot expose a partial training corpus.

{
  "messages": [...],
  "tools": "[...]",
  "provenance": {
    "episode_id": "...",
    "trace_id": "...",
    "branch_index": 0,
    "reward": 1.0,
    "teacher_model": "claude-fable-5",
    "rationale_source": "synthetic_rationale_not_teacher_cot",
    "rationale_model": "kimi-k3",
    "observable_sha256": "..."
  }
}

6The SFT recipe

Both tracks used full-parameter Qwen3-8B SFT with assistant-only loss. Long agent traces drove nearly every systems choice: a 49,152-token training sequence, micro-batch size 1, full activation checkpointing and FlashAttention-2. Qwen3's renderer preserved the reasoning field as thinking content while tool calls stayed in the same schema used at evaluation time.

max_steps = 372

[deployment]
num_gpus = 4

[ckpt]
interval = 45
weights_only = true

[model]
name = "Qwen/Qwen3-8B"
seq_len = 49152
attn = "flash_attention_2"

[model.ac]
mode = "full"

[data]
name = "sft_data/k3_ours_993"
seq_len = 49152
batch_size = 8
micro_batch_size = 1

[renderer]
name = "qwen3"
enable_thinking = true

[optim]
lr = 1e-5

The final direct-Kimi run used three epochs: 372 optimizer steps on four H100s, roughly 36 seconds per step and about four hours total. Training loss fell from 1.25 to 0.38. On an aggressively reclaimed cluster, frequent weight-only checkpoints were the difference between a recoverable run and losing hours to a checkpoint interrupted mid-write.

7Evaluation without fooling ourselves

7.1 Same harness, paired tasks

Base and distilled models are served with the same context limit, tool-call parser, reasoning parser, system prompt and Docker task runtime. Sampling is unshuffled so both models receive the same first N tasks. The strict comparison uses only the task intersection that both runs completed.

vllm serve weights/step_372 \
  --served-model-name qwen3-distilled \
  --tensor-parallel-size 2 \
  --max-model-len 40960 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3

bash eval_student.sh scaleswe_v1 qwen3-base \
  http://server:8101/v1 eval/scale-base 200 6
bash eval_student.sh scaleswe_v1 qwen3-distilled \
  http://server:8100/v1 eval/scale-distilled 200 6

7.2 Decontamination

Every training task's problem statement is hashed. An evaluation task with a matching hash is excluded before pass@1 is reported. Scale-SWE measures held-out in-distribution transfer; SWE-bench Verified provides a second benchmark with no task overlap. Neither alone is enough: the first can reward narrow adaptation, while the second may be noisier or differently distributed.

7.3 Reliability is part of the metric

A model that solves more completed episodes by thinking longer can still be worse operationally if it doubles the timeout rate. We therefore report graded tasks, solves and failure categories, not just the percentage among successful completions. This is especially important for Track B, where synthetic rationale students timed out much more often.

8What broke in production

The ML recipe was short. The campaign survived because the surrounding system treated every external dependency as unreliable.

The unattended collectors used persisted wall-clock deadlines, append-only logs and idempotent workers. Every minute, a snapshot worker rebuilt the canonical corpus; rationale recovery skipped completed envelope IDs; failed annotations went to a sidecar; and a final pass froze a manifest, JSONL and Parquet dataset for training.

9What actually transfers

1. Verified trajectories can teach the shape of agent work. A few hundred successful examples were enough for Qwen3-8B to learn a substantially better loop of inspect, reason, call tools, edit and test.
2. Quality filtering beats raw volume. Direct Kimi distillation improved rapidly on grader-positive traces and then saturated; doubling data after the plateau did not double capability.
3. Reasoning provenance matters. A rationale written after an action is not semantically equivalent to the reasoning supplied by the model that chose the action, even if both fit into the same JSON field.
4. In-distribution gain is not generalization. Fable+Kimi improved on strict held-out Scale-SWE but regressed on SWE-bench and reliability. The cross-benchmark result changes the conclusion.
5. Timeouts are model behavior. Longer synthetic reasoning can change completion rates enough that pass@1 without failure counts becomes misleading.
6. Provenance should be impossible to drop accidentally. Teacher, annotator, reward policy, IDs and observable hash belong in every exported row, not only in a dataset card.
7. Infrastructure determines whether the science is observable. Health gates, atomic snapshots, resumable enrichment, disk guards and frequent checkpoints were experimental controls, not mere deployment polish.

10Compact reproduction recipe

1. Install the harness and training stack. Use Docker-capable Scale-SWE, Prime-RL, Qwen3-8B, FlashAttention-2 and enough disk for task images.
2. Calibrate teacher access. Run 2 tasks at concurrency 1, then 20 at concurrency 2. Measure success, token tail, latency, disk growth and cost before scaling.
3. Collect immutable traces. Store attempts and errors, not just successes. Bound turns, episode time, output tokens and concurrency.
4. Choose an explicit data policy. Track A uses completed reward-positive traces. If unsolved traces are included, record that choice in the manifest and training config.
5. If reasoning is synthetic, make it causal by construction. Pass only the ancestor prefix and chosen action; hash observables; check tool closure and future overlap.
6. Freeze and deduplicate. Export a versioned JSONL/Parquet snapshot with full provenance. Never train directly from a live, growing campaign directory.
7. Train assistant-only at the real trace length. Preserve tool and reasoning formats, use activation checkpointing, and save often enough for the cluster's reclaim behavior.
8. Evaluate base and distilled on paired tasks. Decontaminate against every training problem, report failures and timeouts, and add a no-overlap second benchmark.
9. Measure a scaling curve. A pilot validates plumbing; it does not establish a capability result. Stop buying data when additional scale becomes statistically flat.

Artifacts

• Causal CoT Completion: prefix-only synthetic rationale walkthrough and optional live BYOK annotator.
• bedrock-kimi-distill: Bedrock bridge, causal rationale recovery, validation, SFT export and evaluation helpers.
• Linz99/kimi-k3-scaleswe-distill-traces: direct Kimi-K3 training snapshots at multiple scales.
• Prime-RL: the SFT training stack used for Qwen3-8B.