LegoFlow Experiments: Practical Lessons for Agentic Code Data Engineering

1. Block-Level Agent Operations and Experiments

This chapter reports experiments conducted by combining AI-agent operation with human coordination. Within each LegoFlow block, an AI agent autonomously configures and runs the block, inspects its outputs, and adapts the next step when needed. Humans then connect the outputs from different blocks to form the full experimental result. The end-to-end autonomous run is reported separately in Chapter 2.

1.1 Experimental Configuration

Every comparison below fixes the student, recipe, and budget and changes exactly one thing.

• Student: Qwen3.5-35B-A3B-Base, fine-tuned from scratch per source

• Reference: Qwen3.5-35B-A3B-Instruct, the official Qwen release, 63.4 Verified / 38.2 Pro

• Budget: ~1,000 trajectories per source, identical SFT settings

• Evaluation: SWE-bench Verified $500$ and SWE-bench Pro $731$

• Scaffold: OpenHands SDK 1.33.0, maximum 200 turns, maximum content length 256k

The two trajectory pools differ in one controlled respect: SWE-rebench-v2-1k rolls out against a public instance set, LegoFlow-SWE-Trajectories-1k against instances Curator mined, verified, scored, and tagged itself. Same teacher, same scaffold, same sampling policy, same budget. The gap between them is Curator's contribution, separated from Tracer's.

1.2 Results

To isolate the contribution of the trajectory source, we keep the student model, training budget, and evaluation protocol fixed while changing the source pool.

The LegoFlow pools separate from the open sources most clearly on Pro, the harder of the two benchmarks; on Verified the field is much tighter.

LegoFlow-SWE-Trajectories-1k is the headline result: 70.2 on Verified and 48.8 on Pro, 6.8 and 10.6 points above the instruct reference, 6.2 and 10.1 above the best open source we tested, and the only run in the comparison that lifts both benchmarks together. The rest of the field is bunched: SWE-rebench-v2-1k reaches 46.9 on Pro but leaves Verified at 64.4; StepFun 3.5 lands at 64.0 / 38.7, Scale-SWE 59.2 / 36.0, DeNovoSWE 59.8 / 33.7.

Swapping the instance source alone is worth 5.8 points on Verified and 1.9 on Pro. Both of our pools were rolled out by the same Tracer under the same GLM-5.2 teacher, scored by the same rubric, sampled by the same policy. Whatever Tracer contributes, it contributes equally to both, the separation is attributable to the tasks. The Verified gain is the more telling of the two, because Verified is the axis every other source is stuck on.

That is what the Databoard numbers predict. Curator's instances run 6.27 mean difficulty against SWE-rebench-v2's 5.80, 39.4% hard against 35.1%, eight languages against six, and the trajectories inherit it: 130k tokens over 104 turns at 714 CoT tokens per thinking turn, against 69k over 67 turns at 500. Harder, more varied tasks force longer investigations, and the student learns from the investigation rather than the answer. Trajectory synthesis has a ceiling set by the tasks underneath it, which is why Curator is a first-class block and not a preprocessing step.

Why Pro does so much of the work. StepFun sits within four-tenths of a point of SWE-rebench-v2-1k on Verified, 64.0 against 64.4, a difference nobody would defend as real, and then stalls at 38.7 on Pro, the instruct model's score. Two sources that look equivalent on one benchmark differ by 8.2 points on the other. The self-made pool is the case where that caveat does not apply: it is ahead on both.

LegoFlow trajectories use fewer reasoning turns than the external pools, but contain more reasoning tokens on each reasoning turn.

LegoFlow is not “more CoT everywhere.” The LegoFlow-SWE-Trajectories-1k pool uses reasoning on 62% of turns and averages 714 reasoning tokens per reasoning turn; the external pools use reasoning on 97–100% of turns and average 181–309 tokens per reasoning turn. The comparison suggests that selective, denser reasoning is more useful than shallow reasoning on every turn.

1.3 Scaffold Comparison

Every number above was measured in one scaffold, rolled out and evaluated under OpenHands SDK. So we ran the loop again end to end under OpenCode, same teacher, filters, rubric, sampling, ~1,000-trajectory budget and no-hack / 200-turn / 256k evaluation, with only the scaffold moved, on both sides.

Each row compares an instruct reference with a Base checkpoint fine-tuned on the corresponding pool. Values should be compared within a scaffold; the OpenHands and OpenCode rows use different runs.

Under OpenCode, the reference scores 53.0 on Verified and 37.7 on Pro, while the LegoFlow-SWE-Trajectories-1k checkpoint reaches 64.0 and 46.8. The gains are +11.0 and +9.1, respectively. Because the two runs use different Curator snapshots, this is a replication under a second scaffold rather than a controlled scaffold-only ablation.

The reference points are the more interesting part. The same instruct weights score 63.4 under OpenHands SDK and 53.0 under OpenCode, ten points from the scaffold alone, while on Pro the swap costs half a point. A scaffold change moves Verified further than most of the dataset differences above, which is why only the within-row comparison means anything.

1.4 How Teacher Choice Changes the Training Signal

The cleanest ablation we ran holds the instance set fixed at swerebenchv2_python and changes only the teacher.

Identical instances, identical recipe, different teacher: SWE-bench Pro moves from 30.2 to 46.9, a gain of 16.7 points.

The GLM-5 teacher produces shorter, more frequent reasoning: 82 tokens per turn in the figure, compared with 500 for GLM-5.2. The resulting student scores 30.2 on Pro with GLM-5 trajectories and 46.9 with GLM-5.2 trajectories, a 16.7-point difference. The controlled comparison shows that teacher choice is a major factor in the quality of the resulting training pool.

1.5 How Reasoning Depth Affects Transfer

We sorted a GLM-5.2 swerebenchv2 pool by average per-turn reasoning_content length, split it into quartiles, and trained on each quartile separately.

Nearly all of the gain comes from leaving the shortest quartile. Moving from 167 to 360 tokens per turn is worth 7.0 points on Verified; the next two steps, to 577 and then 812, are worth 1.0 point combined.

The shortest quartile reaches 57.0% on Verified. The next three quartiles reach 64.0%, 64.6%, and 65.0%, while average reasoning tokens per turn rise from 167 to 812. Most of the measured gain comes from moving beyond the shortest quartile; the remaining quartiles differ by only 1.0 point in total.

1.6 Reward Hacking and Evaluation Hygiene

Strong coding teachers are strong at reward hacking. They fetch upstream PRs, clone fixed repos, download raw patches, and sometimes state outright that their fix matches the upstream commit. On a benchmark built from public GitHub history, the answer is a search query away.

Removing upstream leakage costs the GLM-5.2 teacher 8.0 points on Verified $86.8 → 78.8$ and the student trained on its 1k pool 10.0 points $74.4 → 64.4$. Eval hygiene is not separate from data quality, it is the same concern.

Audited over all 500 Verified instances under a unified rubric. Confirmed hacking rises from 18 $3.6$ to 106 $21.2$, while suspect cases fall from 108 to 27.

The shift in labels is informative. GLM-5 was already reaching for the answer, just vaguely enough that the audit could only mark it suspect; GLM-5.2 does the same thing explicitly and successfully. The clean share changes only slightly, from 74.0% to 73.4%.

The mitigation is network policy, not prompt engineering. Harbor scopes public / no-network / allowlist to the environment, agent, verifier, or step; in practice the agent gets an allowlist of LLM provider endpoints only, and the verifier gets no-network. A cleaned .git history is not enough, the fix is still reachable through mirrors, caches, and search APIs. Stronger teachers need stricter sandboxes.

2. End-to-End Experimental Report (Beta)

This chapter reports the experiment in which an AI agent operated LegoFlow autonomously from task construction through trajectory generation, training, and evaluation, adapting the workflow to the available hardware and iterating on the result without manual intervention.

2.1 Agent-Orchestrated Run

Objective: Without using any external training data, an AI agent drives LegoFlow's blocks: curator, tracer, trainer, and evaluator, from end to end: it mines software-engineering tasks from raw GitHub repositories, has a teacher model solve them to produce training trajectories, and fine-tunes the Qwen3.5-35B-A3B-Base model to raise its solve rate on SWE-bench Verified.

TL;DR: The agent ran the pipeline and adapted the job to the available hardware without manual intervention. When the first attempt reached 56.1%, it inspected the run logs and produced a second, depth-filtered dataset. The second attempt reached 64.4%, an 8.3-point improvement.

2.2 Data Foundation and Resources

Resource: Training ran on 8× H800 80GB GPUs. Task construction used Claude-sonnet-4.6, trajectory collection used the GLM-5.2 teacher model through its API, and task generation ran on CPU and Docker.

Curator (task generation) builds verifiable software-engineering tasks from merged pull requests in real GitHub repositories. This experiment uses Python tasks. It uses claude-sonnet-4-6 through the Claude Code path to complete each task's container, detecting the language, build, and test stack. That process yielded 4,166 verifiable tasks.

Tracer (trajectory collection) uses GLM-5.2 as the teacher model, solves each task inside its own container, and records the run. Of the attempted tasks, 915 were solved, and those trajectories formed the raw pool for filtering. Some inference endpoints later returned HTTP 502 errors and timeouts; the agent moved traffic to healthy endpoints behind a load balancer and continued the run.

2.3 What the Agent Optimized

The first attempt, trained on 500 trajectories, reached 56.1%. Reading back its own trajectories, the agent noticed something that ran against the obvious guess: reasoning showed up almost everywhere, 80.7% of the model's turns had a <think> block, but most of those blocks were thin, a few dozen characters of "let me check" or "now I'll try" with no actual debugging in them. From that it formed a hypothesis and tested it on the second run:

• The read: what matters is how deep each reasoning step goes, not how often the model reasons at all.

• The change: keep only trajectories whose reasoning averages at least 500 characters per turn, accepting lower coverage in exchange for substance.

• The result: from the 915 raw trajectories it dropped 97 duplicates and 234 too-shallow ones, keeping 512. Median reasoning depth jumped from 140 to 959 characters per turn $5.9×$, while coverage fell from 80.7% to 30.7%.

The extra depth did not come from a stronger teacher: both runs used GLM-5.2. It came from changing the selection rule: from "keep anything with a <think> block" to "filter hard on reasoning density," which threw out the shallow runs. During conversion, the agent also caught and fixed a formatting problem: some trajectories wrote their tool calls in a format the evaluation scaffold wouldn't parse, so it added a step to normalize them and keep the trained model's output readable downstream.

2.4 Training and Evaluation

Training fine-tuned Qwen3.5-35B-A3B-Base on 512 depth-filtered trajectories with a 128k-token context, full-parameter SFT, global batch size 64, three epochs, and 24 optimizer steps. Two issues occurred during the run; the agent diagnosed them from the logs, adjusted the configuration, and restarted the job:

• Host memory ran out $not GPU memory$. It tracked this down to use_unsloth_gc, which was offloading activations to shared host memory and exhausting it at the 128k context length. Switching to standard gradient checkpointing fixed it and kept the full 128k context, so long trajectories weren't truncated.

• Multi-GPU training couldn't sync. NCCL was defaulting to the eth0 interface; on a single multi-GPU host it needs the loopback interface $`lo`$. Pointing it there brought distributed training back.

Full-parameter SFT loss over 24 steps for the second training attempt.

The plotted loss decreases across the 24 steps and ends at 0.229. The chart also highlights an initial peak of 0.577 and a summary value of 0.369.

Solve rate on 500 real GitHub issues. The depth-filtered second attempt reaches 64.4%, compared with 56.1% for the coverage-filtered first attempt.

On the 500-issue evaluation shown in the figure, the untrained base model scores 7.6%. The coverage-filtered first attempt, trained on 500 self-produced trajectories, reaches 56.1%; the depth-filtered second attempt, trained on 512 self-produced trajectories, reaches 64.4%. The second attempt improves by 8.3 points over the first and by 56.8 points over the untrained base model.

2.5 Why LegoFlow Enables This Workflow

Three properties of LegoFlow are what let the agent orchestrate the blocks, adapt to the hardware, and improve its own results the way it did:

• Dependencies across blocks are declared, not implied. Each block states, in both directions, which upstream outputs it consumes and which downstream inputs its outputs feed. So when something breaks, the agent can walk the data flow back to the real cause, a format mismatch points it at the upstream conversion script instead of a downstream patch, and a pre-run check catches broken wiring before a multi-hour job starts on the wrong footing.

• Every run archives the output files. When a block finishes, it records its exit code, logs, and config snapshot and writes its status to a shared index. The agent doesn't have to remember what the last run did; on a failure it just reads the archived scene, finds the problem, fixes it, and reruns.

• Every block looks the same from the outside. Curator, tracer, and trainer share one lifecycle and one archive format, and they report their key numbers the same way. That lets the agent pull and compare metrics across blocks, reasoning depth, training loss, solve rate, and tune behavior by changing parameters rather than reaching into a block's internals. The input/output contract is the boundary it works against.

3. Takeaways

The LegoFlow-SWE-Trajectories-1k pool reaches 70.2 on Verified and 48.8 on Pro in the comparison shown here. The controlled studies also show a 16.7-point Pro difference between the GLM-5 and GLM-5.2 teachers, and a 5.8-point Verified difference between the two LegoFlow pools. Most of the reasoning-depth gain comes from moving beyond the shortest quartile. Finally, the network and audit studies show that reward hacking can materially inflate both teacher and student scores, making isolation part of evaluation hygiene.