LegoX

9 min read · 2026-08-28

LegoFlow: Easy and Interactive Code Data Engineering

An agent-operable framework that turns task curation, trajectory rollout, training, evaluation, and iteration into one inspectable workflow.

TL;DR

Coding is a core capability of modern LLMs, yet producing high-quality coding data remains surprisingly complex. The pipeline spans multiple platforms, sandboxes, and compute resources, with substantial human effort required at every step. We thus introduce LegoFlow, an easy-to-use and interactive framework for code-data engineering. Users can now simply interact with a coding agent (e.g., Claude Code or Codex) to run the entire pipeline end to end:

  • Agent-native workflows: Repository and PR collection, task verification, trajectory rollout, training, and evaluation are exposed as plugin skills that coding agents can run.
  • Wide coverage: LegoFlow covers more than eight programming languages, more than 20 task tags, and trajectory rollouts across Claude Code, OpenCode, OpenHands, and Terminus.
  • Composable blocks: Each block contains the repositories, scripts, configuration, and runtime outputs for one stage.
  • Live dashboards: Each block has a dashboard for task difficulty, trajectory quality, and model performance.
  • Released dataset: We release LegoFlow-SWE, containing 5,000 verified tasks and GLM-5.2 rollouts collected with OpenHands SDK and OpenCode. A model trained on a sampled 1K subset reaches 70.2% on SWE-bench Verified and 48.8% on SWE-bench Pro.
  • End-to-end iteration: An agent ran the full loop, diagnosed the first fine-tune, and improved Qwen3.5-35B-A3B-Base from 7.6% to 64.4% on SWE-bench Verified.
  • Detailed experiments: See the companion report, LegoFlow Experiments: Practical Lessons for Agentic Code Data Engineering, for the full ablations and end-to-end run details.

Architecture

Figure 1: The overall architecture of LegoFlow
Figure 1: The overall architecture of LegoFlow

LegoFlow organizes the data pipeline as a loop of blocks, each with a specific role. The Root block orchestrates the overall workflow, connecting Curator → Tracer → Trainer → Evaluator.

What is a block?

A complex workflow is often divided into stages, each owned by one person. That owner is responsible for the work within their stage and communicates with neighboring owners when dependencies are required.

We apply this idea to agent workflows to reduce the need for humans in the loop. Each block has a specific role and contains everything associated with that role, including repositories, scripts, and dependencies. Blocks communicate through predefined formats, keeping every handoff clear and traceable. See What is a Block for details.

Triggered through plugin skills. Blocks are designed to be accessible to general users. Users interact with them through coding agents, with each procedure organized as a plugin skill. Users can also create or improve their own blocks by following Adding New Blocks.

Curator: Building verified coding-agent tasks

Curator creates verified coding-agent tasks from GitHub pull requests. It builds on SWE-gen and is being extended with additional data pipelines.

Figure 2: Curator discovers pull requests, rewrites them as clean tasks, builds Harbor environments, verifies them with NOP and Oracle, and publishes a manifest.
Figure 2: Curator discovers pull requests, rewrites them as clean tasks, builds Harbor environments, verifies them with NOP and Oracle, and publishes a manifest.

As shown in Figure 2, Curator moves each candidate through three gates:

  • Discover and filter. Search merged GitHub pull requests across the target languages, reject trivial or unsuitable changes, collect issue and test evidence, and rewrite each case as a leak-free bug report.
  • Construct. Separate the buggy baseline from the held-out fix, package the candidate in the standard Harbor task format, and complete its Docker and test environment.
  • Verify and publish. Run NOP and Oracle validation, then score, tag, and publish only tasks whose buggy baseline fails and whose held-out solution passes.

Task Scoring. Curator assigns each verified task a deterministic difficulty score from 1 to 10. The rubric combines patch scope (30%), logic complexity (25%), context breadth (20%), test complexity (15%), and instruction complexity (10%), then maps the result to easy, medium, or hard.

Tracer: Generating and selecting agent trajectories

Tracer generates trajectories from verified tasks using Claude Code, OpenCode, OpenHands, and other coding-agent scaffolds.

Figure 3: Tracer stages verified tasks, records model calls through a proxy, runs Harbor rollouts, converts successful traces, and updates the dashboard.
Figure 3: Tracer stages verified tasks, records model calls through a proxy, runs Harbor rollouts, converts successful traces, and updates the dashboard.

Figure 3 shows how Tracer turns a verified task into reusable training data. A per-job LiteLLM proxy provides both OpenAI- and Anthropic-compatible interfaces, allowing different coding-agent scaffolds to use the same upstream model while preserving complete model calls for later conversion.

Each rollout runs in an isolated Harbor container and receives a verifier reward. Unsuccessful, duplicate, truncated, malformed, and evaluation-overlapping traces are removed; the remaining trajectories are scored for quality before sampling.

The quality score complements the pass/fail reward and supports later filtering and sampling.

Training and evaluation

Trainer converts valid trajectories into a standard SFT dataset and produces a checkpoint using LLaMA-Factory. Evaluator then runs that checkpoint on a defined benchmark and scaffold, recording the setup and applying network and anti-hacking controls so results remain comparable.

Dashboard visualization

Each block includes a live dashboard for tracking progress and diagnosing failures. For instance, the Curator dashboard below summarizes the current run, from repository and PR intake to constructed and verified tasks.

Figure 4: The Curator dashboard summarizes seven batches: 822 repositories, 8,818 collected pull requests, 926 constructed tasks, and 270 verified tasks. Its funnel exposes retention at each stage, while difficulty and tagging metrics summarize the resulting task pool.
Figure 4: The Curator dashboard summarizes seven batches: 822 repositories, 8,818 collected pull requests, 926 constructed tasks, and 270 verified tasks. Its funnel exposes retention at each stage, while difficulty and tagging metrics summarize the resulting task pool.

Users can open a dashboard through the block's plugin skill. It synchronizes with the latest run artifacts, giving users and agents an up-to-date view without a manual refresh.

Case studies

For the detailed experimental configuration, ablations, evaluation-hygiene findings, and end-to-end iteration log, see the companion LegoFlow Experiments report.

To demonstrate LegoFlow, we provide two case studies for complementary questions:

  1. whether LegoFlow can produce high-quality training data; and
  2. whether an agent can operate the complete workflow and improve it from execution feedback.

Case Study I: Building LegoFlow-SWE from 12M Candidate PRs

Data construction

Software-engineering data construction has become a major community effort since 2024. Recent work has expanded both scale and task scope. For instance, our previous attempt SWE-Lego combines 32K tasks with 18K validated trajectories, enabling its Qwen3-32B model to reach 52.6%. ScaleSWE constructs 100K verified real-world tasks and reports 64% on SWE-bench Verified.

Figure 5: LegoFlow-SWE retains 5,000 tasks from 12M candidate pull requests, then records 9,767 GLM-5.2 rollouts across OpenHands SDK and OpenCode.
Figure 5: LegoFlow-SWE retains 5,000 tasks from 12M candidate pull requests, then records 9,767 GLM-5.2 rollouts across OpenHands SDK and OpenCode.

Given how extensively this domain has been optimized, we find LegoFlow can still improve data quality by constructing LegoFlow-SWE from more than 12M candidate pull requests.

Specifically, we collect more than 700K GitHub repositories and filter them by activity, popularity, language coverage, merged pull requests, and dependency validity, producing 12M candidate pull requests. We then retain pull requests with valid, reasonably sized diffs, test files, and other required signals. An LLM judge removes trivial tasks and those that fail pass-to-pass or fail-to-pass validation, leaving only 5,000 difficult tasks. We generate trajectories using GLM-5.2 on these tasks with two scaffolds. OpenHands SDK 1.33.0 resolves 1,350 of 4,753 rollouts (28.4%), while OpenCode 1.18.7 resolves 1,430 of 5,014 (28.5%).

Main result

We fine-tune Qwen3.5-35B-A3B-Base separately on approximately 1,000 trajectories per source, keeping the scoring, sampling, and SFT recipe fixed. We evaluate on SWE-bench Verified and SWE-bench Pro under OpenHands SDK 1.33.0, with a 200-turn limit and 256k maximum content length.

The model trained on the sampled LegoFlow-SWE subset reached 70.2 on SWE-bench Verified and 48.8 on SWE-bench Pro, which is 6.8 and 10.6 points above the instruct reference, and 6.2 and 10.1 points above the strongest open trajectory source tested. With the teacher, scaffold, scoring, sampling, and budget fixed, replacing SWE-rebench-v2 tasks with LegoFlow tasks improved Verified by 5.8 points and Pro by 1.9 points.

Figure 6: With the teacher, scaffold, scoring, sampling, and budget fixed, the LegoFlow task source improves both Verified and Pro.
Figure 6: With the teacher, scaffold, scoring, sampling, and budget fixed, the LegoFlow task source improves both Verified and Pro.

Case Study II: Autonomous end-to-end iteration (Beta)

This study asks whether the agent can operate LegoFlow as a complete system, recover from runtime failures, and improve the result using archived execution feedback. In this setting, users give the Root block a global objective, and it orchestrates the remaining blocks toward that goal.

Experimental setup

Objective. Without using a pre-existing trajectory dataset, an AI agent drives curator, tracer, trainer, and evaluator. It mines software-engineering tasks from GitHub, collects teacher trajectories, and fine-tunes Qwen3.5-35B-A3B-Base for SWE-bench Verified.

Run construction

  • Run 1, coverage-first. The agent trained on 500 trajectories selected largely by the presence of reasoning. Reasoning appeared in 80.7% of turns, with a reported depth of 140 characters per block. The checkpoint reached 56.1%.
  • Run 2, depth-first. From 915 solved trajectories, the agent retained 512 after deduplication and depth filtering. Reasoning coverage fell to 30.7%, with a reported depth of 959 characters per turn. The depth values use different denominators and should not be read as a direct ratio. The checkpoint reached 64.4%.

Main result

The depth-first checkpoint reached 64.4% on SWE-bench Verified, 8.3 points above the coverage-first run and 56.8 points above the untrained base model.

Figure 7: On 500 SWE-bench Verified issues, the depth-filtered run reaches 64.4%, compared with 56.1% for the first run and 7.6% for the base model.
Figure 7: On 500 SWE-bench Verified issues, the depth-filtered run reaches 64.4%, compared with 56.1% for the first run and 7.6% for the base model.

What’s next?

More types of tasks. We are seeking to extend more types of data pipelines into LegoFlow, including terminal tasks, long-horizon programs for benchmarks like NL2Repo and ProgramBench, and multi-turn coding tasks. These tasks use the Harbor task format so they can reuse the existing pipeline.

Self-recursive improvement. Execution feedback of a block can guide block-level iteration. We also seek to refine handoff boundaries so agents can improve individual blocks while keeping the process legible to people.

We are actively improving LegoFlow. Stay tuned!

Citation

@misc{tao2026legoflow,
  title        = {LegoFlow: Easy and Interactive Code Data Engineering},
  author       = {The LegoX team},
  year         = {2026},
  url          = {https://legox.com/blog/legoflow/},
}