Build suite
Stage 3 of the pipeline. Build turns an approved mapping into a built, tested Fabric table with a pull request, driven from a work-item ticket. It is the original, production-hardened suite: a 14-step end-to-end workflow with two self-correction loops, where a debug engineer agent diagnoses failures and drives retries until the table builds, its data-quality tests pass, and the PR is opened.
The delivery asset stack
Build is a controlled ecosystem rather than a model writing notebooks from scratch. The agents translate approved design into the client-owned framework, while deterministic components execute and verify it.
| Asset | What it contributes | What the client keeps |
|---|---|---|
| Data Platform Agents | Interpret mappings, generate configuration, coordinate work, diagnose bounded failures and assemble evidence. | The repeatable delivery workflow and its auditable controls. |
| Delta-Gen | The deterministic engine that turns declarative table definitions into Fabric transformations. | A consistent implementation standard instead of bespoke notebook logic. |
| Delta-Gen YAML | Version-controlled definitions of sources, joins, keys, transformations, history and quality expectations. | Readable, reviewable data-product specifications. |
| delta-gen-template | The deployable repository structure, template notebooks, configuration layout and CI/CD conventions. | A platform scaffold their team can continue to extend. |
| Fabric + ADO | Fabric executes and stores the data products; ADO governs tickets, source, pipelines, reviews and pull requests. | Working tables plus the complete delivery and change history. |
Fabric greenfield & modernisation
Best when the programme can adopt the reference repository and Delta-Gen conventions. Brownfield teams can onboard table families progressively.
Capability, not code dependency
Automation helps a lean client team deliver consistently while retaining human approval, readable configuration and standard engineering practices.
Repeatable and auditable
The same specification drives the build and its tests; every change is linked to a ticket, evidence and a reviewable pull request.
The agents
| Agent | Role |
|---|---|
MappingParser | mapping doc (MD) → structured spec |
ConfigGenerator | spec → delta-gen YAML |
PipelineValidator | validates the YAML / schema |
DebugEngineer | diagnoses failures, drives the self-correction loops |
Executor | runs the template notebook in Fabric |
Observer | reads build/DQ outcomes |
Reconciler | gold reconciliation against a spec |
ADOIntegrator | commit, tests, PR |
Architect, DeltaGenDeveloper, PluginDeveloper, Coordinator | design review, engine/plugin dev, orchestration |
Deterministic helpers handle the plumbing around them — workspace pooling, artefact upload, lakehouse SQL, reconciliation — so the agents only make the judgement calls.
How a table ships
A ticket names the table and layer. The mapping is parsed into a structured spec, the spec becomes declarative Delta-Gen YAML, the YAML is validated and executed as a notebook run in Fabric, data-quality tests compiled from the same YAML are run against the built table, and the result is committed, tested and raised as a pull request. Failures at any step route through the debug engineer's self-correction loop rather than aborting the run. Alternative workflows reuse the same agents for smaller jobs: a new-table fast path, bug fixes, and plugin development for the engine itself.
Deep dive: the 14-step workflow
This is the exact state machine (EndToEndTableWorkflow). Note the shape: LLM agents only where reasoning is needed (steps 1–3 and the debug loops), deterministic gates and infrastructure everywhere else — no tokens, no hallucination surface.
| # | Step | Who | Kind |
|---|---|---|---|
| 1 | parse_mapping — mapping doc → structured spec | MappingParser | LLM |
| 2 | generate_yaml — spec → Delta-Gen YAML | ConfigGenerator | LLM |
| 3 | validate_schema — YAML validated against the schema | PipelineValidator | LLM |
| 4 | lint_expressions — expression lint over the YAML | ExprLint | gate |
| 5 | resolve_dependencies — upstream tables present? | FabricClient | gate |
| 6 | lease_pool — lease an isolated workspace | WorkspacePoolManager | infra |
| 7 | route_yaml — YAML routed to its layer/source path | SilverRouter | gate |
| 8 | upload_yaml — configs shipped to the lakehouse | FabricUploader | infra |
| 9 | run_layer — template notebook executed in Fabric, polled | Executor | infra |
| 10 | observe_layer — build + data-quality outcomes read back | Observer | gate |
| 11 | verify_table_built — the table really exists, with rows | FabricClient | gate |
| 12 | reconcile — gold output reconciled against the spec | Reconciler | gate |
| 13 | open_pr — commit, tests, pull request | ADOIntegrator | infra |
| 14 | release_pool — workspace returned (always, in finally) | WorkspacePoolManager | infra |
Two bounded self-correction loops (at most 3 debug iterations each) wrap the fragile regions: Loop A around steps 2–4 — a validation or lint failure goes to the DebugEngineer, which diagnoses and regenerates the YAML; Loop B around steps 9–10 — a build failure or a failing data-quality verdict triggers diagnose-and-retry. Bounded loops mean a bad run ends with a clear failure report, never an infinite burn.
A verification mode (--from-yaml --skip-pr) skips the LLM steps entirely and re-runs a known-good YAML through the build — used to verify infrastructure changes against real Fabric without spending tokens. Where this sits in the wider estate: see Architecture & background.