Why these agents exist
Every data-platform engagement repeats the same pipeline: weeks of discovery meetings distilled into requirements by hand, a modelling phase done in whiteboards and spreadsheets, and then a build phase where each table is specified, built and tested one ticket at a time. The expensive parts are not the judgement calls — they are the transcription, the cross-referencing, the "who approved this?" archaeology, and the mechanical translation of an agreed design into working tables.
This platform automates exactly those parts, and deliberately nothing more. The architectural decision (recorded as ADR-001 in the repo) is deterministic workflows that call LLMs only for the genuinely fuzzy steps — not one autonomous agent in a free-running loop. A state machine owns control flow, so runs are predictable, debuggable and resumable. LLM-powered agents handle extraction, generation and diagnosis; everything mechanical is deterministic code with no hallucination surface; every side effect goes through a tool; and guard rails (token and wall-clock budgets, a stall detector, human-in-the-loop gates) bound the whole thing.
The runtime is provider-agnostic by design — Azure OpenAI or Anthropic, selected per client — because a consulting product has to run on whatever the client's estate allows, not on one vendor's hosted agent platform.
The target: a Medallion lakehouse on Microsoft Fabric
Everything the agents do converges on one deliverable — a governed Bronze → Silver → Gold lakehouse in Microsoft Fabric, where every transformation is a declarative Delta-Gen YAML file and the repository layout follows the delta-gen-template reference structure. Medallion defines how data progresses through the platform; Kimball dimensional modelling shapes the Gold layer into facts, dimensions and conformed business structures for semantic models and analytics.
One nuance worth knowing: discovery and modelling are platform-agnostic — a requirements catalogue and a logical model could target any stack. It is the build suite that commits to Fabric, turning mappings into lakehouse tables through Delta-Gen. Swapping the target platform means swapping build, not the pipeline.
Production reference deployment
For a client deployment, the recommended control plane is an Azure Container Apps service in the client subscription. It suits the long-running, resumable workflow better than a short-lived function: the orchestrator can poll Fabric, apply bounded retries and preserve run state without relying on a developer laptop. Local execution remains useful for engineering and controlled demonstrations.
Azure Container Apps
Hosts the Python orchestration and agents; Managed Identity accesses client services; Key Vault holds any unavoidable secrets.
Microsoft Fabric
Runs notebooks and Delta-Gen, stores Bronze/Silver/Gold tables and exposes SQL endpoints for verification and reconciliation.
Azure DevOps
Owns work items, engagement and platform repositories, pipelines, approvals, commits and pull requests.
Client-approved endpoint
Anthropic Claude Opus directly or through an approved Azure model endpoint; orchestration remains in the accelerator runtime.
Azure observability
Application Insights and Log Analytics provide run telemetry, diagnostics and operational evidence.
Catalogue and evidence site
A separately hosted static interface presents documentation, engagement catalogues and approved evidence to stakeholders.
Where everything runs
The agents sit between the client's estate and the engagement record. Artefacts come in from SharePoint and Teams, source systems are profiled read-only, and the build loop works tickets from Azure DevOps against Fabric — building, observing and testing until the table is right, then raising the PR back in ADO. Azure AI Foundry or Anthropic provides model inference only; the agentic behaviour, workflow state and guardrails remain in the accelerator's own orchestration.
The engagement repo is the system of record — every catalogue item, decision and mapping lands there with citations, and this console is rendered straight from it. The dashed box is the roadmap: once Build has shipped the lakehouse, Fabric data agents would let the client query it in natural language — closing the loop from "first conversation" to "ask your data anything".