Modelling & mapping suite
Stage 2 of the pipeline — the missing middle. Modelling turns an approved requirements-catalog.yaml (the discovery output) into the source-to-target mapping files the build suite consumes. On a traditional engagement this bridge between requirements and build is done by hand; here three agents chain together — SourceProfiler → Modeller → SourceToTargetMapper — passing the profile report, then the logical model, then the mapping files down the line. The newest of the three suites: its first scored runs landed on 12 July (see Runs).
Target architecture and client fit
The reference target is a Microsoft Fabric Medallion architecture: Bronze preserves source fidelity, Silver validates and conforms reusable entities, and Gold serves business-ready data products. Within Gold, the suite applies Kimball dimensional modelling—clear grain, facts, dimensions, conformed dimensions, surrogate keys and slowly changing dimension intent—so outputs are ready for semantic models and Power BI.
Greenfield, migration and redesign
Best where the target architecture and modelling standards can be established deliberately rather than inherited piecemeal.
Targeted modernisation
Existing estates can adopt the profiling, modelling or mapping stages selectively when target patterns and interfaces can be agreed.
A model teams can extend
The client receives explicit modelling standards and machine-readable handover assets—not a diagram whose rationale disappears with the project team.
The three agents
| Agent | Role |
|---|---|
SourceProfiler | reads and profiles the real source schemas (strictly read-only) to ground the catalogue's source and data-quality sections |
Modeller | turns approved requirements into entities, relationships, conformed dimensions and facts, keys, and SCD intent — the conceptual and logical model |
SourceToTargetMapper | combines the logical model with the profiled source schemas to emit the gold mapping files build consumes |
Design principles
- Source access is read-only by construction — a connector protocol with no write path: an in-memory SQLite connector for synthetic sources in tests, and a Fabric connector for real schemas and data.
- Profiling is deterministic code, not model output — null rates, cardinality, candidate keys, PII flags and referential gaps are computed, so the model reasons over facts rather than inventing them.
- The logical model schema mirrors the real catalogue — the same
logical_modelblock the Meridian worked example carries. - Fully testable offline — sources are mocked with synthetic SQLite shaped like the real systems, so the whole suite runs without a live Fabric connection.
Deep dive: what a run produces
A modelling run leaves three artefacts in the engagement repo, each consumed by the next step. The profile report is deterministic facts about the real sources — row counts, null rates, candidate keys, referential gaps — that ground everything downstream. The conceptual and logical model maps approved requirements to entities, relationships, conformed dimensions and facts, keys and SCD intent; it uses the same logical_model schema the requirements catalogue carries, so it cross-references catalogue items by ID. The source-to-target mapping files are the handover contract: one per gold table, precise enough that the build suite can turn each into a tested Fabric table without human translation.
Because the catalogue and the logical model are platform-agnostic, this whole stage would survive a change of target platform — only the mappings' consumer (build) is Fabric-specific. The scored runs under Runs show both modes: modelling from the golden catalogue, and modelling from a live discovery output.
Current limitations
The mapper currently emits gold-layer dimension and fact mappings — generating bronze-ingest and silver-conform mappings is the natural next step — and profiling-derived data-quality findings are not yet first-class typed citations. Both are tracked and honest: the run scorecards under Runs show exactly where the suite stands today.