01Market Data Monetization via Federated Data Mesh — Fabric & Databricks PoC
Siloed market data became a governed, externally shareable product platform in one PoC
Role: Lead Solution Engineer / Architect
Executive summary
Designed and led a time-boxed proof-of-concept for a federated data mesh that lets a capital-markets client securely monetize curated market data as products—combining Fabric/OneLake domains, Databricks engineering, embedded Power BI and an AI multi-agent query layer; the client approved a production roadmap based on it.
- Microsoft Fabric
- OneLake
- Azure Databricks
- Power BI
- Microsoft Purview
- Entra ID
- Azure OpenAI
- NestJS
- Python
A leading capital-markets operator (a stock exchange) wanted to monetize its data assets by offering rich data products through a unified platform. Its data sat in siloed raw files with manual ETL by customers, limiting adoption. Strict regulation and multiple lines of business demanded a data-mesh architecture with domain ownership and federated governance to share data products securely with external clients.
- Architect a cross-domain data hub on Microsoft Fabric and Azure Databricks, using OneLake for unified storage and domain-specific workspaces (data mesh).
- Build data-engineering pipelines to ingest, refine and serve curated datasets (market data, trading analytics) with Spark on Fabric and Databricks.
- Add a machine-learning layer for advanced data products (trading anomaly detection, Copilot-driven insights).
- Deliver a customer-facing app: a Python multi-agent backend and a NestJS micro-frontend embedding Power BI analytics.
- Enforce governance & compliance via federated governance with Microsoft Purview, the Fabric domain model, row/column-level security and MFA for external access.
- Keep the PoC honest by scope — prove secure external data-sharing with banks and asset managers under the client's regulatory constraints, not ship a production system.
I led the architecture design with the client's data team and built a PoC on Fabric showing how OneLake and domain workspaces give each business unit isolated control while sharing data through a central marketplace. I configured Fabric medallion pipelines (Bronze/Silver/Gold) and Databricks for heavy transformations. For AI, I introduced a multi-agent approach—an Azure OpenAI agent handling user queries with Retrieval-Augmented Generation (RAG) over documentation, and another translating requests into SQL/Spark. The NestJS micro-frontend embedded interactive Power BI dashboards. I considered extending the client's existing manual file channels for external distribution instead of a platform build, and rejected it in the design review because row/column-level enforcement and consent to share could not be centralized per product—a decision that anchored the rest of the architecture. I coordinated distributed teams across time zones and version-controlled designs and code.
Time-boxed PoC delivered in full scope — domain workspaces per line of business, medallion pipelines end to end, an external subscription-sharing simulation with banks and asset managers, embedded Power BI consumption, and AI-assisted querying over curated products. Purview-governed sharing was verified against the client's regulatory constraints before every demo. On that evidence, the client approved a production roadmap—landing zone, capacity planning and cost model—signing the PoC off as the architectural baseline. The platform shipped instrumented for launch metrics (domain adoption, pipeline latency, cost per tenant); the first measurement cohort followed after my engagement ended, so no outcome numbers are claimed here.
Data-mesh principles can be implemented pragmatically with Fabric domains and Databricks integration. Embedding security from day one—Purview plus Entra ID—gave the client confidence in regulatory compliance. One cost I would manage differently — the multi-agent query layer needed robust orchestration to control shared state, and the PoC's timebox left it under-hardened—that debt shaped the agent designs I delivered on later projects.