Enterprise Data & AI · Financial Services and beyond

Diogo Miyake

Data & AI Engineer | Solution Architect | Cloud | MLOps

I help companies at enterprise-grade level turn complex data estates into secure, production-grade data & AI platforms across Azure and Google Cloud.

View projects Get in touch
Featured projects
25
ML deploy cycle
months → weeks
NL2SQL accuracy (fine-tuned)
~85%

Executive summary

I am a Data & AI Solutions Engineer with experience delivering enterprise-scale data & AI initiatives across Azure and Google Cloud. I work with data at cloud platforms since 2019 with data engineering expertise. Since August 2024, my work has spanned data monetization platforms, generative AI / LLM and agentic solutions, MLOps and production ML, FinOps and governance, and competitive pre-sales, built on Azure Databricks, Microsoft Fabric and Microsoft Foundry. Earlier at Thoughtworks, I designed near-real-time airline and retail data platforms that deepened my foundations in CDC, distributed systems and event-driven architecture. Every initiative emphasizes secure architecture, governance & compliance (Microsoft Purview, Entra ID, RLS/CLS) and production readiness (scalable design, monitoring, drift detection, FinOps), delivered through close cross-team collaboration.

About

I translate ambiguous business goals into reference architectures and working solutions that survive production. I am equally comfortable designing a federated data mesh, fine-tuning an LLM, standing up an MLOps pipeline, or leading a competitive proof-of-concept under deadline.

My engagements pair technical depth with clear storytelling: I build the demo, draw the architecture, run the workshop, and write the proposal—then package the result as a reusable accelerator so the value scales beyond a single customer.

Core competencies

A snapshot of the platforms, frameworks and practices I use to deliver enterprise data & AI solutions.

Data Platforms

  • Microsoft Fabric
  • Azure Databricks
  • OneLake
  • Delta Lake
  • ADLS Gen2
  • Azure Synapse
  • Google Cloud Datastream
  • Google Cloud Dataflow
  • Google Cloud Pub/Sub
  • Spark

Streaming & Distributed Systems

  • Fabric RTI
  • Spark Structured Streaming
  • Change Data Capture (CDC)
  • Event-Driven Architecture
  • Apache Beam
  • Azure Functions
  • Event Ordering & Deduplication
  • Data Consistency & Quality

AI & Machine Learning

  • Azure OpenAI
  • LLM fine-tuning
  • RAG & retrieval architectures
  • LangChain / LangGraph
  • Microsoft Foundry
  • PyTorch
  • MLflow
  • AutoML
  • Azure ML

Governance & Security

  • Ontologies
  • DAMA BOK
  • Semantic Modeling
  • Data Mesh
  • Microsoft Purview
  • Entra ID
  • RLS / CLS
  • Conditional Access
  • FinOps / FOCUS

BI & Visualization

  • Power BI
  • DirectLake
  • Copilot

Engineering & DevOps

  • Python
  • NestJS
  • Azure DevOps
  • Terraform / Bicep
  • CI/CD

Ways of Working

  • Solution Architecture
  • Pre-Sales
  • RFP / RFI
  • Competitive Positioning
  • Architecture Decision Records
  • Stakeholder Management
  • Cross-functional Leadership
  • Workshops & Enablement
  • Partner / ISV Management

Soft Skills

  • Presentation & Storytelling (Under Constant improvement)
  • Mentorship & Enablement
  • Writing & Documentation
  • Leadership (Under Constant improvement)
  • Collaboration
  • Problem Solving
  • Negotiation (Under Constant improvement)
  • Sales (Under Constant improvement)

Languages

  • English (Fluent)
  • Portuguese (Native)
  • Spanish (Intermediate)
  • Japanese (Basic - Studying)
  • French (Basic - Studying)
  • Italian (Basic - Studying)

Project portfolio

25 selected engagements, summarized here and documented in full on their own page using the STARL method (Situation, Tasks, Actions, Results, Lessons Learned). Client identities are anonymized by sector. Use the index, filters or search to navigate.

Data Platform, Mesh & Governance

Federated data mesh, reference blueprints and cross-platform governance.

Data Platform, Mesh & Governance Capital Markets ✓ Client sign-off

01Market Data Monetization via Federated Data Mesh — Fabric & Databricks PoC

Siloed market data became a governed, externally shareable product platform in one PoC

Role: Lead Solution Engineer / Architect

Executive summary

Designed and led a time-boxed proof-of-concept for a federated data mesh that lets a capital-markets client securely monetize curated market data as products—combining Fabric/OneLake domains, Databricks engineering, embedded Power BI and an AI multi-agent query layer; the client approved a production roadmap based on it.

  • Microsoft Fabric
  • OneLake
  • Azure Databricks
  • Power BI
  • Microsoft Purview
  • Entra ID
  • Azure OpenAI
  • NestJS
  • Python
Read the full case study →
Data Platform, Mesh & Governance Financial Services

11“Better Together” — Ending the Databricks-vs-Fabric Debate for FSI Clients

Shifted FSI clients from Databricks-vs-Fabric fear to trials with no rip-and-replace

Role: Solution Engineer / Field Lead

Executive summary

Defined and evangelized “Better Together” reference architectures that integrate Azure Databricks and Microsoft Fabric—shifting client conversations from “vs” to “and” and unlocking new Fabric workloads atop existing Databricks lakehouses.

  • Azure Databricks
  • Microsoft Fabric
  • OneLake
  • Shortcuts
  • Power BI
  • Microsoft Purview
Read the full case study →
Data Platform, Mesh & Governance Financial Services ✓ Peer-reviewed

12Azure Data Mesh Reference Architecture — A Repeatable Enterprise Blueprint

Peer-reviewed blueprint mapping data-mesh theory to concrete Azure services for FSI banks

Role: Architect / Author

Executive summary

Authored an enterprise Azure Data Mesh reference architecture—domain-oriented data products, federated governance and security—reused as a repeatable blueprint across multiple FSI engagements.

  • Azure Databricks
  • Unity Catalog
  • Microsoft Fabric
  • Microsoft Purview
  • ADLS Gen2
  • Delta Lake
  • Entra ID
Read the full case study →
Data Platform, Mesh & Governance Financial Services

13Federated Data Governance — Purview, Databricks & Fabric on One Policy Plane

One governance pane for Databricks and Fabric — RLS gap managed, not hidden

Role: Governance Solution Engineer

Executive summary

Integrated Microsoft Purview with Azure Databricks and Microsoft Fabric to deliver a unified catalog, cross-platform masking and an Entra ID-based access model—giving a financial-services client a single governance pane.

  • Microsoft Purview
  • Azure Databricks
  • Unity Catalog
  • Microsoft Fabric
  • Entra ID
  • RLS / CLS
  • Conditional Access
Read the full case study →
Data Platform, Mesh & Governance Airline Thoughtworks

19Airline Data Platform — An ADR That Killed Custom CDC (and Pivoted to Data Mesh)

An ADR replaced fragile custom CDC with managed CDC — reconciliation effort down ~60%

Role: Cloud Solutions Architect

Executive summary

Assessed and redesigned a major airline's first modern Google Cloud data platform, replacing a fragile custom-CDC direction with a managed-CDC decision path and a scalable, event-driven Data Mesh architecture for reliable near-real-time analytics.

Data-reconciliation effort
~60% lower custom-CDC design baseline
  • Google Cloud
  • Oracle CDC
  • Google Cloud Datastream
  • GKE
  • Dataflow / Apache Beam
  • Pub/Sub
  • Data Mesh
  • Architecture Decision Records
Read the full case study →
Data Platform, Mesh & Governance Retail Thoughtworks ✓ Production

20Smart-Trolley Telemetry on Azure — Real-Time Fulfillment Without Databricks

Shipped real-time trolley telemetry without Databricks — race conditions halved by design

Role: Data Solutions Engineer

Executive summary

Designed and delivered an Azure-native fulfillment data platform for smart shopping-trolley telemetry and product master data, using custom streaming controls to achieve reliable near-real-time operations without Databricks.

  • Microsoft Azure
  • Azure Synapse Analytics
  • Azure Functions
  • Azure Data Factory
  • IoT Streaming
  • Event-Driven Architecture
  • Concurrency Control
  • Data Quality
Read the full case study →

Generative AI, LLMs & Agents

Fine-tuned LLMs, grounded RAG, multi-agent systems and enablement.

Generative AI, LLMs & Agents Banking

02Fine-Tuned LLM for NL2SQL — From ~60% to ~85% SQL Accuracy Across Dialects

Fine-tuned GPT-4.1 lifted NL2SQL accuracy from ~60% to ~85% and routed each question to the right data source

Role: Solution Engineer / ML Lead

Executive summary

Fine-tuned a GPT-4.1 model and built a multi-agent NL2SQL system that lets business users query Databricks, Fabric, SQL and Oracle in natural language—reaching ~85% SQL accuracy.

NL2SQL accuracy (eval on known query types)
~85% ~60% on the base model
  • Azure OpenAI
  • GPT-4.1 fine-tuning
  • Azure ML
  • LangChain / LangGraph
  • Azure AI Search
  • Databricks
  • Fabric Warehouse
  • Oracle
  • SQL
Read the full case study →
Generative AI, LLMs & Agents Insurance ✓ Production

04Grounded RAG Assistant — Contract Q&A With Clause-Level Citations

Contract Q&A over tens of thousands of pages with clause-level citations in under five seconds

Role: Solution Engineer / AI Architect

Executive summary

Delivered a secure, grounded RAG assistant over contracts and knowledge bases using Postgres/pgvector retrieval, GPT-4, Fabric Data Agent and Foundry—answering complex questions with clause-level citations in under five seconds on average.

  • Azure Database for PostgreSQL
  • pgvector
  • Azure OpenAI (GPT-4)
  • LangChain
  • Fabric Data Agent
  • Microsoft Foundry
  • Azure Monitor
  • Databricks
Read the full case study →
Generative AI, LLMs & Agents Enablement & Community

08“LLM From Scratch” Workshop — Training a GPT-2-Scale Model Live on Azure

Colleagues ran a real training loop end to end — and launched their own PoCs right after

Role: Instructor / Author

Executive summary

Designed and delivered a hands-on workshop on building and fine-tuning LLMs/SLMs on Azure (Databricks + Fabric)—enabling colleagues and partners to start their own domain-specific LLM proofs-of-concept.

  • Azure Databricks
  • Microsoft Fabric
  • Hugging Face Transformers
  • PyTorch Lightning
  • OneLake
  • Microsoft Foundry
Read the full case study →
Generative AI, LLMs & Agents Consumer Banking

09Multi-Agent Persona Simulations for Early Product Research (Consumer Banking)

Synthetic focus groups stress-tested product concepts before real research spend

Role: Solution Engineer / Prototyper

View repository on GitHub ↗

Executive summary

Prototyped an AI persona framework using multi-agent LLM simulations to generate synthetic customer personas and stress-test new product concepts before costly market research.

  • Azure OpenAI
  • Microsoft Agent Framework
  • TinyTroupe (Microsoft Research)
  • NLP sentiment analysis
Read the full case study →

Machine Learning & MLOps

Production ML pipelines, model comparison and platform evaluation.

Machine Learning & MLOps Financial Services (Payments) ✓ Production

03Fraud Model to Production — Databricks MLOps, Months to Weeks

Fraud model moved from notebook to live scoring — releases cut from months to weeks

Role: MLOps Solution Engineer

Executive summary

Built an end-to-end MLOps pipeline on Azure Databricks to productionize a PyTorch fraud-detection model with Feature Store, real-time serving and drift monitoring—cutting deployment time from months to weeks.

  • Azure Databricks
  • PyTorch
  • MLflow
  • Feature Store
  • Model Serving
  • Spark Structured Streaming
  • Azure DevOps
Read the full case study →
Machine Learning & MLOps Retail

06Churn Model Showdown on Azure ML — AutoML Beat Custom PyTorch (AUC 0.88 vs 0.84)

Ran a fair AutoML-vs-PyTorch bake-off on Azure ML — and the decision framework mattered more than the winner

Role: Solution Engineer / Data Scientist

Executive summary

Ran a head-to-head AutoML vs custom-PyTorch churn study on Azure ML with MLflow tracking and SHAP explainability; AutoML won (AUC 0.88 vs 0.84) and was deployed as a scoring endpoint.

Best model AUC (AutoML)
0.88 0.84 (custom PyTorch)
  • Azure Machine Learning
  • AutoML
  • PyTorch
  • MLflow
  • SHAP
  • Online Endpoints
Read the full case study →
Machine Learning & MLOps Financial Services (Enablement)

07ML on Microsoft Fabric — FSI Positioning Built From Working Demos, Not Slideware

Live prototypes settled Fabric-vs-Databricks positioning for FSI — measured, not assumed

Role: Pre-Sales Solution Engineer

Executive summary

Created demos and prototypes proving end-to-end PyTorch ML workflows on Microsoft Fabric, clarifying Fabric vs Databricks positioning for FSI clients evaluating Fabric as an ML platform.

  • Microsoft Fabric
  • Fabric Data Science
  • Spark notebooks
  • AutoML
  • PyTorch
  • MLflow
  • Azure Databricks
Read the full case study →

FinOps, DevOps & Reliability

Cloud cost intelligence and infrastructure governance.

FinOps, DevOps & Reliability Financial Services

05FinOps on Fabric — FOCUS-Standard Cost Reporting and Reservation Analytics

FOCUS cost intelligence in Power BI — 78% reservation coverage exposed for action

Role: Solution Engineer / FinOps

Executive summary

Built a Fabric-based FinOps solution that ingests Azure cost data, models it to the FOCUS standard, and surfaces reservation coverage and forecasts in Power BI—driving smarter reservation purchases.

  • Microsoft Fabric
  • Lakehouse
  • Warehouse
  • Dataflows
  • Spark
  • Power BI
  • Azure Cost Management
  • FOCUS
Read the full case study →
FinOps, DevOps & Reliability Fintech ✓ Production

10Automated IaC Drift Detection — Compliance Radar for a Regulated Fintech

Daily drift checks in CI/CD — manual portal changes caught within a day

Role: Advisory Solution Engineer

Executive summary

Designed and implemented automated IaC drift detection that compares deployed Azure resources to Terraform/Bicep state and alerts on drift via Teams and email—improving the client's compliance posture.

  • Terraform
  • Bicep
  • Azure Resource Graph
  • Python
  • Azure DevOps
  • Azure Policy
Read the full case study →

Pre-Sales, Competitive & Delivery

Competitive PoCs, RFP/RFI, partners and reusable accelerators.

Pre-Sales, Competitive & Delivery Insurance (BI)

14Tableau Under Pressure — Power BI PoCs on Real Data, ~40% Faster Loads

Rebuilt live Tableau dashboards on Fabric — ~40% faster loads, Copilot Q&A, and migration pilots approved

Role: Competitive Solution Engineer

Executive summary

Led competitive Power BI + Fabric PoCs that reimplemented Tableau dashboards on the client's data—cutting report load times ~40% and adding Copilot natural-language Q&A—moving clients to pilot a Tableau-to-Power BI transition.

Report load time for the flagship dashboard
~40% faster the client's Tableau implementation of the same report
  • Power BI
  • DirectQuery / DirectLake
  • Vertipaq
  • Microsoft Fabric
  • Lakehouse
  • Warehouse
  • Copilot
  • Oracle
  • RLS
Read the full case study →
Pre-Sales, Competitive & Delivery Financial Services / Public Sector

15Beating AWS & GCP on Print — Azure Target Architectures That Moved Pursuits

Target architectures kept Azure in play — including a national bank pilot over an AWS lean

Role: Competitive Solution Engineer

Executive summary

Crafted competitive target architectures and value narratives positioning the Azure data & AI stack against AWS and GCP—contributing to competitive wins and shortlists in FSI and public-sector pursuits.

  • Azure Databricks
  • Microsoft Fabric
  • Azure Data Factory
  • Azure Synapse
  • Microsoft Purview
Read the full case study →
Pre-Sales, Competitive & Delivery Banking

16Cloudera-to-Azure RFP — Architecture, Costing and Writing to Final Negotiation

Owned architecture, costing and writing on the RFP that advanced a bank bid to final negotiation

Role: Solution Architect (Bid Support)

Executive summary

Drove the architecture and technical-writing portions of major RFP/RFI responses for Financial Services — the flagship being a bank's Cloudera-to-Azure data-platform bid that advanced to final negotiation.

  • Azure Databricks
  • Microsoft Fabric
  • Microsoft Purview
  • Azure Pricing Calculator
  • Visio / PowerPoint
Read the full case study →
Pre-Sales, Competitive & Delivery Cross-Industry

17Steering a Global SI + ISVs — One Architecture, One Plan, On-Schedule Delivery

Led a global SI and ISV delivery under one architecture — rollout kept on schedule

Role: Technical Lead / Partner Manager

Executive summary

Managed and integrated partners and ISVs across enterprise delivery—onboarding them to Azure reference patterns and acting as a quality gate; the flagship was an insurance data-platform rollout where I owned architecture while a global systems integrator built the ETL under it.

  • Microsoft Fabric
  • Microsoft Purview
  • Azure
  • SharePoint / Teams
Read the full case study →
Pre-Sales, Competitive & Delivery Cross-Industry

18Demos, PoCs & Accelerators — The Asset Base Behind Faster Engagements

Tailored demos and instrumented pilots that moved deals — and reusable accelerators

Role: Solution Engineer

Executive summary

Continuously delivered tailored demos, PoCs, pilots and reusable accelerators across Azure Data & AI—progressing deals and creating assets reused by colleagues across regions.

  • Microsoft Fabric
  • Azure Databricks
  • Azure OpenAI
  • Microsoft Foundry
  • Microsoft Purview
Read the full case study →

Independent Projects — Teaching & Open Source

FIAP course authorship, curated learning resources and personal open-source tools built outside client engagements.

Independent Projects — Teaching & Open Source Education (FIAP MLET Postgraduate)

21FIAP MLET Course — Data Drift Detection & Monitoring, Statistics to Streaming

Eight graduate classes from KS tests to streaming-scale drift monitoring, ADRs included

Role: Course Author / Instructor

View repository on GitHub ↗

Executive summary

Authored an 8-class graduate course on detecting, monitoring and mitigating data and concept drift in production ML systems, for FIAP's postgraduate Machine Learning Engineering program.

  • Python
  • scikit-learn
  • Evidently
  • NannyML
  • River
  • Alibi-Detect
  • Hugging Face Transformers
Read the full case study →
Independent Projects — Teaching & Open Source Education (FIAP MLET Postgraduate)

22FIAP MLET Course — Software Engineering for Data Scientists, One Evolving Codebase

Eight classes driving one real credit-scoring codebase through tests, Docker and CI/CD

Role: Course Author / Instructor

View repository on GitHub ↗

Executive summary

Authored an 8-class course teaching data scientists the software-engineering practices — version control, testing, containers, dependency management, code quality and CI/CD — needed to ship reliable, production-ready code.

  • Python
  • Git
  • Docker
  • pytest
  • Linters & Formatters
  • CI/CD
Read the full case study →
Independent Projects — Teaching & Open Source Education (FIAP MLET Postgraduate)

23FIAP MLET Course — LLM Security, Guardrails & Compliance on One Fictional Bank

A fictional bank's assistant hardened class by class — injection to EU AI Act compliance

Role: Course Author / Instructor

View repository on GitHub ↗

Executive summary

Authored an 8-class graduate course on securing LLM-based systems — threats, guardrails, data protection, red teaming, LGPD/GDPR/EU AI Act compliance, LLMOps observability, fairness and secure production deployment — anchored on a fictional banking assistant.

  • NeMo Guardrails
  • PyRIT
  • Garak
  • Microsoft Presidio
  • Microsoft Agent Framework
  • Azure AI Foundry
  • promptfoo
Read the full case study →
Independent Projects — Teaching & Open Source Personal / Open Source

24Analytical Mind — Curated Learning Hub With a “Study, Then Build” Loop

Vetted references plus a project template per data/AI track — study, then build

Role: Creator & Maintainer

View repository on GitHub ↗

Executive summary

Built a curated, community-sourced hub of learning references and a study-project scaffold spanning the Data Analytics, Data Engineering, Data Science, ML Engineering and AI Engineering tracks.

  • Curated open-source resources
  • Markdown
Read the full case study →
Independent Projects — Teaching & Open Source Personal / Experimental ✓ Labeled experiment

25Sekisho — Zero-Long-Lived-Keys Gateway for Entra-Only Microsoft Foundry

OpenAI-SDK clients reach Entra-only Foundry — inbound key, outbound managed identity

Role: Creator & Maintainer

View repository on GitHub ↗

Executive summary

Built an experimental, stateless reverse-proxy gateway that exposes an Entra-only Microsoft Foundry deployment as an OpenAI-compatible endpoint, so any OpenAI-SDK-compatible client can use it without ever handling Entra tokens or a Foundry API key.

  • FastAPI
  • Microsoft Foundry
  • Entra ID (Managed Identity)
  • Azure Key Vault
  • Azure Container Apps
  • Bicep
Read the full case study →

Certifications

Professional certifications and credentials.

This section is ready for your certifications.

Add each credential as a card below. A starter template is included in the HTML comment—copy it for each item.

Writing & talks

Articles, blog posts, videos and conference sessions.

Articles

Add your articles here.

Videos & talks

Add your videos and talks here.

Let’s connect

Open to conversations about enterprise data platforms, AI/ML and pre-sales solution engineering.

LinkedIn GitHub