Enterprise Data & AI · Financial Services and beyond

Diogo Miyake

Data & AI Engineer | Solution Architect | Cloud | MLOps

I help companies at enterprise-grade level turn complex data estates into secure, production-grade data & AI platforms across Azure and Google Cloud.

Featured projects
25
Focus
Data & AI
Clouds
Azure + GCP

Executive summary

I am a Data & AI Solutions Engineer with experience delivering enterprise-scale data & AI initiatives across Azure and Google Cloud. I work with data at cloud platforms since 2019 with data engineering expertise. Since August 2024, my work has spanned data monetization platforms, generative AI / LLM and agentic solutions, MLOps and production ML, FinOps and governance, and competitive pre-sales, built on Azure Databricks, Microsoft Fabric and Microsoft Foundry. Earlier at Thoughtworks, I designed near-real-time airline and retail data platforms that deepened my foundations in CDC, distributed systems and event-driven architecture. Every initiative emphasizes secure architecture, governance & compliance (Microsoft Purview, Entra ID, RLS/CLS) and production readiness (scalable design, monitoring, drift detection, FinOps), delivered through close cross-team collaboration.

About

I translate ambiguous business goals into reference architectures and working solutions that survive production. I am equally comfortable designing a federated data mesh, fine-tuning an LLM, standing up an MLOps pipeline, or leading a competitive proof-of-concept under deadline.

My engagements pair technical depth with clear storytelling: I build the demo, draw the architecture, run the workshop, and write the proposal—then package the result as a reusable accelerator so the value scales beyond a single customer.

Core competencies

A snapshot of the platforms, frameworks and practices I use to deliver enterprise data & AI solutions.

Data Platforms

  • Microsoft Fabric
  • Azure Databricks
  • OneLake
  • Delta Lake
  • ADLS Gen2
  • Azure Synapse
  • Google Cloud Datastream
  • Google Cloud Dataflow
  • Google Cloud Pub/Sub
  • Spark

Streaming & Distributed Systems

  • Fabric RTI
  • Spark Structured Streaming
  • Change Data Capture (CDC)
  • Event-Driven Architecture
  • Apache Beam
  • Azure Functions
  • Event Ordering & Deduplication
  • Data Consistency & Quality

AI & Machine Learning

  • Azure OpenAI
  • LLM fine-tuning
  • RAG & GraphRAG
  • LangChain / LangGraph
  • Microsoft Foundry
  • PyTorch
  • MLflow
  • AutoML
  • Azure ML

Governance & Security

  • Ontologies
  • DAMA BOK
  • Semantic Modeling
  • Data Mesh
  • Microsoft Purview
  • Entra ID
  • RLS / CLS
  • Conditional Access
  • FinOps / FOCUS

BI & Visualization

  • Power BI
  • DirectLake
  • Copilot

Engineering & DevOps

  • Python
  • NestJS
  • Azure DevOps
  • Terraform / Bicep
  • CI/CD

Ways of Working

  • Solution Architecture
  • Pre-Sales
  • RFP / RFI
  • Competitive Positioning
  • Architecture Decision Records
  • Stakeholder Management
  • Cross-functional Leadership
  • Workshops & Enablement
  • Partner / ISV Management

Soft Skills

  • Presentation & Storytelling (Under Constant improvement)
  • Mentorship & Enablement
  • Writing & Documentation
  • Leadership (Under Constant improvement)
  • Collaboration
  • Problem Solving
  • Negotiation (Under Constant improvement)
  • Sales (Under Constant improvement)

Languages

  • English (Fluent)
  • Portuguese (Native)
  • Spanish (Intermediate)
  • Japanese (Basic - Studying)
  • French (Basic - Studying)
  • Italian (Basic - Studying)

Data Platform, Mesh & Governance

Federated data mesh, reference blueprints and cross-platform governance.

Data Platform, Mesh & Governance Capital Markets

01Data Monetization Platform – Federated Data Mesh

Role: Lead Solution Engineer / Architect

Executive summary

Designed and led a proof-of-concept for a federated data mesh that lets a capital-markets client securely monetize curated market data as products—combining Fabric/OneLake domains, Databricks engineering, embedded Power BI and an AI multi-agent query layer.

  • Microsoft Fabric
  • OneLake
  • Azure Databricks
  • Power BI
  • Microsoft Purview
  • Entra ID
  • Azure OpenAI
  • NestJS
  • Python
Data Platform, Mesh & Governance Financial Services

11“Better Together” – Azure Databricks + Microsoft Fabric

Role: Solution Engineer / Field Lead

Executive summary

Defined and evangelized “Better Together” reference architectures that integrate Azure Databricks and Microsoft Fabric—shifting client conversations from “vs” to “and” and unlocking new Fabric workloads atop existing Databricks lakehouses.

  • Azure Databricks
  • Microsoft Fabric
  • OneLake
  • Shortcuts
  • Power BI
  • Microsoft Purview
Data Platform, Mesh & Governance Financial Services

12Azure Data Mesh Reference Architecture (Enterprise Blueprint)

Role: Architect / Author

Executive summary

Authored an enterprise Azure Data Mesh reference architecture—domain-oriented data products, federated governance and security—reused as a repeatable blueprint across multiple FSI engagements.

  • Azure Databricks
  • Unity Catalog
  • Microsoft Fabric
  • Microsoft Purview
  • ADLS Gen2
  • Delta Lake
  • Entra ID
Data Platform, Mesh & Governance Financial Services

13Federated Data Governance (Purview + Fabric + Databricks)

Role: Governance Solution Engineer

Executive summary

Integrated Microsoft Purview with Azure Databricks and Microsoft Fabric to deliver a unified catalog, cross-platform masking and an Entra ID-based access model—giving a financial-services client a single governance pane.

  • Microsoft Purview
  • Azure Databricks
  • Unity Catalog
  • Microsoft Fabric
  • Entra ID
  • RLS / CLS
  • Conditional Access
Data Platform, Mesh & Governance Airline Thoughtworks

19Airline Data Platform – CDC Modernization & Data Mesh

Role: Cloud Solutions Architect

Executive summary

Assessed and redesigned a major airline's first modern Google Cloud data platform, replacing a fragile custom-CDC direction with a managed-CDC decision path and a scalable, event-driven Data Mesh architecture for reliable near-real-time analytics.

  • Google Cloud
  • Oracle CDC
  • Google Cloud Datastream
  • GKE
  • Dataflow / Apache Beam
  • Pub/Sub
  • Data Mesh
  • Architecture Decision Records
Data Platform, Mesh & Governance Retail Thoughtworks

20Retail Fulfillment Platform – Real-Time Sensor Streaming

Role: Data Solutions Engineer

Executive summary

Designed and delivered an Azure-native fulfillment data platform for smart shopping-trolley telemetry and product master data, using custom streaming controls to achieve reliable near-real-time operations without Databricks.

  • Microsoft Azure
  • Azure Synapse Analytics
  • Azure Functions
  • Azure Data Factory
  • IoT Streaming
  • Event-Driven Architecture
  • Concurrency Control
  • Data Quality

Generative AI, LLMs & Agents

Fine-tuned LLMs, RAG/GraphRAG, multi-agent systems and enablement.

Generative AI, LLMs & Agents Banking

02Fine-Tuning an LLM for NL2SQL (Natural-Language-to-SQL Agents)

Role: Solution Engineer / ML Lead

Executive summary

Fine-tuned a GPT-4.1 model and built a multi-agent NL2SQL system that lets business users query Databricks, Fabric, SQL and Oracle in natural language—reaching ~85% SQL accuracy.

  • Azure OpenAI
  • GPT-4.1 fine-tuning
  • Azure ML
  • LangChain / LangGraph
  • Azure AI Search
  • Databricks
  • Fabric Warehouse
  • Oracle
  • SQL
Generative AI, LLMs & Agents Insurance

04GraphRAG Assistant with Azure PostgreSQL & Agents

Role: Solution Engineer / AI Architect

Executive summary

Delivered a secure GraphRAG assistant over contracts and knowledge bases using Postgres/pgvector retrieval, GPT-4, Fabric Data Agent and Foundry—answering complex questions with clause-level citations in under five seconds.

  • Azure Database for PostgreSQL
  • pgvector
  • Azure OpenAI (GPT-4)
  • LangChain
  • Fabric Data Agent
  • Microsoft Foundry
  • Azure Monitor
  • Databricks
Generative AI, LLMs & Agents Enablement & Community

08“LLM From Scratch” Workshop on Azure Platforms

Role: Instructor / Author

Executive summary

Designed and delivered a hands-on workshop on building and fine-tuning LLMs/SLMs on Azure (Databricks + Fabric)—enabling colleagues and partners to start their own domain-specific LLM proofs-of-concept.

  • Azure Databricks
  • Microsoft Fabric
  • Hugging Face Transformers
  • PyTorch Lightning
  • OneLake
  • Microsoft Foundry
Generative AI, LLMs & Agents Consumer Banking

09AI-Driven Marketing Research (Persona Generation & Multi-Agent Simulations)

Role: Solution Engineer / Prototyper

View repository on GitHub ↗

Executive summary

Prototyped an AI persona framework using multi-agent LLM simulations to generate synthetic customer personas and stress-test new product concepts before costly market research.

  • Azure OpenAI
  • Microsoft Agent Framework
  • TinyTroupe (Microsoft Research)
  • NLP sentiment analysis

Machine Learning & MLOps

Production ML pipelines, model comparison and platform evaluation.

Machine Learning & MLOps Financial Services (Payments)

03Databricks MLOps for Fraud Detection (Production ML Pipeline)

Role: MLOps Solution Engineer

Executive summary

Built an end-to-end MLOps pipeline on Azure Databricks to productionize a PyTorch fraud-detection model with Feature Store, real-time serving and drift monitoring—cutting deployment time from months to weeks.

  • Azure Databricks
  • PyTorch
  • MLflow
  • Feature Store
  • Model Serving
  • Spark Structured Streaming
  • Azure DevOps
Machine Learning & MLOps Retail

06Azure ML for Retail Churn Prediction (AutoML vs Code)

Role: Solution Engineer / Data Scientist

Executive summary

Ran a head-to-head AutoML vs custom-PyTorch churn study on Azure ML with MLflow tracking and SHAP explainability; AutoML won (AUC 0.88 vs 0.84) and was deployed as a scoring endpoint.

  • Azure Machine Learning
  • AutoML
  • PyTorch
  • MLflow
  • SHAP
  • Online Endpoints
Machine Learning & MLOps Financial Services (Enablement)

07Machine Learning on Microsoft Fabric (Data Science Experiments)

Role: Pre-Sales Solution Engineer

Executive summary

Created demos and prototypes proving end-to-end PyTorch ML workflows on Microsoft Fabric, clarifying Fabric vs Databricks positioning for FSI clients evaluating Fabric as an ML platform.

  • Microsoft Fabric
  • Fabric Data Science
  • Spark notebooks
  • AutoML
  • PyTorch
  • MLflow
  • Azure Databricks

FinOps, DevOps & Reliability

Cloud cost intelligence and infrastructure governance.

FinOps, DevOps & Reliability Financial Services

05FinOps Monitoring Solution on Microsoft Fabric

Role: Solution Engineer / FinOps

Executive summary

Built a Fabric-based FinOps solution that ingests Azure cost data, models it to the FOCUS standard, and surfaces reservation coverage and forecasts in Power BI—driving smarter reservation purchases.

  • Microsoft Fabric
  • Lakehouse
  • Warehouse
  • Dataflows
  • Spark
  • Power BI
  • Azure Cost Management
  • FOCUS
FinOps, DevOps & Reliability Fintech

10Drift Detection in Infrastructure-as-Code (DevOps Governance)

Role: Advisory Solution Engineer

Executive summary

Designed and implemented automated IaC drift detection that compares deployed Azure resources to Terraform/Bicep state and alerts on drift via Teams and email—improving the client's compliance posture.

  • Terraform
  • Bicep
  • Azure Resource Graph
  • Python
  • Azure DevOps
  • Azure Policy

Pre-Sales, Competitive & Delivery

Competitive PoCs, RFP/RFI, partners and reusable accelerators.

Pre-Sales, Competitive & Delivery Insurance (BI)

14Competing with Tableau: Power BI & Fabric PoCs

Role: Competitive Solution Engineer

Executive summary

Led competitive Power BI + Fabric PoCs that reimplemented Tableau dashboards on the client's data—cutting report load times ~40% and adding Copilot natural-language Q&A—moving clients to pilot a Tableau-to-Power BI transition.

  • Power BI
  • DirectQuery / DirectLake
  • Vertipaq
  • Microsoft Fabric
  • Lakehouse
  • Warehouse
  • Copilot
  • Oracle
  • RLS
Pre-Sales, Competitive & Delivery Financial Services / Public Sector

15Competing with AWS & GCP: Architecture & Value Propositions

Role: Competitive Solution Engineer

Executive summary

Crafted competitive target architectures and value narratives positioning the Azure data & AI stack against AWS and GCP—contributing to competitive wins and shortlists in FSI and public-sector pursuits.

  • Azure Databricks
  • Microsoft Fabric
  • Azure Data Factory
  • Azure Synapse
  • Microsoft Purview
Pre-Sales, Competitive & Delivery Banking

16RFP / RFI Support – Calculators, Architecture, Documentation

Role: Solution Architect (Bid Support)

Executive summary

Drove the architecture and technical-writing portions of major RFP/RFI responses—designing solution architectures, building cost estimates, and writing proposals that advanced bids to final stages.

  • Azure Databricks
  • Microsoft Fabric
  • Microsoft Purview
  • Azure Pricing Calculator
  • Visio / PowerPoint
Pre-Sales, Competitive & Delivery Cross-Industry

17Partner & ISV Engagements (Ecosystem Collaboration)

Role: Technical Lead / Partner Manager

Executive summary

Managed and integrated partners and ISVs across enterprise delivery—onboarding them to Azure reference patterns and acting as a quality gate to deliver seamless, on-time customer outcomes.

  • Microsoft Fabric
  • Microsoft Purview
  • Azure
  • SharePoint / Teams
Pre-Sales, Competitive & Delivery Cross-Industry

18Demos, PoCs, Pilots & Accelerators – Continuous Innovation

Role: Solution Engineer

Executive summary

Continuously delivered tailored demos, PoCs, pilots and reusable accelerators across Azure Data & AI—progressing deals and creating assets reused by colleagues across regions.

  • Microsoft Fabric
  • Azure Databricks
  • Azure OpenAI
  • Microsoft Foundry
  • Microsoft Purview

Independent Projects — Teaching & Open Source

FIAP course authorship, curated learning resources and personal open-source tools built outside client engagements.

Independent Projects — Teaching & Open Source Education (FIAP MLET Postgraduate)

21FIAP MLET Course — Data Drift Detection & Monitoring

Role: Course Author / Instructor

View repository on GitHub ↗

Executive summary

Authored an 8-class graduate course on detecting, monitoring and mitigating data and concept drift in production ML systems, for FIAP's postgraduate Machine Learning Engineering program.

  • Python
  • scikit-learn
  • Evidently
  • NannyML
  • River
  • Alibi-Detect
  • Hugging Face Transformers
Independent Projects — Teaching & Open Source Education (FIAP MLET Postgraduate)

22FIAP MLET Course — Software Engineering for Data Scientists

Role: Course Author / Instructor

View repository on GitHub ↗

Executive summary

Authored an 8-class course teaching data scientists the software-engineering practices — version control, testing, containers, dependency management, code quality and CI/CD — needed to ship reliable, production-ready code.

  • Python
  • Git
  • Docker
  • pytest
  • Linters & Formatters
  • CI/CD
Independent Projects — Teaching & Open Source Education (FIAP MLET Postgraduate)

23FIAP MLET Course — LLM Security, Guardrails & Compliance

Role: Course Author / Instructor

View repository on GitHub ↗

Executive summary

Authored an 8-class graduate course on securing LLM-based systems — threats, guardrails, data protection, red teaming, LGPD/GDPR/EU AI Act compliance, LLMOps observability, fairness and secure production deployment — anchored on a fictional banking assistant.

  • NeMo Guardrails
  • PyRIT
  • Garak
  • Microsoft Presidio
  • Microsoft Agent Framework
  • Azure AI Foundry
  • promptfoo
Independent Projects — Teaching & Open Source Personal / Open Source

24Analytical Mind — Curated Data & AI Learning Hub

Role: Creator & Maintainer

View repository on GitHub ↗

Executive summary

Built a curated, community-sourced hub of learning references and a study-project scaffold spanning the Data Analytics, Data Engineering, Data Science, ML Engineering and AI Engineering tracks.

  • Curated open-source resources
  • Markdown
Independent Projects — Teaching & Open Source Personal / Experimental

25Sekisho — Managed-Identity Gateway for Microsoft Foundry

Role: Creator & Maintainer

View repository on GitHub ↗

Executive summary

Built an experimental, stateless reverse-proxy gateway that exposes an Entra-only Microsoft Foundry deployment as an OpenAI-compatible endpoint, so any OpenAI-SDK-compatible client can use it without ever handling Entra tokens or a Foundry API key.

  • FastAPI
  • Microsoft Foundry
  • Entra ID (Managed Identity)
  • Azure Key Vault
  • Azure Container Apps
  • Bicep

Certifications

Professional certifications and credentials.

This section is ready for your certifications.

Add each credential as a card below. A starter template is included in the HTML comment—copy it for each item.

Writing & talks

Articles, blog posts, videos and conference sessions.

Articles

Add your articles here.

Videos & talks

Add your videos and talks here.

Let’s connect

Open to conversations about enterprise data platforms, AI/ML and pre-sales solution engineering.