20Retail Fulfillment Platform – Real-Time Sensor Streaming
Role: Data Solutions Engineer
Executive summary
Designed and delivered an Azure-native fulfillment data platform for smart shopping-trolley telemetry and product master data, using custom streaming controls to achieve reliable near-real-time operations without Databricks.
- Microsoft Azure
- Azure Synapse Analytics
- Azure Functions
- Azure Data Factory
- IoT Streaming
- Event-Driven Architecture
- Concurrency Control
- Data Quality
A large retail organization operated a fulfillment platform fed by smart shopping trolleys whose beacon-like sensors captured product identifiers and weight measurements during purchases. The platform also had to integrate high-volume product master data and support near-real-time operations, while Databricks was not approved by the customer.
- Design, implement and maintain an end-to-end, Azure-native data platform for streaming telemetry and product information.
- Provide reliable near-real-time processing while respecting the customer's platform constraints and the cloud capabilities available at the time.
- Improve event ordering, concurrency handling, data quality and operational visibility for fulfillment and inventory teams.
I designed the fulfillment architecture around Azure Synapse Analytics and built streaming-ingestion pipelines for trolley telemetry and product data. I assessed the risks of relying heavily on Azure Functions for extraction and transformation, investigated several race conditions caused by concurrent processing and introduced ordering, deduplication and consistency controls. Because Azure Data Factory did not yet provide the required native CDC capability, I implemented custom Function logic to bridge the gap. I also worked closely with business and operations stakeholders to validate fulfillment and inventory tracking against the physical process.
The platform was successfully deployed for near-real-time operational analytics. Reducing race conditions and duplicate-event handling improved processing reliability by approximately 50%, while latency fell from several minutes to near-real-time windows. Better stability and observability accelerated incident investigation, enabled faster inventory and logistics decisions, and created a scalable foundation for additional IoT and sensor-driven workloads.
Streaming architecture decisions directly affect data quality and operational stability. I developed deeper expertise in event-driven systems, concurrency and the connection between retail's physical and digital processes. The engagement also strengthened my ability to work around evolving platform limitations while balancing ideal patterns, customer constraints and delivery timelines.