Staedion: From a 40-Hour Refresh to Under an Hour

Staedion is a major Dutch housing corporation based in The Hague. Behind every home, tenant and repair is a set of operational systems, with Dynamics Empire, the property management system built on Microsoft Dynamics, at the core. Those systems ran on premise, and they were tied together point to point: every new link between two systems was another custom connection to build and maintain.

For the business that meant data in silos. Answering a question that spans systems meant waiting for an export, a refresh, or both, and the refreshes themselves took the better part of two days.

We were brought in to build a modern Azure data platform and unify the integration layer. SolvX was responsible for the data engineering build, alongside our delivery partner LabsData. We moved Dynamics Empire's data from on premise into the cloud, brought it into the data warehouse, and structured it there.

The platform has two halves that work together. On the operational side, an Enterprise Service Bus replaced the point-to-point integrations with one event-driven layer, with Azure Function Apps and webhooks handling events in real time. On the analytical side, pipelines in Azure Synapse and Data Factory feed Azure Databricks, the core of the new platform, organised in a medallion architecture of bronze, silver and gold layers.

Two things made the refreshes slow. Every run moved all the data again, changed or not. And every connection to the source database was rate limited: one connection could only send so many rows a second, so a pipeline that used one connection at a time spent most of its life waiting.

The obvious fix, opening as many connections as possible, does not work either. Each extra thread costs CPU, each open connection costs money, and past a certain point the database starts pushing back with throttling and retries. Too few connections and the refresh takes days; too many and it gets slower and more expensive again.

So we measured instead of guessing. We first made the loading smart, as at POM: import only what changed, and fill in smartly where that is unknown. Then we tested exactly how much CPU each thread needed and how many rows each rate-limited connection could send, and opened exactly the number of parallel connections at which time and cost were both at their best.

Try it below. Move the number of connections and switch smart loading on, and watch the refresh time, the cost per refresh and the CPU find their balance.

With the loading made smart and the connections tuned, Staedion's pipelines went from 30 to 40 hour refreshes to under one hour. Data that used to arrive the day after tomorrow now arrives within the hour, and the business works from one unified Azure architecture that connects the operational and analytical estate.

We did not build it and leave. We worked alongside Staedion's own team, set up the architecture together, and made sure the knowledge stayed in house.

• Introduced the medallion architecture, and why bronze, silver and gold layers make data easier to trust and to change.
• Brought in agile working in Azure DevOps, with the backlog, sprints and pipelines in one place.
• Trained the team in Databricks, so they could build and run the platform themselves.

• Dynamics Empire's data moved from on premise into the cloud, into the data warehouse, and structured there.
• Azure Databricks implemented as the core of the new data platform, in a medallion architecture.
• An Enterprise Service Bus that replaced point-to-point integrations with one event-driven layer across operational systems.
• Azure Function Apps and webhooks for real-time event handling.
• Data pipelines in Azure Synapse and Data Factory feeding the analytical layer.
• Smart loading and parallel connections tuned for the best balance of time and cost: refreshes from 30 to 40 hours to under one.