MSF: Shared Data, Without Losing Autonomy

Médecins Sans Frontières (MSF, in Dutch Artsen zonder Grenzen) provides medical humanitarian care in some 75 countries. Its missions are run by five operational centres, in Amsterdam, Brussels, Geneva, Barcelona-Athens and Paris. Each centre is independent: it runs its own projects, sets its own priorities, and has its own ERP and financial systems.

That independence is deliberate, and it works. But it also means that in one country, several MSF teams can be working side by side, each managed by a different operational centre, each with its own systems, and none of them able to see what the others have.

Take stock. A team running short of malaria tests could order them from abroad and wait weeks, while another MSF team a few hours away has more than it needs. Not because nobody would share, but because the two ERP systems did not talk to each other, so nobody knew. The same went for reporting across sections: it was manual, slow and often out of date by the time it was finished.

Pick a team and an item below, and compare what that team could see before with what it sees once the systems are connected.

We connected the systems of all five headquarters into one intersectional data warehouse on Azure Databricks. ERP systems first, then the financial systems and more, each loaded through core pipelines that we architected and coded ourselves, structured in layers from raw data to business-ready tables.

The whole platform was deployed as infrastructure as code: every workspace, cluster, permission and pipeline described in code and rolled out the same way every time. That made it repeatable, reviewable and, as we will see, easy to copy.

The hard part was not connecting the systems. It was connecting them without taking autonomy away from the teams. A single central platform, owned by one group, would have been simpler to build and impossible to accept for five independent centres.

So we built a federation. A central workspace holds most of the shared data. Any team can take a template from us and deploy its own Databricks workspace in minutes, which it controls completely, while connecting to the data in the central workspace through governed access. That also solved data residency: a workspace can run in the region the law requires, in the US for US data and in the EU for EU data, and still work with everyone else. And it left room for every team's own initiatives and experiments.

Deploy a few workspaces below and see how they connect.

A shared platform is only useful if people can find what is on it. With the catalogue and search in Databricks, every table is indexed: anyone can search for the data they need, see what it contains and who owns it, and request access.

Lineage did the rest. Databricks records how every table is built from the ones before it, so when a number looks wrong, you can see in seconds where it came from, and when a source breaks, which reports it affects. Click any table or report below.

Federation only lasts with clear rules, so we designed the governance structure: who owns which data, who may access it, and which roles decide what. But rules on paper do not reach a logistician in a field project in a remote region. How does someone there know a dashboard exists, who to contact about it, and how?

Our answer was a decentralised ambassador model. In each region we chose well-connected people: known and trusted by the colleagues we wanted to reach, and comfortable enough with the technical side. Everyone could reach us through them, and we could get messages out to everyone through them, like a web. It made communication much easier and management much simpler. Switch the model on below.

Each operational centre had its own priorities and wishes, and all of them had to be heard. We introduced agile delivery with a stakeholder model built for it: a rotating seat among the data leads of the centres. Every three months, a different data lead took the seat and could push through the vision their centre wanted, while the others knew their turn was coming.

That kept five independent headquarters aligned on a single technical roadmap, without any one of them owning it. Step through the quarters below.

The main build was delivered in less than two years. The result is one unified data environment across five headquarters and some 75 countries, replacing siloed, country-level reporting. Country and section teams kept their autonomy; leadership gained a view across all sections for the first time.

It changed how people across MSF find, view and work with data. And it was a full picture project: architecture, deployment as infrastructure as code, data governance, stakeholder management and data engineering, all in one.

• The architecture of a federated platform: a central intersectional warehouse on Azure Databricks, with independently governed team workspaces deployed from a template.
• Deployment of the whole platform as infrastructure as code, so every workspace is built the same way and can run in the region data rules require.
• The core pipelines that integrate the ERP and financial systems of all five headquarters, architected and coded by us.
• The governance structure, ownership, access and roles, that keeps the federation sustainable at scale, with search and lineage so anyone can find data and trace errors.
• A decentralised ambassador model that connects the data team with colleagues in the field, in both directions.
• Agile delivery and a stakeholder model with a rotating seat for the data leads, keeping five headquarters on one roadmap.