blogs & Things

The Common Data Model

Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​

Building a Common Data Model across a Membership Platform under a single view, with multiple lenses whilst avoiding Vendor Lock-In​

*Transcript from cloudThings recent Membership Sector Digital Conference



The  Common Data Model that I’ll be discussing today is one of those things that can often get overlooked but can also massively accelerate a Digital Transformation project if considered early enough.


I’ve been lucky enough in my career to work on some fascinating problems dealing with data, Big Data, Data Lakes and all the associated Data Architecture that comes with it and currently I’m working with an amazing team of Data Analysts and Data Scientists to find innovative solutions for the Membership sector.


If you want to avoid rework and technical debt in your project when it comes to make use of your data, one of the first steps of a project should be to design your data architecture to consider how you will be using that data – and to make sure you have coverage of all your domain entities with a data schema.  Getting this right maybe the basis of numerous workshops between different teams of stakeholders.


But what if you could skip that effort; because it has already been done for you?  Read on to find out more.



So, to begin the journey lets talk about… ‘Data as the new oil’.


That’s an idiom that’s been coined in recent years; recognising the fact that data holds tremendous value for an organisation when utilised correctly.


It’s worth protecting with secure engineering practices and if set up correctly allows you to dive in for insights as to how your organisation should be making decisions.


Unfortunately, the decision as to how data is stored (rather than used) is often left quite late into a Digital Transformation project, by which time it’s often too late to make any fundamental changes to correctly optimise it for its use in AI or ML (Artificial Intelligence & Machine Learning) projects.

The Common Data Model



That’s where the Common Data Model comes in.

In a nutshell it’s there to make sure your Data Architecture is as simple, useable and transferable as possible. Trust me… you no one benefits from a complicated Data Architecture!


Talking about Data Architecture I imagine is a little like plumbers talking about plumbing to their wives… if you’re the person in the charge of it at your business, you will feel like no one else cares until it goes wrong! Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​


That’s why I’m talking today about Data Architecture’s unsung hero… the Common Data Model (or the Common Data Service) that quietly operates behind the scenes, behind all of Microsoft’s PowerPlatform products, and why consideration of it it should be a fundamental part of your next Digital Transformation project.


Before I dive into talking about the Common Data Model specifically though, it’s worth setting the scene as to why it’s such a good thing.

Realising We are All in Debt

Debt is a word with unpleasant connotations.  In general, when it comes to creating solutions, as in life, the lower your debt the cheaper it is to deliver.



If you leave your Data Architecture as an afterthought to a project, you’ll effectively be incurring something called Solution Debt from its inception.   It’s something you’ll eventually have to pay back, most likely whilst you’re trying to make improvements to your Platform in the future.


In other words, good Data Architecture will prevent conversations with your Financial Director that sound like…


“Why does this Data Analysis project cost so much?

Why can’t I have this data in real time?

Why is our GDPR compliance so complicated?

Why can’t I find the data I’m looking for?

Why are our reports so complicated to run?”


What can you do on your next project to avoid ever being asked those questions?

Plan out your Data Architecture now, preferably using the Common Data Model as the basis.


As I’ve mentioned, far too often Solution Debt isn’t considered when planning a Digital Transformation project.


Consider for a moment…

Why do ‘disruptors’ entering a marketplace for the first time do so well against large established brands?


Surely an established organisation should have the capital reserves to pivot and reinvent themselves in a competitive market?

So why is it they so often struggle to move as quickly as a newer, younger competitor?


I would argue it’s because when an organisation is born, it is swift and agile.  It has not yet acquired the Solution or Organisational Debt that occurs as a business grows.  It may go through a merger or two, make several acquisitions or grow naturally as successful business do.

Eventually though, making a change becomes a lot more difficult because there’ll be data everywhere with duplicate data stored in different places meaning making just one, simple change involves having to access four different places.


That’s when you find can’t even move away from these systems anymore because you’re stuck in contracts that last for years with various vendors and providers… sound familiar to anyone?

It’s a difficult problem called Vendor Lock-In and it’s a problem cloudThing have a lot of experience in solving.


I will refer to a few specifically by name, as there are many different types of debt an organisation accumulates naturally as it grows, but that’s not to say with the correct architectural decisions early on you cannot aim to minimise the impact have on your organisational effectiveness. – Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​


Enter then… The Common Data Model!

Benefits Of The Common Data Model

This is where the Common Data Model shines. When implemented early in a programme, it can really ease the headaches in future change.

Benefit #1: It Helps with Data Debt



The first type of debt the Common Data Model can combat is Data Debt but to understand that we’ll need to go back to first principles and consider a concept called Data Gravity.


It’s a phrase coined originally by Dave McCrory, a wellk-known pioneer of noSQL databases back in 2010.  In a nutshell…  the more data you have in one place, the better your Data Gravity will be.

The better your Data Gravity is, the easier it’ll be to use with apps or to analyse it and garner insights from. If you have your data spread around, and siloed, you have low data gravity situation.  And it will cost you to bring it together before you will be able to fully unlock all the value held within.


If you only take one thing away from this article, I’d hope it was this as it’s such an important concept.


For instance… Have you ever been caught in a scenario where you wanted to integrate two different membership databases but couldn’t because the tables holding those members had different columns or even completely different data types, meaning it wasn’t as simple as just combining all the records together?


Or maybe you’ve got the data and you want it on your application database but can’t until the data has been sanitised. For instance, there may be missing or broken records.


Those are just a couple of examples of how Data Debt can harm your organisation.


It makes the simple act of using your own data in just a slightly different form from the one its currently in difficult. You have to pay off that debt before being able to do the things you really want to with your data. – Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​


So how does the Common Data Model help with that?


First of all, it’s true the Common Data Model is a Microsoft market design but I should point out that it’s been designed to a non-vendor specific standard that they created for all their business applications.

There are elements which Microsoft consider ‘core’ such as Account and Safety, Contact, Currency, Email, Letter, Notes etc… Things that every organisation has.

Then they’ve got segments with things like Sales, Service, Finance, Supply Chain, Commerce, Marketing, Emails and Marketing Pages.

So contact on there would be the same as your core, which is how you can share data across both of those domains.  There are tables of entities that are specifically created for that standard that include many Solutions for common sector problems. e.g. Finance, SupplyChain, Marketing, Healthcare sector.


The main takeaway I want to get across though is that any Development Team can implement this and if they follow the Common Data Model guidelines then any application that uses this standard will fit together with other apps like Lego.Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader



However… Microsoft being Microsoft they’ve have taken it way beyond that!


One of the big benefits of the Common Data Model is that there is already a ready-to-use implementation that Microsoft has built called the Common Data Service.

That means (circling back to Data Gravity for a second) if you store your data in the CDS (Common Data Service) you literally have to do nothing to be able to use the standard.  But let’s go over the other details that get uses geeks excited….

Benefit #2: The Common Data Service Talks to Almost Everything



The next benefit of the Common Data Service is that you have a way of getting data from almost anywhere.  Within the Power Platform ecosystem there are already over three hundred pre-built connectors and that includes 3rd parties like Gmail, Twitter, LinkedIn, Salesforce, DropBox with all the usual connectors into Microsoft Office 365.


Because you have so much Data Gravity in your CDS, these connectors let you assemble a phenomenal amount of value with relatively little effort.   You can literally treat it as the centre of your ‘Octopus’ with its ‘tentacles’ pulling data from all sorts of sources. – Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​

Benefit #3: Out-the-Box DataLake Integration



Whilst a clear benefit to the Common Data Service is that it’s great for building apps, where you need your data in real-time; it’s also great if you want to do very, very deep Analytics.  When you do offline analytics, you might be combining many different data sources to augment your source data, some of these may not be sanitised yet.  You won’t want this polluting your application database.  But it maybe considered ok for analysis purposes.


Step in the out-the-box feature to plug straight into Microsoft Azure’s Datalake, which is Microsoft’s store for big data and deep analytics using DataBricks.  So, you get a vital component of your data architecture for free. Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​


Benefit #4: Enterprise Ready



So, you’ve may have all your data in-one-place, but now you’ve realised it’s become a very juicy target for attack.  If you rolled your own data architecture, now you to think about designing security features.  If you forgot to do this from the beginning, that’s technical debt right there.


Well Microsoft have thought about that by building in a range of Enterprise features.

Security for instance… having all your data in one place makes it a massive target for cyber actors unless it’s very well secured.

Privacy features… Who’s got that data, who’s using it and for what?


All of these things are provided for by the Common Data Service as an out-of-the-Box solution so in the end your team needs to design a lot less than they would if they went down a customisation route.  You still need to design the appropriate policies, but at least you know that if created correctly your data is safe.

Benefit #5: Easy to Build On

Another type of debt to consider is Process Debt and again.


This is not so much debt caused by bad solutions architecture, as a natural debt that occurs as the processes in an organisation form; especially as they undergo mergers and end up with multiple systems doing similar jobs.


CDS is also at the heart of solving this through its ability to be the centre of your PowerPlatform universe and how you can use the connectors, PoweApps and PowerAutomate to ‘glue’ your disparate systems together in an orchestrated way.  Also, AI Builder, as Microsoft’s low code way of building AI automatic decision-making modules for linking to your process automation.


Within that system we can create apps, analyse data with Power BI, create bots through Power Virtual Agent and have the ability to knit together any number of systems using Power Automate. Power Automate is a great way of stringing together multi-stage processes which can link together as many different systems as you like.


A re-occurring pattern in the way cloudThing use the Common Data Service is as a single source-truth, linking into Microsoft Dynamics 365 but also other non-Microsoft systems as well. Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​



So if you’ve got five or six different payroll systems for example you might use Power Automate to keep those in-sync but ultimately it works much better if the Common Data Service is you’re single source of truth.


Imagine an email coming in from a volunteer with an expense request for example.

Power Automate can funnel it into an approval process which triggers a PO to be raised in your finance package, a response can then automatically be sent back to the requestee saying it’s been authorised. You could even imagine PowerAutomate ordering an item from a supplier if these were re-occurring orders.


Those are the types of things that PowerAutomate is good at, whilst still bringing everything back into your Common Data Model for analysis later.

Power Virtual Agent

Whilst Power Automate is for gluing together other systems, there are other tools at your disposal.

Power Virtual Agent has been designed to let you run a customer facing bot that is pre-scripted for tracking conversations and their results all in one place

Power Apps

Finally Microsoft’s Power Apps is great for building low code apps with all that data flowing straight back into your CDS.


So you can, for instance, personalise that app for a member, tracking how they engage with it so you can understand what features, activity and information they might be showing interest in and their Power BI can report on what’s occurring in those other apps you’ve built; whether through a Power Virtual Agent, through Power Automate or through Power Apps, its ultimately all stored in one place for you to analyse.


This has been a journey that I hope you have enjoyed; we started off with so fairly tenuous concept for data nerds, but we end up concluding that the strength of Common Data Model is the strength of the Microsoft ecosystem the surrounds this universe. Your organisation will undoubtedly care more about the fancy shapes you can build, but Common Data Model is the most important piece of Lego: the base that you start from. Ed Yau – cloudThing ​Solution Architect & Predictive Science Team Leader​

More blogs & Things

More blogs & Things

James Crossland in NonProfit

AI + Automation: Reducing Donor Churn & Maintaining Sponsor Interest

Churn management is a vital element of any marketing strategy, and the NonProfit sector is no exception. Knowing what to track and having a joined up view of all your donations data is vital for getting this right, and also opens the door to building innovative data-driven campaigns.   At our recent DataScience and Transformation in Charities […]

James Crossland in NonProfit

Dynamics 365 In NonProfit’s

Charities have unique funding concerns, and an obligation to spend as much as possible on their chosen cause. However, an investment in technology can offer ROI in the form of more than just improved fundraising. Dynamics 365 can help rework complex business processes, ensure compliance with stringent safeguarding and financial regulations, as well as consolidate […]

James Crossland in Tech

8 Ways Your Business Can Increase Turnover With Big Data

Understand how Big Data and Data Science can transform your business…   Big Data is the phrase that’s used to categorise any data that’s too large, complex, cumbersome or complicated to be managed and processed by conventional technology. To put that into a relatable context; being able to recommend your customers content, products or offers based […]

James Crossland in NonProfit

How To Reduce Donor Churn In NonProfits

Reducing Donor Churn doesn’t have to be a big task but does need to be a fundamental part of a NonProfit’s day to day processes   What Is Donor Churn? Donor Churn is the likelihood of an individual stopping their donations to a charitable cause for a variety of different reasons resulting in the non-profit organisation […]

James Crossland in Tech

Agile: Cutting Costs, Improving Quality & Accessing Talent

After using Agile to develop software products for several years, we thought we’d share the challenges we encountered at the start, what we did to change and the results we saw (which were ultimately uplifts in quality and efficiency)…   My development team has been using Agile to develop software product since 2007. Personally, I’ve seen many […]

James Crossland in Tech


What’s the difference between UI and UX?   Simply put UI (or User Interface) are the pages, screens, buttons, icons and any other visual aspects of a website or App that let you interact with it… or to expand on that into the non-virtual world… UI is how you experience using something – For instance in opening a fridge, […]