Big Data is the term used to label any data that’s too large, complex, cumbersome or complicated to be managed and processed by conventional technology.

To put that into a relatable context; searching Twitter can give a business unrivalled insight into their chosen market or demographic… but you certainly couldn’t copy and paste the entirety of that demographics Tweets into an excel spreadsheet!

When analysed and applied correctly Big Data can offer a business in-depth knowledge of the people using their website or business as well as offering help to predict future trends, allowing you to plan accordingly.

Whilst there’s no official definition of Big Data everyone is agreed that Big Data is BIG.

Imagine how much data your organisation must store, process and analyse on a daily basis then consider just how much more Amazon, Netflix or Facebook might have to handle.

• It’s thought that by next year there will be at least 5,200 gigabytes of data stored on every person in the world; of which only 15% will be stored in the Cloud!
• As of October ’19 around 6,000 Tweets were sent… every second! That’s over 350,000 per minute; which is just shy of 520 million tweets a day or put another way, just under 200 billion tweets a year (and that’s not even on a leap year!)
• The ‘average’ person will receive 88 emails per day and send a further 34… which equates to around 200 billion emails a day for the entire population of Earth.

As we’re sure you can imagine, that’s a lot of data to keep track of let alone glean useful information or trends from!

As we’ve already said, although there’s no one definition of Big Data the general consensus is that four separate terms can be used to define it.

Most people who talk about Big Data (cloudThing included) call these the four V’s or volume, variety, velocity and veracity.

It’s in the name isn’t it?

The sheet volume of data available to organisations can sometimes be overwhelming.  Talking about storage solutions in terms of minimum storage units just doesn’t makes sense for a lot of business as the average amount of data generated grows exponentially year on year.

As of January 2019, there were over 1.94 billion websites on the internet with Google alone processing over 7 billion searches daily worldwide.

Putting aside analysing the information or identifying trends for a moment, just capturing and storing that much data can be a challenge for many businesses.

If unstructured data (volume) is why Big Data became such an important ‘thing’, velocity is the measure of why it became so important so quickly.

Velocity can be defined as the frequency of incoming data to your business that needs processing, storing, analysing and hopefully acting on.

A streaming application like Amazon Web Services Kinesis is a great example of an application that handles the velocity of data well.

Velocity and volume are the reasons that Big Data is useless on its own. An organisation needs to have the right tools to break down the incoming and historic data they hold into actionable information.

The reason Big Data has become so big can be seen in the difference between structured and unstructured data.

Structured data can be defined as a ‘traditional’ data type. If you consider a passport for instance, they’ll all contain the same type of data, i.e. name, DOB, passport no. etc which can be easily formatted, quantified and analysed into an understandable database (even if it’s really ‘big’).

Unstructured data however are things like social media RSS feeds, audio files or images, even web pages themselves can be considered as data.

Anything on which information can be captured or stored but doesn’t have a meta model (a set of rules to define the data) can be considered unstructured.

As you can probably imagine, unstructured data has played a huge role in the rise and importance of big data.  

The goal of Big Data analysts like CloudThing is to use technology to take that unstructured data and make sense of it.

New technology has allowed us to query unstructured data as performantly as structured data which is a game changer for a businesses as it means you can now collect and store data without having to know ahead of time the type of queries, you’ll be making on it (a data lake). Then, you can build structured data warehouses downstream based on your specific needs at the time.

Perhaps one of the most important aspects of Big Data for any business or organisation is its veracity.

Collecting exabytes of information is only useful if it can be trusted.

As contrite as it may sound, not all data is good data. In fact, having data you can’t trust is correct, complete and representative is more likely to have a detrimental effect on your business than a positive one.

And that’s the four V’s of Big Data.

Oh but wait… we promised you a fifth didn’t we?

Without belabouring the point to much, collecting endless reams of data, checking its veracity and then just storing it somewhere isn’t a good use of any company’s resources.

The real benefit to Big Data comes in being able to break it down into manageable insights.

To properly harness the Big Data revolution, companies need to start building a data driven culture, making sure decision makers / analysts have the tools they need to get the answers they need out of the data, quickly and painlessly.  

That latter step is hard and is where a proper analytics function adds value.

Those analytic insights can then lead to a wide array of possible actions for a business. It may identify an untapped market for a product or even identify a need for a new product. It could result in cross selling opportunities hitherto not considered or highlight areas where cost cutting could be useful.

cloudThing are experienced in global Azure devops based deployments of Dynamics 365 (D365) that utilises a data first approach to cutting edge technology such as AI, Machine Learning, Real-time Big Data analysis and more.

If you need Dynamics 365 to do something bespoke then our in-house UX design, development and DevOps team can build an extension that will make it happen.  

We offer a fixed-price envisioning service to help you understand the potential of Dynamics 365 in your business, with a clear plan detailing how to get there.