Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
Is it just me or is everyone just about fed up with this overused, overhyped term…Big Data? Data has always been big; it just got bigger, so deal with it. We didn't go through a period saying we had big software, or big Internet...so why the obsession with Big Data?
At the Big Data Roadshow in Manchester last week, Dave Elliman, principal technology consultant, and I gave the keynote speech to open up what was an interesting conference. We took the 'big' out of big data and focused the audience back onto some core Agile Analytic approaches, using retail experiences to tell the story.
As a technology community we have an accelerating growth of technologies that can deal with the big data conundrum. But before we can go out and use them or select a new tool or platform, we must make the business folks aware of the possibilities, and work together to agree what ‘building the right thing' means. We are not proposing to throw out the legacy or big data warehouses but rather simply figuring out how to now sweat those assets whilst using newer approaches by the side, on top or around to solve the new challenges of volume, variety and velocity.
The entrepreneurial tech communities are already doing this and our next generation of data scientists and analysts are using completely different tool sets to those currently in place or proposed by big vendors. They are using and creating open source tools to solve the problem. Whether it is YARN, Hadoop, cloud, or massive parallel processing, these communities are finding better, more efficient, faster ways of doing things.
A Dutch haulage firm provided a good example of how to put a value on data. This company needed to focus on fuel/carbon reduction in order to receive tax breaks so it fitted GPS devices to their several hundred vehicles and ended up with a data set that they sold back to and continue to sell to the government. Brilliant.
Which begs an interesting question, if we were to put a value on our data, could we stick it on the balance sheet? Now that would get the attention of business leaders.
Dave and I ended our session by asking the audience to reflect on the following - how much of your business today is run on human intuition versus machine learning? This cross-over will happen at some point in the future, maybe in many more years to come, but we must understand ourselves and where we are on this curve to understand the opportunities in data for our business in the future.
What is your answer?
Learn more about an Agile Analytics engagement.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
Thoughtworks acknowledges the Traditional Owners of the land where we work and live, and their continued connection to Country. We pay our respects to Elders past and present. Aboriginal and Torres Strait Islander peoples were the world's first scientists, technologists, engineers and mathematicians. We celebrate the stories, culture and traditions of Aboriginal and Torres Strait Islander Elders of all communities who also work and live on this land.
As a company, we invite Thoughtworkers to be actively engaged in advancing reconciliation and strengthen their solidarity with the First Peoples of Australia. Since 2019, we have been working with Reconciliation Australia to formalize our commitment and take meaningful action to advance reconciliation. We invite you to review our Reconciliation Action Plan.