A data paradigm that focuses on collecting and rapidly making sense of large amounts of information from a wide variety of sources.
Big data is defined by the ‘three Vs’: volume, variety and velocity. That means handling huge amounts of data, which is being generated quickly from numerous sources. Big data capabilities are key to enabling your organization to monitor everything going on in your markets and your operations and being able to react.
What is it?
Big data is something of a catch-all term. There’s no single point where data becomes ‘big’, but big data is likely to have some of these characteristics: it is being generated in significant volumes; it is coming into your enterprise at velocity; and it is coming from a variety of sources.
Traditionally, organizations derived performance information from their internal business systems — that typically meant the data was highly structured — so it has a prescribed format, perhaps including a customer name, order number and shipping address. It was likely updated every day. So-called big data can arrive continuously and from external as well as internal sources, and is likely to be unstructured. Such data may include call center transcripts, geospatial images, weblogs or video, which are a far cry from the tabular, structured data traditionally used to gain business intelligence from.
These characteristics of big data mean traditional tools, such as relational databases and data warehouses, are ill-suited to handling big data. Instead, enterprises have turned to a new generation of big data tools, such as Hadoop (used to process large datasets), Spark (platform for large-scale SQL, batch processing, stream processing), Cassandra and other NoSQL technologies. Advances in cloud computing have fueled the big data revolution, thanks to cloud’s ease and relative cheapness when it comes to scaling.
What’s in for you?
The allure of big data is that it promises to give you better information to make decisions. It can enable you to reduce the time taken to respond to threats or opportunities. It can also super-charge your ability to experiment and innovate by using vast quantities of data to discover new patterns of customer behavior or to rapidly spot new trends.
Organizations with powerful big data capabilities can track what’s happening internally with their operations and the market at large.
What are the trade offs?
Becoming adept at handling big data isn’t easy. You’ll likely need to employ highly sought after data scientists, which can be expensive.
You also need to consider how suited your organization is to handling big data. For data-driven internet companies, such as Google, Facebook and Netflix, the need for big data capabilities is a given. If you aspire to become a data-driven enterprise, you’ll need some form of big data platform to improve your analytical capabilities.
But many traditional enterprises run on critical legacy systems that were never intended to use big data. In such cases, a move to become more data-driven may well be desirable but it doesn’t necessitate making significant investments in big data immediately.