IoT – The Data Management Challenge
With billions of sensors, smart machines, and connected devices generating and disseminating data every second, the Internet of Things (IoT) will place unprecedented demands on your organization’s data storage, processing, and analytics capabilities. Existing systems and enterprise architectures will be hard pressed to effectively manage and process all of the data coming in from these new, streaming data sources.
To put this in perspective, consider the amount of data generated by some of the connected entities or devices. A connected car generates up to 25 GB of data per hour, an offshore oil rig can generate 1-2 TB of data every day; and similarly a turbine could generate up to 1-5 TB of data every single day. Due to infrastructure constraints and data management limitations, a lot of this data is currently not being utilized or analyzed, let alone captured in many cases. In fact, according to a recent study from McKinsey, organizations manage to utilize less than 1% of the data that is generated from sensors, and 99% of the IoT data is not utilized for operational decision making.
And, by the way, it is not just the volume of IoT or sensor data that organizations have to deal with. It is also the variability and variety of data types and formats that they need to account for. Our customers today are bringing in a wide variety of data from their smart machines and connected devices – including everything from intermittent readings of temperature and pressure, to noise levels or acoustic data from turbines, and live video streams from surveillance cameras for video analytics. Data can also come in diverse data structures, schemas and formats based on these new sources, and more importantly, it may come in streams (real-time) or in batches.
Given the complexity, variability, and scale of IoT data, it opens up a fundamental set of questions about how to effectively manage and analyze IoT data streams:
- How can you effectively ingest, store, process, and drive analytics from all of this IoT data?
- How can organizations efficiently deal with the scale and variety of IoT data?
- What happens at the edge and what happens at the centralized data hub?
- What are some of the top IoT use cases and real-world examples that we are starting to see emerge today?
I recently had an opportunity to chat with Jeff Mucci, CEO & Editorial Director of RCR Wireless News, at Mobile World Congress in Barcelona, to address and shed light on some of these topics. See below a summary of the video interview with Jeff, which touches on some of these relevant questions from an IoT perspective.
A Data management platform for IoT
Cloudera delivers a platform for machine learning and advanced analytics—built on the latest open source technologies—that enables organizations to efficiently capture, store, process, and, more importantly, drive analytics, in real-time, from all the data generated from IoT.
Cloudera Enterprise enables organizations to easily ingest, process, and manage all of the data at rest, and data in motion in one unified platform. All of the streaming data from sensors can now be directly ingested and combined with data from other internal and external sources to drive insights and analytics, at considerably lower cost per terabyte.
From predictive maintenance and industrial IoT to connected cars, connected homes, fleet telematics, usage-based insurance, smart healthcare, and smart cities, Cloudera is powering some of the most compelling and innovative IoT use cases across diverse industry verticals.
For more details:
Download the Cloudera IoT solution kit
Visit Cloudera for the IoT solution page