Plotting the data-driven journey

Categories: Data Warehouse Machine Learning Spark

“Becoming data-driven is a multi-year journey, not a simple implementation.” It’s one of the first things we tell our customers. Acquiring and using data in a way that simply wasn’t possible up until very recently, requires a huge cultural shift. It’s far more than just technology.

Last year our CTO and co-founder, Amr Awadallah suggested in The Malaysian Digest that we may be in the early stages of a multi-decade revolution. A revolution where machines will be able to use data to automate decision making much more accurately than humans ever could. In my previous blog, “Three Ways to Change Your Approach to Data,” I wrote about how distributed platforms for analytics are enabling this revolution.

To better understand this idea, let’s rewind a few decades. The late 1980’s saw the commercialization of the Internet. Businesses used this new wave of connectivity to make existing transactions (communications, sales, etc) faster, safer, and more secure. In the next decade (the late 90s), the Internet was used to combine and consolidate business processes, ultimately driving more transparency and efficiency via emerging cloud-based platforms like Now more than three decades later, we all are experiencing the Internet economy. We now live in a world where the largest taxi company does not own a single car, and the largest media company creates no original content. Instead, these companies monetized the Internet to transform entire industries.

Now let’s apply this multi-decade approach to data – currently well recognized as one of the world’s most critical resources. The good news is that we’re well underway in transforming IT to take advantage of the new data economy. Open source projects like Apache Hadoop, Spark, Impala, and many others are changing the way companies capture, store, process, and analyze big data. Several companies are commercializing the technology and providing the codification of code and services to bring it to the masses. We are seeing early adopters move quickly from playing with the technology to bringing mission-critical use cases to production.

We’ve learned from these innovators that business transformation with data happens in three predictable phases, represented by the graphic below.

Screen Shot 2017-12-11 at 10.13.50 PM

The data-driven journey often begins with simple, tactical use cases – typically around visibility. How do I look across all my silos of data? How do I find bring together that information?

Step two often involves broadening the use case to apply self-service analytics, ultimately increasing business agility. As your data engineers and business analysts are starting to recognize some key wins – whether that be in driving efficiencies or growing revenue – the company is gathering more (and better) data, broadening access to data, and deepening its experience with analytics. Both are needed steps to ultimately drive business transformation, where data and machine learning are powering the business through automation.

The path to transformation happens on different timelines, influenced by myriad conditions (business size, legacy system use, leadership, available talent, market drivers, etc). However, I’ve never met a customer that starts their data journey by transforming their business. Just as no baby runs before they crawl.

Thinking about data as a journey vs. an implementation helps reframe the conversation. It allows our customers to think through how they’re going to approach people, processes, and technology over time. It limits frustration, and presents the opportunity to try new things, and fail at them, and try something else.

I like to think about the path to business transformation with data as a multi-year journey rather than the four-decade voyage. I told Amr when his prediction came out that “I’m not going to live that long.”

Of course, he had a pithy response.

“Amy, have you heard of the precision medicine initiative using Cloudera? Don’t worry, they’ll be able to fix you all up. You’re fine.”


Leave a Reply