Bringing AIOps to Machine Learning & Analytics

Categories: Machine Learning

Two years ago I founded Hyperpilot with the mission to enable autopilot for container infrastructure. We learned a lot about data center automation based on real-time application and diagnostic feedback using applied machine learning. Last month, I joined Cloudera along with former team members Xiaoyun Zhu and Che-Yuan Liang to bring our expertise in intelligent automation to Cloudera’s modern platform for machine learning and analytics. We’re excited about this unique opportunity to push the boundary of what’s possible with data and set a new benchmark for ease of operations and cost efficiency of machine learning and analytics at scale.

Cloud-native architectures, fueled by large-scale workloads including big data and machine learning, are creating growing challenges for IT in the configuration and optimization of supporting infrastructure. Without the ability to automate configuration choices, teams create infrastructure silos supported by custom configuration arrived at by trial and error, resulting in inefficient utilization and cluster sprawl. At Hyperpilot, we witnessed these challenges in every public and private cloud customer we engaged. Despite the promise of cloud native architectures, customers lacked the necessary tools to effectively optimize their infrastructure, resulting in millions in dollars of losses due to low utilization and lost productivity. Witnessing these challenges, we focused on solving them through machine learning applied to workload and cluster optimization. We leveraged our team’s involvement in academic and industry research that demonstrated 4-5X efficiency improvement in large-scale clusters at Google, building on techniques such as configuration auto-tuning, performance isolation, and machine learning based scheduling.

Large-scale data and machine learning lie at the heart of Cloudera’s platform. Today, Cloudera’s platform runs on millions of nodes across on-premise, private cloud, public cloud, and hybrid cloud deployments. This scale represents a tremendous opportunity to apply our experience from Hyperpilot to deliver value to customers. We believe Cloudera is well positioned to unlock the potential of automation based on a strong history of innovation along with a compelling vision for a data platform for machine learning and analytics, optimized for the cloud.  The Hyperpilot team is also well at home with Cloudera’s strong engineering culture and focus on providing customers an open platform for innovation.

As my team joins Cloudera, we’re excited to build on our proven ability to bring the latest research in infrastructure automation to Cloudera’s platform. We look forward to empowering Cloudera’s customers to own the future of their business, making what is impossible today, possible tomorrow.

Author – Timothy Chen, CEO and Cofounder, Hyperpilot


Leave a Reply