Last February, we announced the acquisition of Xplain.io to further build out the data management capabilities in Cloudera Navigator. Since the acquisition, the team has been hard at work building on the original vision of a modern workload optimization tool, and customers have been unanimous in their interest. Even under restricted access, 15 customers have successfully used Navigator Optimizer to profile over 1.4M queries in total. Today, we are proud to announce the beta launch of Navigator Optimizer.
So why is there so much interest in this tool? To put it simply – because workload optimization is a very hard, very real problem that almost every enterprise today faces.
Enterprises generate an astronomical amount of data every day. The costs of storing and querying that data can be very high. The more business users are running queries on enterprise data, the less efficient the data management organization becomes over time. Running inefficient workloads also costs the enterprise money – in lost productivity: data architects spend their time firefighting on brittle ETL pipelines, business analysts are held up waiting for reports, ad hoc queries slow things down for everyone, and an entire team of data engineers is required to maintain complex handwritten legacy queries.
For a typical enterprise with multiple data warehouses, thousands of reports, and hundreds of thousands of ETL jobs being executed every day, this loss of productivity is a real problem. Add to all of this the complex handwritten SQL queries, and you have nearly a million queries being executed every month that desperately need to be optimized. This inefficiency is not only costing time and money for existing systems, but it’s impeding the ability to get the best results from Apache Hadoop. How can enterprises easily identify which workloads will benefit the most from Hadoop and how can these be redesigned to fit new Hadoop data models without extensive development time?
Cloudera Navigator Optimizer is built for database administrators who want to build a holistic optimization model for their SQL workloads and get peak performance from Hadoop. Navigator Optimizer looks at existing enterprise SQL and analyzes it for duplication, complexity, and compatibility with Hadoop. It then uses this data to create an intelligent optimization strategy for an enterprise’s specific business needs and workloads. Using Navigator Optimizer, customers can exploit four categories of workload optimization opportunities:
- ETL consolidation. Database administrators can combine multiple workflows to reduce the number of workflows that need to be monitored. For instance, a bank was able to eliminate thousands of temporary tables that were created by daily ETL.
- Report materialization. Business Intelligence architects can identify common access patterns to materialize into pre-aggregated tables. For example, a large telecom vendor was able to reduce the average wait time of thousands of reports by creating aggregated tables.
- Ad hoc query de-duplication. There is often major duplication in analysts’ and data scientists’ workflows. In one case, a healthcare organization was able to eliminate nearly 90,000 queries that accessed the same table with different filters.
- Complex query simplification. Queries that are thousands of lines long can be broken down into smaller components that are easier to manage. In one proof of concept, a marketing analytics company was able to reduce a 22,000 line SQL query to less than 1000 lines by eliminating common blocks of business logic.
Having proposed an optimization strategy, Navigator Optimizer then identifies the most impactful queries to focus on in Hadoop and assists in redesigning them for the recommended Hadoop data models.
Cloudera Navigator Optimizer fills an important niche in the Hadoop strategy tool belt. For more details on how Navigator Optimizer provides instant workload insights and optimization guidance, check out the demo video below:
Cloudera Navigator Optimizer is now available in limited beta. To request beta access, visit optimizer.cloudera.com. To learn more about Navigator Optimizer, register for the webinar, “Unlocking Hadoop Success with Cloudera Navigator Optimizer”