Update on Apache Spot: Tremendous advancement in cybersecurity data analytics and event management capabilities

Categories: Security, Risk, and Compliance
The trend of applying machine learning and artificial intelligence to the mission of cyber defense is one of the most promising activities in the cybersecurity community. The trend towards eliminating data stovepipes to allow analysts to work over all relevant security data is also a very positive movement. Both of those trends are apparent in the new Apache Spot project.
Apache Spot is a community-drive cybersecurity project undergoing incubation at the Apache Software Foundation (ASF). The project is based on Cloudera‘s big data platform on Intel hardware, leveraging Apache Hadoop for infinite log management and data storage, Apache Spark for machine learning and near real-time anomaly detection, a suite of packaged analytics that provide tools of immediate use to any security ops team (including visualizations, analytics and machine learning tools). This is all integrated together with other tools in a way that just works. It includes a well developed data model for all relevant threat, technology and incident data. And, since it is an open platform, this model is tailorable for any unique needs.
I’ve been tracking Apache Spot for quite a while. It had its roots in an Intel project and has had great leadership and contributions from Cloudera as well as some of the greatest names in cybersecurity technology. But was very pleased to be able to get a personal demo from Cloudera’s director of cybersecurity strategy Sam Heywood during the RSA conference. There is nothing like seeing it in action and clicking buttons myself (a photo from my demo is here): 
image (2)
This is a great capability that pulls together all the relevant data that any SOC would want/need for just about every conceivable cybersecurity use case. It can be used for analysis before breach to continue improvements, can be used to assess the nature of threats that might be exploring and attempting breach, and, during or after the attack can be used to rapidly assess what is going on. Since it is based on an open data model and since great thought has already gone into most use cases it is totally extensible to just about any data source and easily tailorable to any need.
Beyond the SOC, Apache Spot will have use cases for compliance teams, hunt teams, and any other specialized function that needs intuitive access to analytical tools over all relevant security data.
If you have any market survey underway for any SIEM tool, log management tool, insider threat capability, forensic tool or other security data capability I would most strongly recommend you check out Apache Spot before making any decision. Spot provides a single consolidated platform for security data that you can put at the center of all your security operation. This is a great way to address the issue of fragmentation and stovepiping of security data that exists in all enterprises today. And, since it works with Apache Spark it comes with fantastic machine learning and artificial intelligence capabilities out of the box and since it also comes with an apps marketplace those and other solutions will only grow.
I would also recommend an in-person demo. Till you can arrange that, the next best option is to see the video below:

This blog originally appeared on March 6, 2017 on CTOVision.com by Bob Gourley.


Bob Gourley is a Co-founder and Partner at Cognitio and the publisher of CTOvision.com and ThreatBrief.com. Bob’s background is as an all source intelligence analyst and an enterprise CTO. Find him on Twitter at @BobGourley.

Leave a Reply