Big Data and Security – the next big disruptor?

Last quarter I was invited to a Cloudera sales event in Las Vegas. Some impressive stats on last year’s performance, a lot of enthusiasm and in particular a great session from Charles Zedlewski @zedlewski outlining some of the product and Apache initiatives coming soon.

Two in particular are now announced

• the Open Network Insight (ONI) https://vision.cloudera.com/introducing-open-network-insight-accelerating-cybersecurity-analytics-solutions/ ONI is effectively Cloudera’s move into core security functions such as threat detection.
• Apache Arrow https://arrow.apache.org/ Arrow leverages latest SIMD (Single input multiple data) operations optimization of analytical data processing. Arrow is Cloudera’s attempt to gain control of in-memory columnar data processing.

So far, so good, but these two announcement will make a huge impact in the IT Security market. for sometime now there has been little innovation in Security. the main players are all offering incremental enhancements to technology that has been around for years.

Big Data and the Hadoop eco-system can (and already has) disrupt the ITSec market. Principally it’s a cost/scale dynamic. SIEM’s, Vulnerability Management, Configuration Management tools and others are essentially about reacting to events that have already happened. they also use Metadata structured repositories to normalize, correlate and report. Look at any SIEM vendors details and you will see this common theme. Detect and fix something that has already happened.

With Hadoop and it’s various components and, in particular, the continuing path to maturity in machine learning products, this old style architecture is going to disappear. Sometime between now and 2020 the Enterprise Security Warehouse concept will be widely adopted. All data from all sources poured into a massive data lake (in real-time of course), with an HDFS/Kudu style repository for persistence and machine learning algorithms constantly monitoring what is happening and taking appropriate action as the threats happen  not after they happen. Gartner predicted this back in 2014 so it must be true….. http://www.gartner.com/newsroom/id/2778417

In our discussions with clients we see a gradual realization, usually in the biggest clients first, that the old style Security Architectures have failed to keep up and new architectures built on big Data eco-systems and machine learning in particular, offer the greatest potential for the next disruptor. Look at how Splunk has built a $600m business on just this premise but without the machine learning part.

For an alternative view of ML and Security read Matt Harrigan’s post @mattharrigan at Tech Crunch. http://techcrunch.com/2016/02/29/machine-learning-is-not-the-answer-to-better-network-security/

What do you think, is Machine learning already the big disruptor in Cyber Security?