It’s time for security teams to embrace security data lakes

The average corporate security organization spends $18 million annually but is largely ineffective at preventing breaches, IP theft and data loss. Why? The fragmented approach we’re currently using in the security operations center (SOC) does not work.

Here’s a quick refresher on security operations and how we got where we are today: A decade ago, we protected our applications and websites by monitoring event logs — digital records of every activity that occurred in our cyber environment, ranging from logins to emails to configuration changes. Logs were audited, flags were raised, suspicious activities were investigated, and data was stored for compliance purposes.

As malicious actors and adversaries became more active, and their tactics, techniques and procedures (or TTP’s, in security parlance) grew more sophisticated, simple logging evolved into an approach called “security information and event management” (SIEM), which involves using software to provide real-time analysis of security alerts generated by applications and network hardware. SIEM software uses rule-driven correlation and analytics to turn raw event data into potentially valuable intelligence.

Although it was no magic bullet (it’s challenging to implement and make everything work properly), the ability to find the so-called “needle in the haystack” and identify attacks in progress was a huge step forward.

Today, SIEMs still exist, and the market is largely led by Splunk and IBM QRadar. Of course, the technology has advanced significantly because new use cases emerge constantly. Many companies have finally moved into cloud-native deployments and are leveraging machine learning and sophisticated behavioral analytics. However, new enterprise SIEM deployments are fewer, costs are greater, and — most importantly — the overall needs of the CISO and the hard-working team in the SOC have changed.

New security demands are asking too much of SIEM

First, data has exploded and SIEM is too narrowly focused. The mere collection of security events is no longer sufficient because the aperture on this dataset is too narrow. While there is likely a massive amount of event data to capture and process from your events, you are missing out on vast amounts of additional information such as OSINT (open-source intelligence information), consumable external-threat feeds, and valuable information such as malware and IP reputation databases, as well as reports from dark web activity. There are endless sources of intelligence, far too many for the dated architecture of a SIEM.

Additionally, data exploded alongside costs. Data explosion + hardware + license costs = spiraling total cost of ownership. With so much infrastructure, both physical and virtual, the amount of information being captured has exploded. Machine-generated data has grown at 50x, while the average security budget grows 14% year on year.

The cost to store all of this information makes the SIEM cost-prohibitive. The average cost of a SIEM has skyrocketed to close to $1 million annually, which is only for license and hardware costs. The economics force teams in the SOC to capture and/or retain less information in an attempt to keep costs in check. This causes the effectiveness of the SIEM to become even further reduced. I recently spoke with a SOC team who wanted to query large datasets searching for evidence of fraud, but doing so in Splunk was cost-prohibitive and a slow, arduous process, leading the team to explore alternatives.

The shortcomings of the SIEM approach today are dangerous and terrifying. A recent survey by the Ponemon Institute surveyed almost 600 IT security leaders and found that, despite spending an average of $18.4 million annually and using an average of 47 products, a whopping 53% of IT security leaders “did not know if their products were even working.” It’s clearly time for change.