Organisations are increasingly looking to security analytics to help identify suspicious events and behaviours. While it is common to assume that breach detection via security analytics is a straightforward matter of sending logs to a server for analysis, though, the reality is not so simple.
Comprehensive logs are indeed the essential raw material for any security analytics engine, but extracting actionable intelligence from them is complex. Here are five of the most common issues to be aware of.
1. Sheer Weight of Traffic
Network devices, endpoints, security systems, applications, storage devices and proxies all generate prodigious amounts of log data. Most organisations log hundreds of millions of events every day, with diverse device types and vendors writing log entries in different formats.
2. You Can’t Just Analyse Them
Before analysis can be undertaken, logs must be parsed to identify what they are describing – users, devices, logon events and so on. Parsing raw logs puts the information they contain into a common format so that objects from one log can be related to those from others. Because logs are written in various formats, each requires its own parsing algorithm.
3. Logs Lack Context
Parsed or not, no log can tell you anything beyond the data it contains, and they offer no context. However, contextual information is essential for security analysts to prioritise and investigate log entries flagged as alerts. Tracking such information – who the user is, what they do and where they are geographically, for example – can be time-consuming, delaying threat responses and driving costs up.
4. Individual Events Lack Context
Just as logs lack context, so do the events recorded in them. Security events develop over long periods of time, but each log entry is a discreet island of data, disconnected from earlier events and those happening on other systems
For example, a log entry might show John Smith logging into the company network from his desk in the head office at 8 am on a Thursday morning. If John Smith happened to be on a business trip on the other side of the world that week, this login would be highly suspicious. If was expected in the office that day, and had swiped in ten minutes earlier, it wouldn’t. Analysts often need to review thousands of events in order to build up sufficient context to understand a single incident.
5. The Trail Goes Cold
Data access activity is often left unlogged. As a result, when asking the most critical question of all – “Is our data safe?” – the trail has often gone cold by the time analysts inspect it. User interactions with files and emails, for example, are in many organisations not captured, despite being the subject of many data breaches.
Because of these and other pitfalls, raw logs are of limited use in terms of raising meaningful security alerts, and extracting useful intelligence from them is time-consuming and labour intensive.
Correctly applied, security analytics can overcome these pitfalls, reducing false positives, accelerating investigations and stopping attacks more quickly. Here’s a brief overview of how.
Simply passing raw log data to your analytics service is wasteful and rarely productive. Pruning and parsing logs at the point of collection can reduce data volumes by some 70-80%, as well as pre-preparing the data for analysis. An intelligent collector can also perform initial analytics and alerting, accelerating responses to certain security events.
Enrichment and Analytics
As we have noted, to differentiate between innocuous events and genuine security alerts, context about users, systems and data is essential.
Machine learning can be highly effective in developing context over time, building and maintaining baseline pictures of ‘normal’ interactions between users, systems and data. Security analytics engines can then use this baseline to detect deviations from the norm.
Meanwhile, data-centric technology providing context about data usage and sensitivity can be a powerful aid to answering that all-important question, “Is our data safe?”
With context established for individual events and over time, security analytics engines can sift genuine security alerts from false positives far more effectively, presenting analysts with fewer, more meaningful alerts which are easier and quicker to analyse, accelerating threat responses, reducing the damage resulting from any actual breaches, and improving the overall security stance of the organisation.