Metrics: their charms and insidiousness

Maxim Lunev, Security Vision

They have surrounded us

Metrics have firmly entered our lives. Finance and sports, business and social networks - it is almost impossible to find an area where one or another quantitative characteristic or another would not solve much, if not everything. Metrics ruin careers and put them on a pedestal, depress and promise golden mountains, they can become best friends or worst enemies. The world of information security has not been spared.

More and more modern SOC-centres are actively implementing monitoring and measurement of their efficiency in their activities. This allows them to quickly identify and eliminate bottle necks, assess the quality of response teams, identify their weaknesses and draw up a development strategy. And, of course, to illustrate your value to the company's management, justifying all the investments you have made.

Where to start?

It doesn't take long to get lost in the world of statistics. There are many types of metrics and methods of using them, but the most common ones will cover most of the needs:

• Absolute
  
• Relative
  
• Averages
  
• Various distributions and their parameters
  

The first (and most important) rule of metrics use should be emphasised here: any quantitative metric is only valuable in context. For example, absolute metrics are generally regarded with scepticism. Indeed, the total number of incidents per month or the number of false positives alone will not tell you anything. But the total number of unclosed incidents at the end of the reporting period should attract attention and require additional scrutiny. The measurement period also plays an important role: for short time periods (e.g., one shift on duty), absolute metrics can accurately describe the picture of what is happening (total number of incidents, number of closed incidents, number of false positives, and so on). Over longer stretches, it is more reasonable to rely on relative values.

Relative metrics have a reputation for reliability. Based on them, it is easy to build a system of targets and monitor their fulfilment. For example, exceeding a set percentage of wrong verdicts by analysts when analysing incidents may indicate insufficient skills of the team and the need for additional training. Or a high percentage of false positives will immediately point to errors in SOC configuration settings or an insufficiently reliable external source of events.

At the same time, you should not abuse the desire to improve this or that indicator at all costs. Let's introduce a second rule of thumb for metrics: most metrics have an optimal value from which to base their performance. A good example is the number of phishing incidents. Let's say your SOC was experiencing 10 phishing incidents per month, and all of a sudden that number goes to 0. This could indicate that the attacker has found your defences impenetrable and given up, but it's also possible that they have succeeded and gained access to your organisation.

Is timing everything?

Time has become incredibly important. Anything too late, any delay and the consequences can be dire - both financial and reputational. Of course, the issue of responding to information security incidents is no exception. The faster an incident is detected, the faster the response and remediation, the less damage to the organisation.

The timeline for the existence of any incident is broken down into several key points. Threat detection, prioritisation, response and remediation times are just the most popular metrics, although a full breakdown of them is worthy of a separate article. Here I would like to focus on how to handle them and avoid falling into statistical traps.

Of course, all these time parameters should be averaged for a clear picture of what is going on. It is necessary to remember that there are three types of average: arithmetic mean, median and mode. Let's discard the mode at once, it is the value that is most often found in the sample, and when we are talking about indicators with accuracy to the second, its use will not be fully correct.

The arithmetic mean is the best known mean (and very easy to calculate), and many people automatically use it. In some cases it is justified, for example, if you need to estimate the speed of taking an incident to work, which is limited from above by a strict KPI of the organisation. But even in this case, you need a separate metric that will capture the violation of this KPI. For example, the number of overdue incidents per week (note that using a metric with an absolute value is fine here).

However, using the arithmetic mean will immediately distort the picture if there are several extremely low or extremely high values in the sample. Indeed, an average incident handling time of 10 minutes may mean that all incidents are handled in the range of 8 to 12 minutes, or it may mean that all incidents were handled in 5 minutes and one was handled in an hour. In such a situation, the median average comes to the rescue. This is the value that is in the middle of the sample (i.e. 50% of the values are below the median and 50% are above it). Then for the second case from the example above, the median average incident processing time will be equal to 5 minutes, and the extreme value of an hour will be discarded. Well, in the first case, the median average will simply be equal to the arithmetic mean (10 minutes) and will fairly objectively describe the picture of what is happening.

It should be said that the median average is most often applied to the time values of the incident cycle, but it is always necessary to specify in a report, presentation or dashboard which average is used to avoid confusion and misinterpretation.

Mean error and the importance of ‘long tails’

And then the question arises: is the approach of neglecting extreme values valid? A famous American scientist Stephen Gould received a shocking diagnosis from doctors with a prediction to live for about 8 more months. After delving deeper into the question, he realised that he was talking about the median average, which predicts that 50% of patients will not live to 8 months and 50% will live longer. How much longer - the median no longer ‘knows’. Gould lived another 20 years and wrote the famous article ‘Don't Trust the Median’ which has been a source of hope for many patients.

Returning to our SOC, consider a threat that could not be contained for several hours with an average median containment time of 10 minutes. Obviously, such an incident requires increased attention because the potential damage from it can be very painful. This is precisely the case where the average is deceptive and overlooks valuable information. The best solution is to set an acceptable deviation from the mean and further analyse cases that fall outside this limit. Or stick to the practice of analysing the top 5% of values that deviate from the mean. These are the ‘long tails’ (originally named after the shape of the normal distribution), which are rare but often have the most devastating consequences.

Such analyses are more complex and require some effort and expense. It can be useful to graphically represent such samples and visually observe significant deviations from the usual norm.

How many metrics will we need?

Unfortunately, there is no universal answer to this question. Finding meaningful metrics is not an easy task. A quality metric should:

• Answer a specific question
  
• Be transparent and unambiguous to the target audience
  
• Be collected or calculated in an automated manner
  
• Be easy to visualise and report

A sensible approach is to use a GQM (goal-question-metric) system, where goals are formulated, the right questions are asked, and metrics are developed to quantitatively describe the achievement of these goals. This creates a direct link between the quantitative metrics and progress towards the goals. It also makes sense to break down the metrics by target audience and create a separate dashboard or report for each. Accordingly, different aspects of SOC activities will be covered by their own metrics, clearly understood by those who operate them. Expert analytical dashboards can contain more data (but preferably no more than 10 metrics), while presentations for management should be limited to 3-4 key metrics.

Conclusion

The popular quote ‘What gets measured, gets managed’ attributed to Peter Drucker confirms the importance of metrics in any area of life. This phrase has a now forgotten sequel: ‘...even when it is pointless to measure and manage, and even when it is detrimental to the organisation's objectives’, which contains a kind of warning against over-obsession with metrics. Indeed, with the right approach and a certain amount of effort, metrics will quickly come to your side and bring tangible benefits, but they should not be abused, nor should you forget their insidiousness and carefully avoid all the traps they set.