Measuring the effectiveness of cybersecurity processes. IS metrics. Part 1

Ruslan Rakhmetov, Security Vision

Heads of information security departments are often faced not so much with the issues of organising the performance of direct information protection and incident response functions as with the need to demonstrate the effectiveness of cybersecurity and protect the annual budgets that the company is willing to allocate to IS.

It is no secret that for many top managers all non-core functions, such as IT and/or IS, appear to be ‘expendable’ rather than ‘earning’. And if the measurement of business value and economic efficiency of IT processes can be relatively easily translated into automation, cost reduction, increased convenience and productivity, then in the case of IS processes similar parameters will not be obvious. How can one prove to the management that an IS incident did not occur due to the long and systematic work of the IS department to create internal regulations, train employees, and configure security systems? If a cyber incident did occur, but did not result in significant damage, how can you make it clear that this was due to an effective system of prompt detection and response to cyber incidents?

If the company has no doubts about the importance and usefulness of the IS department, then in such a mature environment there will be other questions: how to measure the performance of the IS department as a whole and certain employees in particular, what KPIs should be applied, how to understand the expediency and productivity of applying certain measures and means of information protection? And of course, it is better to answer all these questions not qualitatively and using an expert method, but with the help of quantitative metrics and a verifiable calculation algorithm. In this and subsequent articles, we will show how to evaluate the cost-effectiveness of the protection systems used, how to measure the incident damage reduction indicators, and also refer to the best international practices and standards dedicated to calculating the effectiveness of IS management systems.

In economics, there is such a concept as ROI - Return on Investment, or return on investment, which characterises the effectiveness of investment in business. In the context of IS it is possible to apply the parameter ROSI - Return on Security Investment, i.e. Return on Security Investment, which characterises the economic efficiency of information protection systems. It is considered that if the ROSI parameter is greater than one, then investment in a security protection system is justified, and the greater the value of the ROSI parameter, the higher will be the economic efficiency of using a particular security protection system.

To assess the cost-effectiveness of a protection system, we will use the parameter TCO, i.e. Total Cost of Ownership, i.e. the total cost of ownership of the protection system. This parameter may include the cost of the protection system itself, the cost of its implementation, technical support from the vendor, regular updates, and the salaries of the personnel administering the protection system. Protection means, as well as software and hardware, can be purchased according to two fundamentally different models characterised by CAPEX or OPEX costs. CAPEX (CAPital EXpenditure) is a capital expenditure that assumes that a company acquires software or hardware for permanent use, e.g. by purchasing a permanent licence for an IPS, purchasing hardware, ‘boxed’ software. OPEX (OPerational EXpenditure) is the operating cost of services or products that operate on a subscription model, such as cloud infrastructure, leased data centre capacity, and the use of subscription-based security software. When calculating the TCO (Total Cost of Ownership) parameter in the case of CAPEX, it will be necessary to take into account not only the one-time costs of the initial purchase of a protection system, but also its upgrade, administration, and configuration of the platform on which the protection system will operate (OS, DBMS, virtualisation systems, and powerful hardware may be required).

To calculate the ROSI parameter, you can use the formula

ROSI = (ARO*SLE*MF - TCO) / TCO

where:

ARO - annualised rate of occurrence, average number of incidents per year according to statistical data

SLE - single loss expectancy, the expected single loss, i.e. the ‘cost’ of a single incident.

MF - mitigation factor, the factor of threat reduction with the help of an IPS (in %).

The ARO value can be obtained by analysing historical data on incidents that have occurred, as well as by evaluating data provided in analytical reports from vendors and companies in the same sector. The SLE value can be obtained from a risk management system, which may take into account IT assets, their value and the one-off damage from the realisation of a particular cyber risk. The MF parameter can be provided either by the vendor or obtained independently on the basis of analyses and piloting of certain protection systems.

Let's consider an example:

The task is to select a new DLP solution to protect against data breaches in a company. Based on the data of retrospective analysis of the old DLP-system, for the last 10 years the company detected an average of 10 data leaks per year (ARO=10), and the average damage from one leak is 5000000 rubles (SLE=5000000). The value of the MF parameter obtained as a result of a three-month piloting of the DLP system in the real infrastructure of the company and in ‘combat’ scenarios of use is 50% (MF=0.5). The TCO value is made up of the cost of a new DLP-system (300000000 rubles), annual compensation of the DLP-system administrator (200000000 rubles), the cost of implementation by the IT-integrator (500000 rubles), i.e. TCO=3000000 + 200000000 + 500000 = 5500000.

The calculation of the ROSI parameter will be as follows:

ROSI = (10*5000000*0.5 - 5500000) / 5500000 = 3.54

Thus, financial investments in the new DLP-system are economically justified, as the value of the ROSI parameter is greater than one.

Let us now consider the metrics of IS efficiency from the point of view of minimising the damage from a cyberattack. No company can be 100% protected from a carefully planned cyberattack, even if it has built an echeloned defence system in good faith. Modern malware can be very stealthy during the initial infection, but the longer the malware is in the attacked infrastructure, the greater the damage will be - resulting in stolen data, damaged (encrypted) drives, and stolen money from the firm's account.

Thus, sooner or later, IS professionals will have to deal with a realised threat, and this will require cyber incident handling actions. Preparations will need to be made to repel the incident, then detect and analyse the incident in a timely manner, then perform containment, remediation and post-incident recovery operations. Finally, post-incident actions will need to be performed, analysed, response plans adjusted, and possibly defences reconfigured. It makes sense to link the predicted damage from an incident to the time to complete some of the listed active response steps: detection, analysis, containment, remediation, and recovery. Consider a cyber incident response KPI such as MTTD (Mean Time To Detect), i.e. the average time to detect (detect) an incident - the shorter this time parameter is when processing a cyber incident, the faster further action will be taken and the threat will not be able to develop to truly devastating proportions. Another metric - MTTR (Mean Time To Respond), i.e. the average time to respond to an incident - characterises the time standard for performing direct active countermeasures against the threat: network isolation of infected devices, removal of malicious objects, and restoration of infrastructure and attacked devices to their pre-attack state.

Let's consider an example:

An inattentive user opened a phishing email and clicked on the link, downloading and running a malicious file on their PC. This malware was modular, and the first component (dropper) did not take destructive actions in the company's infrastructure, performing ‘reconnaissance’ by collecting data about the infected PC, user account name, and domain name. Within 1 minute, SOC analysts received a message from the SIEM system that a suspicious email had been delivered to the company's email system containing a link to a resource catalogued by the cyber-intelligence system as phishing. Within the next 5 minutes, SOC analysts using the SOAR system queried the list of recipient users of the message, obtained data from the corporate proxy server and which employees had clicked on the link, and contacted the user, who confirmed their hunch. Within the next 10 minutes, the user's device was isolated from the network, cleaned of the malicious dropper module, the password to the user's account was changed, and a rule was created on the network screen to block connections to the malware infrastructure. As you can see, the cyber incident was responded to within 16 minutes, and the damage from this attack can be considered negligible. If these actions were not carried out within, say, 1-2 hours, then the attackers, having realised that their dropper module was in a large company, would have given the command to download additional malicious components that would have started the process of encrypting the user's device and network drives connected to the PC in order to receive a ransom for decryption, the amount of which could be up to several million roubles.

In addition to the described principle of measuring response time metrics, some companies also apply unconventional estimates of the labour intensity of successful attacks against them, for example, by comparing the cost of ‘breaching’ a company's customer data on the Darknet compared to the cost of a similar illegal service for companies in the same segment. Some organisations can also assess the effectiveness of their HR system by analysing whether insiders are offering to provide internal company information on illegal message boards. Penetration tests, the results of Red Team tests (emulation of real attackers with long preparation and systematic implementation), and data from cyber intelligence units regarding the attractiveness and difficulty of hacking a company among cybercriminals can also provide data on the effectiveness of the defence system.