MITRE's publication ‘11 Strategies for a World-Class SOC Centre’. Strategy #10 ‘Apply performance metrics to improve SOC performance’

Ruslan Rakhmetov, Security Vision

Measuring SOC performance will help the customer company understand how well the SOC centre performs its tasks, as well as identify strengths and weaknesses, and determine areas for improvement. Despite the importance of identifying priority areas and measuring performance in them, according to the authors of the publication, only half of SOC centres (among those surveyed by SANS Institute) use formalised performance metrics, so the recommendations of this strategy can be useful for both new and mature SOC centres.

Within the description of this strategy, the authors use the following terms:

- Indicator/metric: an indicator is a single value (e.g., number of incidents per month) and a metric can be a composite value (e.g., year-to-year percentage change in the number of incidents);

- KPIs (Key Performance Indicator): indicators/metrics that demonstrate achievement of key business objectives;

- Assessment: the approach, process or method of evaluation that results in the generation of indicators/metrics.

1 Elements of a SOC Centre's Metrics Programme

A SOC Centre can assess its own performance using the SOC's internal metrics programme to identify, measure, report on the achievement of KPIs of the SOC Centre's operational processes, performance results, and the situational awareness of the customer company that the SOC provides. The following elements should be considered in the SOC's internal metrics programme:

- Business objectives: the reason for measuring the performance of the SOC;

- Data sources and information gathering: information about the SOC's performance that can be used to generate indicators/metrics;

- Synthesis, data analysis: generation of metrics based on the information obtained;

- Reporting: presenting metrics to stakeholders;

- Decision making and action: further utilisation of the generated metrics.

1.1 Business objectives:

Business objectives include the rationale and expected outcomes of collecting, analysing data, reporting and performing further actions based on the generated metrics. Business objectives can be internal (for internal measurement of SOC effectiveness), external (for providing information on SOC effectiveness to customers and stakeholders), for use outside the SOC (assessment of cyber risks and IS state of the customer company as a whole based on SOC data).

1.1.1 Internal business objectives:

The audience for the results of the internal business objectives assessment will be SOC analysts, individual SOC team leaders, and the SOC as a whole. These metrics assess the speed and quality of the SOC's performance and are fairly low-level. Internal evaluation methods can be used to:

- Improve the efficiency and quality of analyst actions (monitoring, detection, investigation, etc.);

- Control the quality and stability of the SOC Centre's systems and tools;

- Continuous improvement and optimisation of SOC processes;

- Assessing and remediating deficiencies in detecting and preventing attacker TTPs (e.g., by assessing TTPs according to the MITRE ATT&CK matrix);

- Understanding and assessing the readiness of the SOC centre for new objectives, functionality, tools that are planned to be implemented (e.g. whether the SOC centre is ready for proactive cyber threat hunting, VPO analysis, implementation of deception solution).

1.1.2 External Business Objectives:

The audience for the results of the external business objectives assessment will be the customers of the SOC services: the executives standing ‘above’ the SOC, the SOC centre steering committee, IT block managers, IT service owners, and those who pay for the services provided by the SOC centre. SLA and SLO metrics are also often used in performance evaluation: SLA (Service Level Agreement) are mandatory and contractually approved metrics that can be penalised for non-compliance; SLO (Service Level Objective) are metrics that characterise the performance of the SOC that are not legally mandated. External assessment methods can be used to:

- Reflecting the state and level of readiness of the SOC centre to perform its functions in terms of personnel, processes, technologies;

- Control of compliance of SLA and SLO with the expectations of customers of SOC-centre services;

- Control of compliance of customer infrastructure with cyber security requirements;

- Demonstrating return on investment in cyber security.

1.1.3 Business Objectives for use outside the SOC:

The audience for the results of assessing the achievement of business objectives outside the SOC will be various employees of the customer company, particularly IT service owners and company executives. The means of assessing the achievement of business objectives outside the SOC can be used to:

- Monitoring the progress of the implementation of information protection measures;

- Effectiveness of the results of the implementation of information protection measures;

- The impact of SOC performance on the customer company's business, including reporting on incidents worked in the SOC;

- The contribution of the SOC to the customer company's overall cyber security and cyber risk performance.

1.2 Data Sources and Information Collection

In implementing a metrics programme, the SOC centre may use data that is either already being processed in the SOC's own systems or comes from other parts of the company. Data sources may include:

- Log Management and SIEM systems, analytics platforms;

- Ticketing systems, case management systems, SOAR platforms;

- SOC centre repositories, task management systems for DevOps;

- SOC-centre budget control systems;

- Asset management systems;

- Vulnerability management systems and vulnerability scans;

- Cyberattack emulation platforms.

The following ways of generating data for analysis can also be used to assess the effectiveness of the SOC:

- Results from exercises and cyber-attack emulations: training, exercises, critical incident response emulation will help assess how the SOC will behave in ‘combat’ conditions and evaluate the quality of processes, communication, response actions;

- Focused technical performance assessments: conducting Red/Purple Team events, using special cyberattack emulation platforms will help assess the danger of current cyberthreats and test the response to them;

- Use of frameworks to assess SOC maturity: a comprehensive assessment of SOC centre personnel, processes and technology can help provide a more holistic picture of the state of the SOC; such an assessment can be conducted using open frameworks (e.g., NIST Cybersecurity Framework or SOC-CMM).

1.3 Data Synthesis, Analysis

The authors of this publication recommend a repeatable and automated process for reporting on collected data, with data analysis results generated both regularly, according to a set schedule, and on demand. Data analysis should be automated and simplified as much as possible, based on aggregated and pre-processed information. The analyses should result in understandable and actionable information.

1.4 Reporting

The format and target audience for the reporting to be generated should be determined. The authors recommend:

- Consider audience and timing: political, financial, or technical decisions may be made as a result of SOC performance reporting, so each group of recipients can benefit from a report that is as easy to understand as possible, including a small number of the most relevant metrics, rather than overwhelming readers with a plethora of metrics;

- Take into account the possibility of further work with the report: metrics and reports may be shared between different employees of the company, so the methodology used should be included along with the report. In addition, you should be prepared to provide access to the ‘raw’ technical data on the basis of which the report was generated;

- Use the format adopted by the customer: it is recommended to use the tools used by the customer company to build the reports, which allow to automate the updating of data in the report;

- Use visualisation: the use of visualisation tools will help to better communicate information to the audience.

1.5 Decision-making and action

The result of applying the information collected, analysed and generated in the previous steps should be a decision or action. The result can be either action (a change in SOC performance) or inaction (if the metrics show that the SOC is performing according to the target values). If changes are made to the SOC centre based on the results of the performance metrics, further analysis of the metrics that demonstrate the effectiveness of the changes should also be considered.

2 Using a third-party organisation to assess SOC performance

The authors of the publication point out that some elements of a SOC centre's metrics programme may require the assistance of an external organisation to assess performance. An independent external review will assess the state of the SOC centre with fresh eyes, help identify previously unnoticed complexities, can bring in scarce expertise for evaluation, and help meet regulatory requirements. The most logical option would be to engage another SOC centre to conduct an independent performance review: this could be a partner SOC, a coordinating SOC or a national SOC; such SOC centres often offer such services, may already have a performance assessment methodology and an understanding of the SOC's context. Alternatively, specialised consulting companies may be approached who have the relevant experience and frameworks to conduct the assessment. In the case of an independent evaluation of the SOC centre's performance by an external organisation, it should be ensured that the requested information is communicated correctly, that the SOC staff and representatives of the evaluating organisation cooperate, that the externally communicated information is protected, and that the evaluation is legally secured, for example by signing a non-disclosure agreement.

3. Examples of metrics for assessing SOC performance

The publication provides the following tips and guidelines for the application of SOC performance assessment metrics:

- Not all metrics are applicable to all SOC centres and in all situations: one should focus on the size and objectives of the SOC to select applicable performance evaluation metrics;

- Some metrics may not be technically feasible in certain cases and in certain SOCs;

- Adapt the metrics used to the target audience: different individuals may be interested in different metrics, and difficulties may arise if metrics are used that are not understood by someone who does not work directly in the SOC centre;

- Use compensating controls for metrics that may be manipulated (‘chit-chatting’) by SOC staff;

- Use median and percentile calculations: these will help to reduce the impact of rare outlier events, thus providing more objective data;

- Apply graded metrics: target metrics can be more stringent for high-priority customers depending on the level of risk, criticality and regulatory requirements.

3.1 Examples of metrics for assessing achievement of internal business objectives:

- State of ‘health’ of SOC tools and technologies;

- State of ‘health’ of data sources;

- Data latency in processing by SOC tools;

- Coverage of the MITRE ATT&CK matrix by detection tools;

- Ratio of false positive to true positive alerts / incidents;

- Detection rule development rate and number of rules;

- Ratio of incidents handled by the scenario analyst using automation tools to those handled manually by the analyst;

- Volume of alerts / incidents escalated by triage to the next level that were then closed as true positive incidents;

- Volume of alerts / incidents for which no further response was conducted;

- Time spent by analysts on routine operations;

- Time spent by analysts working with the metrics programme.

3.2 Examples of metrics to assess the achievement of external business objectives:

- The level of readiness of the SOC to fulfil its objectives;

- Median/average time to detect an incident (from initial penetration of the infrastructure by attackers to detection);

- Median/average time for the SOC to respond to a customer (with a request, call, email about a suspected incident);

- Median/average incident response time (from the moment a cyberattack is detected to the execution of active response actions);

- Median/median time to contain the threat (assessing the scale of the attack and blocking further spread; with response automation technologies, this metric can be significantly reduced);

- Median/median time to eliminate the threat (clearing the infrastructure of attacker presence);

- Assignment of owners for assets (should strive for 100% coverage for all asset types and infrastructures in use);

- Asset management (should aim for 100% coverage of the asset management process, including configuration and patch management, for all asset types and infrastructures in use);

- Scanning coverage (should aim for 100 per cent coverage of assets by the vulnerability scanning process);

- Monitoring coverage (should aim for 100 per cent percentage coverage of assets by the IS monitoring process);

- Depth of monitoring (the percentage of monitoring coverage of assets at the hardware, system, application level, taking into account also the completeness of monitoring according to the MITRE ATT&CK matrix);

- Distribution of the number of incidents by customer;

- Distribution of the number of incidents by attacker (if identified).

3.3 Examples of metrics to assess the achievement of business objectives for use outside the SOC:

- Installed security updates (target value could be 90% of assets patched within 7 days for standard procedure);

- Uninstalled security updates (updates are available but not installed for various reasons; the target value should be no such occurrences unless this risk has been correctly handled);

- Presence of end-of-life software (the target value should be the absence of such phenomena unless this risk has been correctly handled);

- Utilisation of IT resources (hardware capacity utilisation, software utilisation);

- Implementation of IS training;

- Results of training phishing mailings (percentage of users who clicked on a link in a training phishing mailing (target - less than 5%) and reported receiving the mailing (target - more than 50%));

- Use and correctness of antivirus protection (e.g., coverage of the infrastructure with installed and updated antivirus and executable code signature verification tools, with a target value of 100%);

- Use of high-risk services (the infrastructure's use of services with known vulnerabilities or accessible from the Internet);

- Use of high-risk accounts (use of persistent accounts with administrative privileges).

In conclusion, the authors of the publication point out that it is often important to measure not absolute values of metrics, but to pay attention to trends and changes (e.g., month-to-month, year-to-year changes) - this will allow not only to control and timely adjust processes at the operational and tactical levels, but also to assess the effectiveness of SOC and the state of cybersecurity of the company at the strategic level. In addition, the use of SOC performance evaluation metrics can lead to negative results, such as SOC analysts' pursuit of KPIs, formalisation of work for the sake of good indicators, fear of mistakes and lack of creativity, and insufficiently in-depth consideration of complex cases. To prevent such consequences, you should discuss with the SOC-centre team the introduction of any metrics, avoid ‘cheating’ with metrics, do not overload the team with a large number of indicators, and use metrics not to punish, but to identify weaknesses and improve the work of the SOC-centre.