False or not false?

Eva Belyaeva, Security Vision

Why do we need this knowledge... and why we can't do without it

When investigating incidents, there are several key points of this process for analysts, which we will focus on in this article - the correctness of the investigation itself, the correct categorisation of the incident, and the priority. Properly prioritised, namely, the correct criticality of the incident, has a significant impact on the speed of the analyst's decision making and the overall effectiveness of the analyst's work.

The SOC is a set of people and processes that, like a living thing, is constantly developing and evolving. It's hard to imagine an investigative team without it. And a clear categorisation, in this case a clear system of metrics and concepts, helps to build the processes that begin when the investigation is complete. It's about preparing for the lessons learnt process, or post-incident: the playbook, regulations, and often automation may need to be edited and tuned - for example, the source or SIEM may need to be fine-tuned. And just to know where to look and what to pay attention to, an analyst conducting such work is guided by the results of the previous stages - categorisation of the incident in terms of whether it was or was not, whether it was detected correctly or incorrectly.

A lot can depend on the flag received during the response and investigation phase - both the analyst's time and the area of finding the error with further correction.

Understanding where the problem lies, how to deal with it and how to approach it, even what system the problem is in - all of this can be learnt by looking at confirmed/unconfirmed incidents and false positives.

Where do false positives come from?

Let's start unravelling the tangle by looking at what in the nature of the infrastructure can produce a false positive incident. Let's start from what we have to fix at the post-incident stage:

- Configuration issues. This can be either the data source, SIEM or any other incident-generating system (TIP, UEBA, etc.).

- Source. Often incorrect source configuration generates a flow of incorrect data or data that is correct but transmitted in error - for example, duplicate logs that are sent several times to the correlator.

- NWI. It happens that an alert that was found by an antivirus, DLP or WAF is actually the result of a failure or misinterpretation of the input data.

- SIEM. Probably the most common case is correlation rules ‘fouling’: the wrong time range here, the wrong frequency of events there - and we have a whole set of incidents in SOAR or IRP, which will go to the false positive folder.

- Lack of threat as such. There are several possibilities here:

- The investigation resulted in a legitimate user being the subject of the action - the incident is essentially unconfirmed. The user's actions were legitimate, but the pattern of those actions inadvertently matched the attacker's behaviour. It happens to the best of us when a password is forgotten by itself after a holiday.

- The action was legitimate, but at the same time there was no notification from the subject (for example, when it comes to scans initiated by IT colleagues) and the rule exception list was not populated in a timely manner.

- The incident that occurred was due to an exercise, testing a new rule.

In all these cases, the testing algorithm is roughly the same, but the ways to fix the problem can vary greatly.

How are falsehoods found and interpreted?

There are best practices and whole methods of working with FP, but we can say it in a simpler way: first of all, you should look for proofs of triggering - check source logs, parsing them for the correlator, even ask the user a direct question. Often SOCs allocate a separate process and resources for working and testing rules, writing use cases - rules are thoroughly tested on test data before they are sent to the combat system.

The interpretation is not so clear.

What is in the documents?

On paper, everything is obvious, transparent and simple: there are true threats or no threats. The first is a true positive - a confirmed incident, while the second is non-existent traffic, the result of configuration errors, some additional noise caused by incorrect threshold values, if we are talking about an adaptive system. In a few words, false is an event that didn't really happen. This is where the adventure begins.

And how do SOAR/IRP vendors propose to treat false positive out of the box?

A few options to choose from:

- Duplicating the flag from the SIEM - and this flag is manually set.

- A constructor from various incident card properties, sometimes adding an acknowledgement step and an additional flag other than FP.

- A different threat hazard metric based on scoring or criticality.

- A separate flag for error in correlation rules - and a separate boxed scenario.

It's not uncommon for SOCs or specific customers to lack these metrics and interpretations and ask to add their own.

What will the integrator configure?

An integrator can come in with either their expertise or specific TORs for existing processes and regulations. The expertise will include both regulatory recommendations, a wealth of project experience, and some sort of proprietary view of the problem. Most likely there will be two flags - FP and validation, aka legitimacy - on which the scenarios will be based.

How are customers doing?

Over time (or even immediately) the number of concepts grows into a whole ‘zoo’. What flags we encountered in the cards, how they could be identified and what response was offered:

- Not a threat. The event happened, but there is no risk to the customer - it can be ignored and the incident with this flag can be closed, and time and resources can be spent on something more significant. How this flag is set:

- By asset scoring. The incident may not have been critical, it may have been, but the location of the incident is so non-critical to the organisation or does not affect processes and overall security that the incident itself loses its significance.

- Other scoring, for example, is the result of a malware scan: if the score of such software is less than one (or any other threshold), then the incident is considered insignificant and not worthy of attention. By analogy, this logic can be extrapolated to WAF (non-critical segment, non-critical vulnerability) and other protection systems.

- Duplicates. Multiple alerts about one and the same alarm are sometimes marked ‘head-on’ - with the verdict ‘duplicate’, and sometimes closed with the label ‘false positive’. In general, there are arguments for both concepts: a more transparent verdict will help in the future when collecting statistics and building reports, as it will give an understanding of the proportion of such incidents from the total number, while an error in generating incidents indicates a failure and that in fact at the specified time at the collection point the event at the source did not occur.

- Illegitimacy/Legitimacy. This mark evaluates the user's action rather than the incident itself, and is fine on a card with an FP flag. In fact, the correlation rule may be set up correctly, and the events came as expected, but human error occurred, and the exception lists used in the correlation rules were not filled in (e.g., a scheduled privilege change for a user was displayed as an incident because there was no memo associated with the event), or the subject of the intended attack was simply doing his or her job. Evaluating the nature of the subject's action as part of the post-incident process allows for a clearer and better description of rule testing scenarios going forward.

- Miscellaneous errors. Very rarely do they appear in the form of any flags, but as a fork in the decision tree - for sure they will. After all, depending on the context and nature of errors, actions to eliminate them are determined, and again, it is useful for statistics:

- Correlation rule error. This is the most popular problem that requires special attention. It is important to identify this particular error not so much for statistics as for the scenario.

- Source error. Ghost logs, wrong timing and transmission failures - it's all here.

- Parsing error. Incorrect field mapping, incorrect card filling - and here even a tested and reliable correlation rule will start to fail on incorrect input data.

- Exercises and tests. Events and incidents that are mistakenly included in the total score, which, on the one hand, according to all parameters are dangerous triggering, and on the other hand - emulation of real threats. In this context, customers often replay FPs in the opposite direction, which can cause confusion - we have seen simulation incidents marked as FPs only because the result did not match the intended result in the use case.

Thus, the SOC in the process of investigation faces a difficult task to find a strange creature, in fact a mushroom from the world of IS - a true false positive, a real fake incident that can take any guise and interpretation. During the response phase, it is critical to set this flag correctly, as it further influences the choice of the path for the process: what to fix first, who to work with next, and how to transform related processes.

Is it possible to leave the detection of false positive incidents to AI?

We can try, but it's not an easy task.

In general, how can we offer AI to predict whether an incident from a source or correlator has occurred or not? There is such an option as detecting false through frequency and bursts of events in traffic: chaotic false positives, or events that didn't happen, go against a buggy correlation rule that generates a mass of such flagged events, one after another. But from which model to base this?

In such a situation, it seems most logical to build a model of the behaviour of event sources, determine the nature of the traffic and search for or identify anomalies: for example, there was such a case - one of our customers had a domain controller fail strictly at lunchtime for an hour, after which it sent accumulated logs with incorrect time stamps for some time, and even in double or triple the amount. Cases of such anomalies are rare, but there is a possibility that artificial intelligence can somehow learn to distinguish the real flow of events from the erroneous one.

In practice, there is a risk of losing important data and truly critical incidents. It's better to tick the box that there is no threat when you see logs that came out of nowhere.

Bottom line

So, with such a wide range of terms and approaches, what can we do? Let's go back to the very beginning - the goal of cyclic process updates - and divide the problem into two vectors by solution type:

1. Incorrect automation - we need to reconfigure a tool that for some reason produces an actual result different from the expected one. In this case, it is acceptable to build a separate playbook for one common task, as well as a set of such scripts for each type of error. Rules can be rolled out, settings can be rechecked - and then fewer and fewer FPs will come to the orchestrator.

2- Automation is fine, but human factor is at play - need to hold an extraordinary session for users and everyone involved. Once again stipulate the existing processes, test the theory in practice. If the protection systems and sources are well configured, no additional process is needed, and such a factor can be resolved in the investigation/response workflow. Properly flagged will reduce the utilisation of the analytical resource at least here.

The decision to divide the flags into two blocks - confirmations and false positives - will immediately highlight specific problems for the analyst and will prevent those that can be corrected from being hidden among general errors.