Lessons Learned: why it's never a shame to take on and redo everything

Eva Belyaeva, Security Vision

How we started building SOSI

This story will be somewhat personal - with one of the previous teams, in which I was once just an engineer, many years ago we started building SOC for customers, not really understanding how to do it properly.

We used everything - international standards, common sense and the desire to ‘do it well’. As a result, looking back, we see that what the centres are now and what they were then are obviously very different things. But also what the SOC we put together in 1-2 years became was strikingly different from everything that was there in the beginning - from ideology to execution.

Several times in each of the projects the very framework was redesigned - incident types, playbooks, teams of analysts. Roles, classifications, and the very approach to how to investigate incidents changed.

Initially, we laid down in each project a spiral development, a ‘Done? Redo’ for each link in that chain. That is, an integral part of any playbook was the step ‘Look retrospectively at all the steps and decide how to change this for future work’. In general, if you initially set up a playbook as a set of defined steps, you can iteratively improve them at a comfortable pace, but changing the concept itself on the fly is difficult, expensive (at least in terms of time) and, most importantly, it doesn't give any value to the project, not to mention the fact that each step has to be documented. Here you seem to have done something, but that version is never the end-all, never the best and in fact sometimes not even the fact that it works.

We've been torn between helping to set up processes and helping to close the project and fulfil our obligations to the customer, and it's actually rare that the two mean the same thing. The balance between staying on schedule, keeping the team from burning out and being able to pull multiple projects at once, and making sure the outcome is good is rarely achievable, especially the first time round.

In the end, at some point, the result was recognised as the end result, but it was really just beginning.

How all the cool ideas found their place

It's not uncommon for a few people to come up with countless innovative and good ideas during the design (or build) process; it's a shame that there aren't always the resources to test those hypotheses.

And it's a shame when the receiving end doesn't catch fire with those ideas, which is generally understandable when you're working with a team that's grabbing at everything in search of the perfect fit. In fact, something clicks when everything is done right - you look at the result of your work and don't want to change anything, because everything is in place and well done. We got the biggest return by working with visualisation and performance metrics: when you start interacting with people from the SOC-team, understand their pains and needs and take their place for a while, it becomes much easier to understand what exactly you would like to see at the moment. Later on in Security Vision it turned into the idea of full-fledged personal personal accounts for both the customer and individual analysts - and each element was selected on the basis of our personal experience of building a SOC or working in it. That is, the toolkit is not really taken from nowhere and is not written off from various regulations and orders - it is mostly experience.

A lot of things had to be postponed due to lack of time, but some of them waited for their time after years - for example, our desire not to get stuck on a shardcoded playbook and create something dynamic in real time :)

How many times did we redesign everything

back then

Honestly, a lot. How many times we started from scratch, that's how accurate it would be to say. As many times as we built - as many times as we started. But actually, no kidding, that's how it was. And every time we came to a new customer and started ‘doing SOC’, we tried to find the very ‘formula for success’ time after time. Nowadays SOC, including commercial SOC and SOCaaS, is quite widespread and has acquired its own explicit and implicit standards, but then, 8 years ago, it was difficult.

Did the experience of foreign colleagues help? Partly yes. But in the end, having worked in an international company for a while, I can personally say that the processes are very different - from the choice of tools to the approach to creating workflow. It all goes from top to bottom, from business to IS, and different needs require different realisations.

And now

Now our entire Security Vision team, of which I am a part, develops ready-made box solutions for internal processes. And how many times we have worked through the hypothesis and concept of one box all together is beyond counting.

Sometimes it is funny to compare before and after in retrospect, putting next to the process plan at the beginning and the final version already demonstrated to the customer - most often the difference is great, and the changes concern various parts of the product.

For example, we worked out several times how to implement response actions in the system in the best understandable way, and finally came to an object-oriented approach - almost the same one that was used by one foreign vendor for several years (not that one), but which did not take off due to the specifics of analysts' work in our company.

Why do redesigns arise?

hypotheses

We are always coming up with something new. We always want to find the silver bullet for everything - unique, unlike anything else, new - and do it first. This applies both to UI/UX issues, banal playbooks, workstations, and in general to the global approach to incidents and assets - sometimes working out a hypothesis requires interference in the neighbouring IS processes, working with the same IT department and business.

Hypotheses arise both from experience and from related areas - for example, from BI we have learnt a traditional approach to planning and setting up dashboards, and from some open community tools we have learnt a vector for building expertise.

expertise

Analysts do not stand still. Technologies allow us to automate routine tasks more and more, freeing up time for something non-standard. Then we are also faced with the challenge of standardising this and automating it again and again.

What used to seem like manual labour and a set of complex rules - the same classification - now finds its implementation in dynamic execution, sometimes even using ML.

The increased level of expertise also dictates a different level of requirements to the product - for example, it used to be considered a good result just to be able to take and write a connector in the system, then - the availability of a marketplace and a set of connectors somewhere, and now - a preconfigured library of integrations working out of the box.

changing world

Other vendors and specialists are not standing still either - in addition to technical approaches to incident management, there are more and more new best practice investigations in principle - where one now looks for clues, how one interacts with the same TI to get the full picture and markers of the attacker in a seemingly simple story. Not only are there new classes of solutions to integrate with, but the logic coupled with the expertise embedded in them gives more scope.

trends

For example, dealing with network forensics and new toolkits.

New methods for attackers to gain information that are now categorised differently than before.

Not to mention AI that is being used on both sides of the barricades.

How to do it well?

The most important thing is to realise that you can never do something perfectly the first time.

optimisation

Both the processes and the final details of the solution are optimised - whether it's the operation of a particular connector or even an algorithm for filling a table with data on an incident card.

The chain of steps that an analyst uses in an investigation should be looked at each time in terms of the number of attempts a person makes within a single algorithm: something is redundant, something is missing time after time, and that is what needs to be changed; combine and reduce everything that will achieve the goal faster and clearer.

Hear the customer's needs

Sometimes it's not worth redesigning something already established just because it's not optimal. Habit can work more effectively than innovation even in the long run.

And, on the other hand, you should never dismiss new ideas, even if you personally find them untenable or just silly. After testing, a seemingly innocuous desire to move a button or implement communication in a different way can significantly speed up the work process.

more automation

But only if it is more efficient than manual labour over time.

You should not automate absolutely everything in a row, without a clear plan and any proven algorithm for implementation. Sometimes in the area of expertise, automation is not possible (yet), and time can be spent on something else.

Nevertheless, speaking about the same trends and tendencies, we can draw conclusions that any habitual action of an analyst should turn into at least a button, rather than wandering in the labyrinths of another system.

Review the process on a regular basis

And we're not just talking about a specific playbook here, but the incident management process as a whole.

How people communicate, how a manager accepts and evaluates work - these all need regular review as well, in addition to technical tweaks. Sometimes communication is much more important than the under-hood engine, and a clear reward system (hello, gamification!) motivates much better than dry numbers on a dashboard.

You can't get a good and fast incident handling mechanism at the same time if you have an untouchable paper volume with flowchart printouts under a layer of dust.

Interpreting LL in the context of an incident

In fact, this whole article could have been all about that.

What is it, that final step, the last phase of NIST and the discouraging need for some to go back to the very beginning? Traditionally, bug work is the job of the analyst, often the L3 analyst combines both the investigation and this challenging process.

We are working on giving this process to automation (and there have even been attempts), but for now it is up to a person to look at the result of their work and take, for example, several hundred incidents of the same type that have been closed in one way or another, and decide at what point something was missing or unnecessary. What the investigation was struggling with each time and losing speed and content. Some changes could be applied immediately, while others would take time and require reworking an entire concept. Let's take a closer look:

Rules

This is probably the biggest challenge and pain - getting the event collection and correlation right. To optimise them correctly is a huge task, even for more than one year, and most SOCs use complex multi-level rules, ‘if it wasn't’ in addition to ‘if it was’ and so on. It's not just about handling false positives, although that too, it's also about how to properly apply the result of investigations - it might be useful to change the logic, approach and aggregation of rules, change something in the configuration of receiving events from the source.

Rules don't always lead to false of all possible problems, it can affect performance and the likelihood of getting the expected result. How many times we have rewritten rules for our first SOC is beyond counting: adapting to the peculiarities of the infrastructure, to other related processes in the company (e.g. policy management), to be able to get the same thing but faster, and so on.

infrastructure

It's a co-builder's nightmare - to make everything close to perfect, to make friends with IT and IS teams, to test all integrations and to face the fact that the usual scheme is very difficult to scale when the customer has internal infrastructure restructuring. When one protection system is replaced by another, but with a slightly different principle of operation, when network segments move apart and now the old buttons, approaches, communications - everything doesn't work.

And on the other hand, it will never work without it, because the post-incident stage and hardening just contain the need to change this infrastructure, to configure it differently and protect it more reliably.

After an investigated incident, and especially a real one, the changes can be quite dramatic, up to the point that the customer may undertake to develop an independent detection solution.

playbooks

Something that has undergone changes on a regular basis.

At first - on paper, when the real system was still far away and the company's processes were just being built. Then the logic of work and the approach to description changed - sometimes it was UML-diagrams with actors, sometimes we used BPMN-notation (as some vendors did), sometimes it was just text with numbering, and an uninformed person looking at such a document might think that it was a strange role-playing game - if you got a virus, open page 211 - there was something from the world of games in it.

After that - already in the system itself, when all the necessary milestones were defined (what policies work, where the areas of competence lie), there came the moment of testing, during which hypotheses were cut off, and specially untrained people were forced to go through all the stages like an analyst (so we understood how accessible the product was to a person with no experience, because by the end of the tests, the eye was washed out).

When that was over too, the final version was given to the trained people, and we kept an eye on the analysts - what they were looking for, where they were looking - to better tweak the visuals and make them more user-friendly.

And finally, we reviewed the fruits of our labours after the job was finished to see if everything (or almost everything) had been done wrong.

Conclusions

A lot of copies were broken along the way.

What is now the Russian SOC in a vacuum is a stack of technologies and the experience of many analytical teams, which with time and persistence have developed a certain standard of the incident management process.

Solutions and products are trying to adjust to this standard in order to catch up with the experience and open the way for something more. The main thing to remember is that redesigning is neither shameful nor sad; it is a very important step towards development and the development of something truly useful.