Architecture, tools, and culture of secure software development

Ruslan Rakhmetov , Security Vision

In this article, we, as a software developer, want to talk about DevSecOps (Development, Security, and Operations) and how this process works in practice. On the one hand, market competition and business demands the continuous release of new features, maximum fault tolerance, and immediate response to user requests. This process can be broken down and automated, as historically, development and operations have been combined into a single DevOps cycle to maximize the speed of value delivery.

In this hyper-fast pace, information security (IS) has long remained an isolated stage, traditionally applied at the very end of the development lifecycle, just before release. This approach (often referred to as "bolted-on security") created critical systemic inconsistencies. Security specialists became a bottleneck, blocking releases due to vulnerabilities discovered at the last minute, while developers perceived the IS department as an obstacle to innovation and KPI achievement (we discussed this in the article on the resource-service model). Traditional scanning tools couldn't cope with the pace of CI/CD, generating a huge number of false positives and requiring time-consuming manual analysis, but the focus on cybersecurity and the drive for automation added a third factor (security) to the process.

When preparing a complex culinary dish, the chef tastes the sauce, checks the freshness of the ingredients, and monitors the temperature at every stage of the cooking process, not just before serving. Quality control here works like IT security; it's continuously integrated into the cooking process itself. Imagine car production on an assembly line: in the modern auto industry, crash tests and brake system checks aren't postponed until the car is assembled and leaves the production line. Numerous sensors and quality control systems inspect every component directly during the step-by-step assembly process. Dev, Sec, and Ops work in sync.

The foundation of modern software development architecture is the Secure Development Life Cycle (SDLC). Shift Left Security is critical to the implementation of this cycle. It is the shifting of security focus to the earliest stages of design and coding (left on the scale from planning to operations). Shifting security to the early stages of development means beginning to analyze the system, identify vulnerabilities, and implement policies when the cost of making architectural changes and fixing software bugs is minimal.

This works like document checks before a vacation, where we check for ticket availability, passport expiration dates, and hotel reservations early on, rather than upon arrival at the airport check-in counter, where fixing the error would be extremely expensive and stressful. Similarly, a technical vulnerability discovered during the requirements gathering or architectural design phase requires minimal effort to fix (the same vulnerability discovered during active coding requires a developer's time to rewrite the function). However, if a critical logic or security error makes it into production and is discovered as a result of a cyberattack, the cost of fixing it increases hundreds of times, including direct financial losses, regulatory fines, reputational damage, and the cost of emergency fixes (hotfixes).

Shift Strategy Left Security not only speeds up the fix process, but also creates a continuous learning cycle: the developer sees their error in the context of the code they just wrote, understands why the security rule was triggered, and is less likely to repeat it in the future.17 This prevents the accumulation of so-called “security debt”.

Mature organizations also recognize the importance of the Shift Right concept, the approach emphasizes that certain classes of vulnerabilities, particularly those related to environment configuration, behavioral anomalies, and complex business logic, may only become apparent after the software is deployed in a live environment and is used by customers. This approach includes continuous monitoring, the use of Runtime Application Self-Protection, RASP, threat intelligence collection and conducting regular exercises of "red" teams (Red Teaming).

An effective DevSecOps process is a closed loop where data from production is instantly fed to developers and architects to adapt scanning rules and security policies.

The central lifeblood of a modern software factory is Continuous Integration/Continuous Delivery, CI/CD. Embedding security into this pipeline and comprehensively protecting the pipeline transforms DevSecOps theory into a working, scalable practice.

Imagine security at a modern airport: passengers and their luggage pass through sensitive metal detectors and X-ray machines. Security is seamlessly integrated into the passenger journey without having to shut down the entire airport.

Thus, security automation solves two problems at once:

 - Firstly, automatic launch of a whole arsenal of scanners (SAST, DAST, SCA, IaC) at the appropriate stages of the build to identify vulnerabilities in the product code;

 - Secondly, protecting the pipeline itself (CI/CD Security), which has become a tasty target for hackers today.

Security includes strict secrets management (using dedicated storage rather than hard-coding tokens in environment variables), using an immutable infrastructure, and implementing the principle of least privilege (RBAC) for build agents.

The main engineering challenge when embedding security scanners is to implement the checks painlessly, without slowing down the feedback loop or angering developers. Security should feel like a co-pilot, not a traffic cop.

During the initial stages of integration, any new scanners should operate exclusively in audit and monitoring mode, where they provide developers with analytical metrics and alerts but do not abort the build due to the thousands of inevitable false positives. The audit period is used by information security engineers and Security Champions. For fine-tuning rules, creating baselines, and generating exception lists. Only after the noise level has been reduced to an acceptable minimum and the team has become accustomed to the tool are critical checks switched to a hard blocking mode (with blocking occurring only when new critical vulnerabilities are discovered in user code that are realistically reachable and exploitable).

You can't manage what you can't measure, and this golden rule applies to secure development as well. Therefore, it is recommended to introduce key performance indicators (KPIs). Traditional metrics such as hours worked or the total number of vulnerabilities found don't reflect real risk reduction and often conceal operational inefficiencies. Leaders (CISO, CTO) need to focus on metrics that demonstrate the impact of security on delivery speed and the actual security of the product. Therefore, KPIs in a mature DevSecOps culture include:

1) Mean Time to Remediate (MTTR) is the average time from the discovery of a vulnerability to its complete mitigation and patch deployment. This is the key metric for operational efficiency. Successful teams with active Security Champions see this metric drop significantly (sometimes by 40-50% per quarter).

2) Repeat Violation rate (RVR) is a metric that shows how often the same classes of vulnerabilities (e.g., XSS or password hardcoding) appear in new commits. A decrease in this metric directly indicates that developer training is paying off.

3) DevOps Research & Assessment (DORA), frequency deployments. If the implementation of information security tools reduces the release frequency (the pipeline "stands still" due to long scans or an abundance of false positives), DevSecOps integration has been implemented incorrectly. Security should not hinder speed.

4) False Positive Rate (FPR) is when security alerts prove irrelevant. A high FPR is the main driver of alert fatigue among developers, while a decrease in this metric indicates high-quality configuration of information security tools.

5) Security Issues by Stage (SIS), defect density by stage, for example, a decrease in the number of bugs found at the external pentest stages (where fixing is already expensive).

DevSecOps isn't a one-time initiative to purchase licenses for expensive vulnerability scanners, but a complex, continuous process of organizational and technological improvement. Experience from hundreds of leading corporations shows that attempting to immediately include all blocking checks in CI/CD inevitably leads to development paralysis, engineer rebellion, and the discrediting of the very idea of security.

Successful (and painless) implementation requires a delicate strategic balance: starting with scanners operating exclusively in audit and metrics collection mode, gradually adapting the tools to the company's unique stack, carefully configuring false-positive filtering, and only then gradually enabling blocking rules for critical threats.