DevSecOps vs Traditional: Winning the Software Security Game

This game theory analysis illustrates that collaboration, rather than compartmentalization, is the key to achieving the best outcomes in today's complex software development landscape which is a growing target for sophisticated attacks at increasing frequency.

2024-09-09

Vikrant Kahlir - Consulting Partner

Ransomware attacks are not only costly but also disruptive, with businesses experiencing an average of 21 days of downtime and full recovery often taking even longer. As software systems become increasingly distributed and decentralized, relying heavily on open-source and third-party libraries, traditional security approaches are proving insufficient.

In Q2 2024, Check Point Research reported a 30% year-over-year increase in global cyberattacks, with organizations facing an average of 1,636 weekly attacks. The MIT AI Risk Repository published a database of 700+ ways AI can go wrong. Research from a leading cybersecurity university highlights the following alarming trends:

/ 5

CISOs see more threats in tough times.

surge in cloud environment intrusions.

organizations feel confident against cyberattacks.

In light of these emerging trends, organizations must enhance and prioritize their software security measures to effectively mitigate potential risks and protect sensitive data.

The Importance of Software Security and the Need to Evolve

With increasing reliance on third-party components and the growing sophistication of digital supply chain attacks, software security is more critical than ever. Organizations face the challenge of securing their software without hindering development speed, making the choice between traditional security and DevSecOps pivotal.

Given these evolving threats, it’s time to rethink and evolve the role of the security team in software development. The choice between traditional security models and DevSecOps approaches has profound implications for organizations.

In this article, we explore the practices and concepts of DevSecOps and discuss its strategic importance to organizations' branding, profits, and revenue. Using the principles of game theory, we explore the winning strategy for enterprises in the age of digital and AI that enables decision-makers the rational to analyze and improve the security of their software supply chain via the evolution of traditional security practices.

Shift Left in Software Development

Before diving into DevSecOps, let's understand the origin and genesis of a concept that gave birth to the idea of DevSecOps. The "shift left" approach prioritizes early detection of defects, inspired by the Toyota Production System. The use of Test Driven Development and DevOps formalized the concept of Shift Left in Software Development.

The idea of Shift Left is that fixing defects early in the software development life cycle (SDLC) is generally more cost-effective than fixing them later. The cost of fixing defects can increase exponentially as the software progresses through the SDLC. For example, according to IBM, fixing an error after a product is released can cost four to five times as much as fixing it during design and up to 100 times more than fixing it during maintenance.

Traditional Approach

Testing and security checks are done at the end of the development process, which can be costly if issues are found. The repetitive security checks are mostly manual.

Shift Left Approach

Shift Left promotes the integration of security controls throughout the SDLC and encourages automation of rpetitive controls using tools built into the code pipeline. It introduces the concept of security as code throughout the development process, ensuring compliance without hindering innovation.

The Culture, Practice, and Tools in the World of DevSecOps

DevSecOps isn’t just a technical concept; it’s a cultural evolution. It changes how we build, secure, and deliver software. For folks who have already mastered DevOps, they should not stop. They should ask themselves what the next opportunity to improve is and dive deeper. The ideal state is to achieve that state where a developer blindly commits a change and pushes it through the pipeline, and it won’t break anything.

Key Practices

Several key practices that will help navigate the DevSecOps landscape include:

Infrastructure as Code(IaC): Think of it as a blueprint that ensures security configurations are consistent and reliable. IaC is powerful because it allows you to automate configuration and adapt quickly to feedback loops.
CI/CD Integration: Weave your security checks seamlessly into your build and delivery pipeline to ensure that security always remains top of mind. You don't have to do it; your smart code pipelines will do it for you. Your target should be 100% automation of all repetitive tasks.
Automated Testing: As the code progresses through SDLC, the cost and the time complexity of testing increases. Let’s compare Unit Testing versus Pen Testing. Unit Testing happens in minutes without delays, and the cost is developer time and machine time. Pen Testing is a sophisticated endeavor that can take up to 4 weeks and has to be planned ahead. The tools and talent used are significantly costly. It is important to find opportunities to automate what is simple and to make the automation of complex tasks possible by defining them more objectively. However, automated tests don't yet beat the performance and effectiveness of manual pen tests done by the pen test team or by the red team. It is so because humans react to discovery at every step and follow specific paths that machines find hard to discover. The value of automation here is that automation frees the red team time so they can go deeper into specific paths.
Threat Modeling: The proactive approach helps us identify potential weaknesses before they become problems. Security teams must engage developers in closed-room private sessions to educate them on current potential threats and how to mitigate them with good code practices.
Security Champions: Designate team members as Security Champions in your scrum teams. Foster a culture where security is a shared value. Ensure every product team has a sponsor in the security team to discuss and validate ideas to improve and iterate.
Dashboard: Measuring and reporting are often overlooked. But the security improvements flywheel doesn’t take off without incorporating feedback loops and continuous learning. If there is a shared dashboard that provides visibility and transparency, it will develop a shared consciousness about security. Here are a few items to include in the dashboard.
- Tracks scanning frequency and results
- Monitors types and density of vulnerabilities within applications
- Measures vulnerabilities per X lines of code
- Show where in the development lifecycle a vulnerability was introduced and by whom
- Ties in with learning management systems (LMS) and organizational charts
Essential Toolkit: Automation is key to DevSecOps. Automation is not possible without the right tools. The table below captures popular tool types(not exhaustive) and potential threats each mitigates. You don’t have to use all, it is just to give you the confidence that most repetitive tasks in security can be automated with tools.

Tool	Threats Mitigated
SAST (Static Application Security Testing)	• Code injection vulnerabilities (e.g., SQL injection, XSS) • Buffer overflows • Insecure cryptographic practices • Hardcoded credentials • Unsafe use of APIs • Memory leaks • Uninitialized variables • Race conditions
DAST (Dynamic Application Security Testing)	• Cross-site scripting (XSS) • SQL injection in runtime • Authentication and session management flaws • Server misconfiguration • Insecure direct object references • Cross-site request forgery (CSRF) • Security misconfiguration • Unvalidated redirects and forwards
Container Security Tools	• Vulnerable container images • Excessive container privileges • Insecure container configurations • Container escape vulnerabilities • Malware in container images • Unauthorized access to container resources • Insecure inter-container communication • Container runtime vulnerabilities
Cloud Security Posture Management (CSPM)	• Misconfigurations in cloud services • Overly permissive access controls • Unencrypted data storage • Non-compliance with security standards • Insecure network configurations • Unpatched vulnerabilities in cloud infrastructure • Inadequate logging and monitoring • Data exposure due to misconfigured storage buckets
Incident Response Platforms	• Delayed detection of security incidents • Uncoordinated response to security events • Inefficient communication during incidents • Incomplete documentation of incident handling • Inconsistent application of incident response procedures • Difficulty in post-incident analysis and learning • Slow containment and eradication of threats • Challenges in maintaining business continuity during incidents
Network Vulnerability Scanners	• Open ports and services • Outdated software versions • Missing security patches • Weak network protocols • Misconfigured firewalls • Default or weak credentials • Insecure network shares • Vulnerable network devices
Web Application Firewalls(WAF)	• SQL injection attacks • Cross-site scripting (XSS) • Remote file inclusion (RFI) • Local file inclusion (LFI) • Denial of Service (DoS) attacks • Cookie tampering • Protocol violations • Automated bot attacks
Endpoint Detection and Response(EDR)	• Malware infections • Fileless malware attacks • Ransomware • Zero-day exploits • Lateral movement within networks • Unauthorized software installations • Data theft from endpoints • Living-off-the-land attacks
API Security Testing Tools	• Broken object level authorization • Broken authentication in APIs • Excessive data exposure • Lack of resources & rate limiting • Broken function level authorization • Mass assignment vulnerabilities • Security misconfiguration in APIs • Improper assets management
Mobile Application Security Strategy	• Insecure data storage on devices • Weak cryptography in mobile apps • Insecure communication with backends • Insufficient code obfuscation • Reverse engineering of apps • Malicious code injection in runtime • Insecure authentication mechanisms • Sensitive data leakage through logs

Continuous Improvement: DevSecOps is not just a destination; it’s a journey. You win one hour, one day at a time, and it's a new game with a new challenge tomorrow. The landscape is always changing with new threats, vulnerabilities, and organizations. Each day brings new challenges but also new opportunities to learn and innovate. The collaborative and consultative approach, with a continuous feedback loop, is important for DevSecOps excellence.

Strategic Importance of DevSecOps to Organization Brand, Revenue, and Profits

You will not find a person in any meeting room in any organization who will not acknowledge that security is extremely important. However, regarding security upgrades, most organizations may upgrade a tool but won’t invest in DevSecOps evolution which includes people and processes.

The next few sections dissect the subject of Traditional vs. DevSecOps using a purely business lens, using principles of game theory to define a winning strategy for enterprise customers and explain why DevSecOps is a rational choice.

We use the concept of Nash equilibrium to find the best possible outcome individually for each security scenario—traditional model vs. DevSecOps model—in an enterprise company and discover that transitioning to DevSecOps is a rational business decision with higher long-term payoffs.

Nash Equilibrium Explained: “A Beautiful Mind” Example

The Movie: “A Beautiful Mind” is based on the life of John Nash, a brilliant mathematician who made groundbreaking contributions to game theory, which later earned him the Nobel Prize in Economics. Game theory is a powerful tool used in business to improve decision-making by analyzing strategic interactions between different players, such as competitors, customers, or partners.

The Scene: In the movie, there's a famous scene where John Nash and his friends are at a bar, and they see a group of women, including a beautiful blonde. Naturally, all the guys want to approach her. But Nash realizes something important: if they all go after the blonde, they'll block each other, and none of them will succeed. Plus, if they all ignore her friends, the group of women might get offended, and none of the guys will end up with a date.

Winning Strategy: Rather than following the knee-jerk reaction, Nash proposes a more thoughtful approach. Instead of going after the blonde, everyone should focus on her friends. That way, they won't block each other, and everyone has a better chance of success.

Nash Equilibrium in the scene is when each guy chooses to go for one of the friends instead of competing for the blonde. No one is tempted to change their decision because doing so would make things worse for everyone, including themselves. They’ve found a balance where their choices lead to the best possible outcome for the group.

Nash Equilibrium in Traditional vs. DevSecOps Security

Scenario 1

Scenario 1: Traditional Security

The main players in this game are the Security Team and the Developers, with an indirect influence from Engineering Leadership. The Security team and Developers communicate mostly through the ticketing system.

Game Setup:

Players:

Security Team: Focuses on enforcing strict security measures.
Developers: Prioritize fast deployment and often seek exceptions to security protocols.

Strategies:

Security Team: Can either enforce security strictly or grant exceptions to developers.
Developers: Can either comply with security protocols or seek exceptions.

Payoff Criteria:

Revenue: Impact on revenue generation.
Profit: Impact on profit margins.
Brand Recognition: Impact on brand reputation.
The payoffs are represented as a triple (Revenue, Profit, Brand) with values ranging from 0 (low) to 3 (high).

Explanation of Payoffs:

(1, 2, 1): Security Enforces Strictly, Developers Seek Exceptions
- Revenue (1): Low revenue due to delays caused by conflicts between enforcing strict security and developers seeking exceptions.
- Profit (2): Moderate profit because while there are higher operational costs due to inefficiencies, some revenue is still generated.
- Brand (1): Low brand recognition due to potential internal conflicts becoming visible externally, possibly leading to negative perceptions.
(2, 3, 2): Security Enforces Strictly, Developers Comply
- Revenue (2): Moderate revenue because the product is secured and released, but slower due to strict security measures.
- Profit (3): High profit as the product is secure, reducing future costs related to security breaches.
- Brand (2): Moderate brand recognition, with a positive reputation for security but potential frustration from delayed releases.
(3, 2, 2): Security Grants Exceptions, Developers Seek Exceptions
- Revenue (3): High revenue as the product reaches the market faster due to flexibility in security enforcement.
- Profit (2): Moderate profit because while costs are controlled, the potential for future security breaches might impact profitability.
- Brand (2): Moderate brand recognition due to a balance between speed and security, though with some risk if security issues arise.
(0, 1, 0): Security Grants Exceptions, Developers Comply
- Revenue (0): Very low revenue because the flexibility in security isn’t utilized efficiently, leading to wasted resources and delays.
- Profit (1): Low profit as the lack of direction and inefficiency increases costs while generating minimal revenue.
- Brand (0): Very low brand recognition due to the perception of inefficiency and lack of strong security, potentially damaging the company's reputation.

Analysis - Nash Equilibrium (3, 2, 2)

The Nash equilibrium in this scenario occurs when the Security Team grants exceptions, and Developers seek exceptions. This setup creates a moderate balance across revenue, profit, and brand recognition but highlights the risks of compromising security. While this situation is stable, it underscores the inefficiencies and potential conflicts that arise from misaligned interests between the Security and Development teams.

Scenario 2

Scenario 2: DevSecOps Security

The main players in the game have direct communication, and security takes a consultative approach.

Game Setup:

Players:

Security Team: Collaborates closely with developers to integrate security throughout the development process.
Developers: Work with security teams to ensure that security is embedded in the software from the start.

Strategies:

Security Team: Can enforce strict security or relax measures to accelerate development.
Developers: Can prioritize security or focus on speed.

Payoff Criteria:

Revenue: Impact on revenue generation.
Profit: Impact on profit margins.
Brand Recognition: Impact on brand reputation.
The payoffs are represented as a triple (Revenue, Profit, Brand) with values ranging from 0 (low) to 3 (high).

Explanation of Payoffs:

(2, 2, 2): Security Enforces Strictly, Developers Focus on Speed
- Revenue (2): Revenue is moderate because, although speed is prioritized, the strict security measures slow down the process.
- Profit (2): Profit is moderate due to the balanced approach; while costs are controlled, the slower time-to-market may limit potential gains.
- Brand (2): Brand recognition is moderate, as the balance between speed and security prevents any significant issues but doesn't significantly enhance the brand.
(3, 2, 1): Security Relaxes, Developers Focus on Speed
- Revenue (3): Revenue is high due to fast market entry, allowing the company to capture market opportunities quickly.
- Profit (2): Profit is moderate; while the speed boosts revenue, the relaxed security could lead to future costs related to security breaches.
- Brand (1): Brand recognition is low because frequent security compromises could damage the company's reputation, even if the product reaches the market quickly.
(2, 3, 3): Security Enforces Strictly, Developers Prioritize Security
- Revenue (2): Revenue is moderate because the product development process is thorough, though slower.
- Profit (3): Profit is high as robust security reduces future costs from breaches or product recalls, contributing to long-term financial stability.
- Brand (3): Brand recognition is high because a secure, reliable product enhances the company's reputation, especially in industries where security is a significant concern.
(2, 2, 2): Security Relaxes, Developers Prioritize Security
- Revenue (2): Revenue is moderate as the process is balanced; however, the relaxation of security doesn’t significantly accelerate time-to-market.
- Profit (2): Profit is moderate because, while security is still prioritized, the flexibility might reduce costs but doesn’t lead to significant gains.
- Brand (2): Brand recognition is moderate; the company's reputation remains stable but doesn’t gain a significant boost due to the balanced approach.

Nash Equilibrium:

Nash Equilibrium: The Nash Equilibrium is at (2, 3, 3), where Security Enforces Strictly and Developers Prioritize Security.

Why (2, 3, 3) is the Nash Equilibrium:

Security Team: With strict security, the Security Team achieves high profit and brand recognition (3) because the product is secure, and they avoid potential future costs from security incidents. They wouldn't benefit from relaxing security, as it could reduce the overall payoff.
Developers: By prioritizing security, Developers achieve high brand recognition (3) and profit (3), even if revenue is moderate. If they were to focus on speed instead, the brand recognition would drop, making the strategy less appealing.

Winning Strategy and Practical Considerations:

In real-world terms, this payoff matrix emphasizes the critical role of coordination between security teams and developers, especially when considering the impacts on brand, revenue, and profits:

Scenario 1 (Developers Use Exceptions): When developers frequently seek exceptions to bypass strict security protocols, it might seem like a way to accelerate development and meet tight deadlines. However, this approach is not sustainable in the long run, as it introduces inconsistencies and potential vulnerabilities into the product. Over time, these exceptions can accumulate, leading to significant security gaps that undermine the brand's reputation, increase the risk of costly breaches, and result in higher long-term expenses that erode profits.
Scenario 2 Alignment on Security(Winner): When both teams align on prioritizing security, the result is a secure and stable product. This approach may moderate the pace of development, but it significantly enhances brand recognition, as customers and stakeholders perceive the company as trustworthy and reliable. While the revenue might not peak due to the slower time-to-market, the long-term profits are maximized by avoiding costly security breaches and maintaining customer loyalty.

Success Stories

Most digital native companies and some forward-looking enterprise companies such as Capital One, Verizon, and Comcast have implemented DevSecOps to enhance security and agility. By integrating devsecops into SDLC, they reduced vulnerabilities and improved compliance, and simultaneously improved feature velocity demonstrating the effectiveness of novel strategies described in scenario 2.

However, don’t try to copy the unrealistic targets of ‘zero incidents’ and ‘100% availability’ of forward-looking tech companies. You have to find your fastest medium pace, minimum viable release workflow, and easy things that you can get right first.

This is where a lot of companies make mistakes, they set unrealistic targets and set themselves for failure or paralyzed projects after a short runway of initial excitement.

Capital One: DevSecOps Integration in DevOps Pipeline

Capital One automated vulnerability and compliance checks in its DevOps pipeline, embedding security into its development process.

Key Challenges:

Manual and time-consuming security assessment process for Amazon Machine Images (AMIs)
Long delays in deploying secure code to production
Limited assessment coverage of IP addresses
Need to secure container environments in addition to virtual machines

Key outcomes:

Reduced average time to remediate critical vulnerabilities from 18 hours to minutes
Achieved 95% assessment coverage of IP addresses
Accelerated security assessments from weeks to less than 24 hours
Enabled near real-time detection of security and compliance issues

Verizon: Developer Dashboard for Vulnerability Management

Verizon IT's AppSec team implemented a DevSecOps approach to facilitate secure DevOps practices as they moved to the cloud. They created a developer dashboard program that combines technical aspects of vulnerability management with individual accountability.

Key Challenges:

Facilitating secure DevOps practices during cloud migration
Driving a cultural change toward security awareness among developers
Managing vulnerabilities across a large number of applications (2,100)
Providing real-time feedback to developers on potential security risks they introduce

Key outcomes:

Real-time visibility into vulnerabilities across 2,100 applications
Improved developer accountability and security mindset
Near real-time feedback to developers on potential risks
Long-term cultural change toward security awareness

Comcast: Scaling DevSecOps Transformation

Comcast implemented a DevSecOps transformation program, starting with a small pilot and scaling to about 100 software development teams.

Key Challenges:

Scaling DevSecOps from a small pilot program to about 100 software development teams
Integrating security into the existing DevOps pipeline without disrupting productivity
Reducing security incidents in production environments
Accelerating security testing processes

Key outcomes:

85% fewer security incidents in production compared to legacy teams
Reduced time for security testing from weeks/months to near real-time
Improved developer workflow and productivity.

DevSecOps Maturity

Most organizations have invested in some form of tools to protect themselves from cyber-attacks, so we can assume they are already at a certain basic level of security. The main calibration we need is how to assess organizations' maturity to deal with software security at scale and speed.

On the DevSecOps maturity curve, we can classify most customers into three levels—initial, automated, and advanced. It is also fair to assume that most digital natives' customers and some forward-leaning organizations have leapfrogged themselves into a more advanced stage.

While the security benchmarks of individual companies are not publicly available. We can draw inferences from results published by the MIT CISR Survey. According to a survey, approximately 75% of companies do not invest in operational efficiency. These are the firms that I believe are in the initial stage of DevSecOps. The other firms, mostly tech companies or enterprise companies that operate like tech companies such as Capital One and JP Morgan Chase, are operating either at Automated or Optimized maturity levels.

The important information in the table below that I would pay attention to is the organization experience, which tells you what it feels like when you are working in an automated or optimized environment. I personally worked in a few that I can correlate back to initial, automated, and optimized. All I can tell you is that it feels obviously different. Just deploy a simple dummy 3-tier web app, and you will experience most of it in black and white.

Addressing Counterarguments

While DevSecOps offers numerous benefits, challenges remain.

Here are some key counter-arguments or potential drawbacks to implementing DevSecOps followed by potential solution to counter arguments inline:

Initial capital costs: Implementing DevSecOps can be expensive, especially initially, as it requires investment in new tools, processes, and training for staff. [Use Toil and Wait Time as Value Lever] You can justify it with the operational cost of toil and wait time involved in traditional security to get things done.
Slowed-down software rollout: Adding security checks and processes throughout the development lifecycle can potentially slow down software delivery, at least in the short term. [Lead with Facts] Estimate the total time to perform the checks. In DevSecOps it is in minutes, not hours, which is significantly better than days in traditional security.
Lack of expertise: There may be a shortage of professionals with the right mix of development, security, and operations skills needed for effective DevSecOps implementation.[Find a Right Partner] You can find a right partner like Ollion to support you.
Complexity: DevSecOps introduces additional complexity to the development process, which can be challenging to manage, especially for smaller teams or organizations.[Find a Balance] Initial assessment is important to justify investment. DevSecOps is a solution to economy of scale and economy of speed. But, if an organization doesn’t doesn’t have scale, you can wait a little longer, or do the minimum that doesn’t feel overwhelming.
Tool overload: No single tool can provide a complete DevSecOps process. Organizations may need to integrate multiple tools, which can lead to tool sprawl and integration challenges.[Find a Balance] DevSecOps is a journey, you have the option to keep it simple. Start small and automate incrementally.
False positives: Security tools can generate many false positives, overwhelming teams if they are not properly configured and customized for the specific application and environment. [Find a Balance] The signal vs noise ratio is important. You control the flood gates. Monitor what is critical and then go into minor details.
Cultural resistance: Implementing DevSecOps requires significant cultural changes, which can be met with resistance from team members accustomed to traditional development practices. [Wait or Align] You can either wait for a crisis to drive focus and attention or proactively find a way to lean in.
Initial productivity drop: As teams adapt to new processes and tools, productivity may initially decrease before the benefits of DevSecOps are realized.[It's a strategic investment] You have to invest time. Think of it as an investment in operational efficiency to drive unit cost of digital products and serving customers down.
Potential overemphasis on tools: Organizations might focus too much on implementing new tools rather than addressing fundamental security practices and cultural changes. [Control Tool Innovation] It is a valid concern. Manage tool innovation. Enforce standardization and value justification.
Difficulty in measuring ROI: It can be challenging to quantify the return on investment for DevSecOps initiatives, especially in the short term. [Keep Short Term Short]. It’s an investment but you can plan it to keep it short term short. Start with Minimum Release Process support by minimum controls. Scale adoption and build incrementally. Add depth while scaling adoption.

While these counter-arguments exist, it's important to note that many organizations still find the long-term benefits of DevSecOps outweigh these potential drawbacks when implemented effectively.

Practical Advice for Starting DevSecOps

Minimum Viable Release Process: If you are not sure where to start, pick a simple app and go through the release process. This is your MVP, which you have to adapt to DevSecOps. You will have enough scope to execute
Create a DevSecOps Vision: Develop a shared vision that values security as an integral part of development. Tools are important, but people and processes matter more. It will allow security and development to lean in towards collaboration.
Build Multidisciplinary Teams: Encourage collaboration by having team members from different disciplines work together, fostering mutual understanding and respect. The idea is to have a security person as a sponsor in each product team and engineer security champions in engineering.
Define Controls Objectively: Ensure security controls are clearly defined and updated with technological advancements. In most organizations, controls are very subjective, and often, security feedback is reviewer-specific.
Balance Innovation and Risk: Combine guardrails and controls to balance innovation and compliance. Guardrails work best for new systems that are evolving. Strict controls work for stable systems that are used at scale.
Seek External Support: Consider partnering with experts like Ollion to help you bootstrap your DevSecOps journey.

Conclusion

Organizations must carefully consider their approach to software security. By adopting a DevSecOps model, they can align the goals of their security and development teams, reducing risks and improving overall efficiency. This game theory analysis illustrates that collaboration, rather than compartmentalization, is the key to achieving the best outcomes in today's complex software development landscape which is a growing target for sophisticated attacks at increasing frequency.

Why Ollion?

Conventional security measures cannot cope with the ever-evolving threat scenario. Definitely, our DevSecOps framework here at Ollion will be able to provide you with a comprehensive solution to protect your software while minimizing risks.

We integrate security at every step of the development cycle to help you reduce vulnerabilities, enhance compliance and accelerate development.

Let's chat