Introduction to DevSecOps

Table of Contents

Introduction What Is DevSecOps? Key DevSecOps Themes Getting Started With DevSecOps DevSecOps Core Practices DevSecOps Additional Practices What’s Next for DevSecOps? References

Section 1

Introduction

Today, most companies have implemented DevOps practices within their organization. DevOps provides a culture where teams can deliver reliable software and updates faster. This approach presents an opportunity for teams to focus on quality rather than wasting time on operations. However, as a result, security practices are often left to security specialists at the end of the delivery pipeline. A specialized security approach then creates unnecessary overhead within the delivery process as unexpected issues frequently arise at the end of delivery. Consequently, teams lose time fixing the code and starting the same process repeatedly, ultimately making delivery costly and inefficient.

DevSecOps has become increasingly important as most companies have embarked on digital transformation. With these plans, companies are moving to the cloud. This results in moving away from on-premises infrastructures and transitioning to public cloud solutions. Cloud providers offer cost-effective, scalable, highly available, and reliable solutions. However, these advantages come with new security challenges.

Security deserves a higher priority than ever before. With cloud solutions, there is no room for mistakes. By not following security requirements, you might open the door of your network to dozens of security threats. Therefore, security should be considered earlier during the design phase. Developers are more successful with this approach because they first create a secure environment before developing their features. Additionally, developers are more involved and aware of security requirements since it is now part of development.

DevSecOps integrates security into DevOps as an integral component of the SDLC instead of observing security as an afterthought. It also distributes security responsibilities amongst team members. In collaboration with security specialists, teams can implement a “security as code” culture that encourages security to be treated like other software components of the SDLC pipeline.

Diagram

Description automatically generated

Section 2

What Is DevSecOps?

DevSecOps is the full stack: DevSecOps spans the entire IT stack and includes network, host, container, server, cloud, mobile, and application security. All of these layers are increasingly turning into software, which makes application security a critical focus for DevSecOps. DevSecOps spans the entire software development lifecycle, including development and operations. In development, security focuses on identifying and preventing vulnerabilities, while in operations, monitoring and defending against attacks are the main objectives.

Can teams apply DevSecOps practices and tools to non-DevOps projects? Absolutely. The ideas in this Refcard apply to almost any software project. If the goal is to produce highly secure software in the most cost-effective way possible, DevSecOps is the path forward.

Organizations practicing DevSecOps have shown impressive results. According to Gartner, these early adopters are 2.6x more likely to have security testing keep up with frequent application updates and show a 2x reduction in time to fix vulnerabilities.

Understanding the different types of security work and their value to the organization is critical to successful DevSecOps initiatives. Until teams truly understand the work, it will be difficult to deliver it effectively. To learn more about this topic and DevOps in general, read books like The Phoenix Project and The DevOps Handbook.

Section 3

Key DevSecOps Themes

Every DevSecOps program is a little bit different. it is best to view DevSecOps as a journey. As organizations progress, they may find that different teams are at different points down the path. The themes below are not specific activities. Instead, they are guideposts to help make decisions along the journey.

DevSecOps Guideposts
Theme	Description
Empowering engineering teams	Development and operations are empowered to deliver secure applications into production themselves. Security experts provide support as coaches and tool smiths but do not have primary responsibility for security. Ensure that tools and processes are designed for developers and operations, not security experts. This way, teams can share expertise amongst all members of a team.
Making security visible	In many organizations, security work is hidden, unknown, and untracked. In the end, the value of security is often not easy to understand. In DevSecOps, we make small security tasks that can be tracked, tasked, and measured like any other type of work. Additionally, security becomes part of day-to-day responsibilities, allowing security to be transparent to everyone.
Shift left	From developers to DevOps engineers, everyone is involved in security. Shifting security “left” means that security activities start during development and extend throughout the SDLC, with continuous feedback at every stage from development to production. See “DevSecOps Additional Practices” below for more examples.
Security as Code	Like continuous integration and continuous deployment, continuous security means that teams respond to continuous threats with security activities that are performed continuously, as part of the development and operations process, and integrated into the tools team members are already using. Security as code is the key to automating security operations, which leads to an end-to-end process for security practices.
Integrate security into CI/CD workflows	As a result of the shift-left approach, security is considered right away from the development stage. CI/CD workflow is not only for building and deploying the application anymore. Security is the new part of this workflow in which every line of code is scanned and verified against vulnerabilities. New tools make this relatively easy for developers. These tools identify the security issues and immediately provide actionable solutions to fix the issues. Besides security scanning, there are various methods to conduct security testing. These are discussed later in the “Creating Your DevSecOps Pipeline” section.
Continuous security	Security activities are performed continuously as part of the development and operations process and integrated into the tools team members are already using to address continuous threats.
Continuous monitoring	When development is complete, monitoring begins; we need to constantly track the behavior of our applications to detect anomalies and possible attacks. Continuous monitoring shows insight into the application’s security and provides feedback for the development team.
Prevent and protect	No team will ever produce perfect code. Nor will they ever detect or stop all attackers. Therefore, the best security strategies involve balancing secure coding during development (DevSec) and runtime protection during operations (SecOps).
Cloud security	Despite the numerous benefits introduced by cloud solutions, there is still a big challenge from a security standpoint. On the cloud, developers can create resources in a couple of minutes and connect to the public cloud. From a DevOps standpoint, this is quite interesting since most organizations need to protect customer data and/or avoid a security breach. At the same time, organizations want to utilize the benefits of cloud solutions. Here, a DevSecOps approach becomes mandatory, and most security decisions should be made during the design phase. It is also easier to provide isolated environments for developers on the cloud to try out new features without compromising any customer data.

Three Mistakes to Avoid in Enterprise Organizations

Before jumping into core practices, let’s address common problems seen in most enterprise organizations:

Most Common Security Challenges
Challenge	Description
Security in a silo	Today, most companies have adopted Agile and DevOps. However, it is still common for many of them to keep security out of these processes and security as a separate entity that specialized people can only manage. On the one hand, this situation creates a silo around security and prevents engineers from understanding the necessity of security or applying security measures from the beginning. Siloed security results in an inefficient workflow for software development. The best practice is to share these responsibilities across all team members instead of having a specialized security engineer.
Unbalanced security	Sometimes security becomes the priority, and then the organization loses creativity and quality performance as a price for security. Financial institutions are an excellent example of where security is often unbalanced. Security has the highest priority over any other software development project in these scenarios. While this approach is not wrong, the focus on security can lead to a detrimental imbalance. As a result, other factors such as performance and quality are neglected. The focus and resource capacity are primarily spent making sure everything is secure. Consider both security and other essential application components during the design stage to treat security as a regular aspect of the application. Then developers can focus on development while feeling confident about security instead of security departments delaying development due to a lack of security considerations.
Lack of playground for developers	Security obsession in some organizations leads to a strict environment with no room for creativity. With security as the most critical factor, there is no playground environment; every new experiment or piece of software must go through a complicated process and verification against security compliances before being used by developers. This prevents developers from having the space to try out new things. Therefore, DevOps teams cannot do their job properly, which delays and negatively impacts the entire SDLC. DevSecOps aims to keep things secure so that everyone can do their job at their pace. Developers are the source of creativity in the software development workflow. This means that the more freedom they get, the more value they might deliver. Part of a developer's job is to try new things out, break things, and learn from failure. Sometimes developers need environments to test new software without common security limitations. These environments are known as a "SandBox," which are temporarily isolated environments. It is common that these environments have no connection to any internal network and have no customer data. This way, engineers can easily try new futures without being worried about not following common security compliances.

The “Three Ways” of Security

For decades, both software and security have struggled with poor quality results, cost overruns, and processes that require experts. While DevOps has shown promise on the software side, security is still practiced in very traditional ways. DevSecOps is not just shoving traditional security practices and tools into DevOps.

Instead, we must rethink the security work. We will need new practices and technologies to perform this work. We can give this transformation structure using the “Three Ways” from The Phoenix Project. By framing the problem this way, we can see that we need to get security work flowing, ensure instant security feedback, and create a security culture.

The “Three Ways” of Security
Approach	Description
Get your security work flowing	Most security work is monolithic and attempts to cover all risks in a single task, like a complete security architecture or security scan. How do we get security work flowing? Make the work visible. Work a single security challenge at a time. Limit work in process. Reduce handoffs. Automate everything.
Ensure instant security feedback	Security is one of the most common causes of technical debt, and the cost of this work increases dramatically the farther it progresses across the SDLC. Several reasons include lack of knowledge and limited security specialists. How do we keep security work on track? Increase awareness about the importance of security. Identify potential problems. Make problems instantly visible. Swarm on the problem. Seek the cause. Ensure security “findings” are designed for easy consumption. Focus on providing a solution rather than exaggerating the problem.
Encourage a security culture	Many organizations have a security culture of blind trust, blame, and hiding that prevents developers and operations from working with security. How do we create a culture of security? Empower everyone to challenge security design and implementation. Take every opportunity to make security threats, policies, architecture, and vulnerabilities visible. Allow everyone on the team to participate in security. Trust that engineering teams want to do the right thing. Celebrate the knowledge gleaned from security issues rather than blaming those involved. Spend more effort on upgrading practices and preventive measures than vulnerability remediation and incident response. Plan trainings and conduct workshops to maintain continuous security throughout all teams.

By following these three ways, teams will see security as a concrete output from the development process. It is a combination of security features and assurance, captured in a tangible way. By applying DevOps concepts, we can produce this concrete security continuously and effectively as a part of standard software development.

Section 4

Getting Started With DevSecOps

Traditionally, security has been performed as a series of massive tasks spanning all risks. For example, security consisted of writing comprehensive security requirements, designing a broad security architecture, performing a thorough security test, etc. But agility requires a risk-based approach. To accomplish security work in a DevOps organization, we can prioritize our security tasks and break them into small pieces for implementation.

Chart

Description automatically generated with medium confidence

This diagram shows how security fits into the normal DevOps development cycle at a very high level. Notice that these security augmentations are designed to fit naturally into the process: no extra steps, no gates, no delays. Instead, teams cycle quickly on small security tasks that are structured to be delivered by the development and operations teams using the tools they already use. Let’s explore these core practices in detail a bit later.

“Hello, World!” A First Step Towards DevSecOps

Let’s use a straightforward example to demonstrate. Imagine that we have a web application being built by a DevOps project. All we know is that the application didn’t do well on a recent security scan. There is no threat model or security architecture. How should we get started with DevSecOps?

Analyze: Identify the next most critical security challenge
Security: Implement a defense strategy
Verify: Automate security testing
Monitor and Defend: Detect attacks and prevent exploits
Documentation

1. Analyze

Do not secure everything at once. The most critical scan findings indicated that this application has obvious SQL injection problems. The goal is to make tangible progress quickly, so it is best to deal with this first. First, build defenses and assurance over time in small pieces, not all at once. Create a JIRA ticket to address the SQL injection.

2. Secure

Next, look at the SQL injections to determine which parts of the application are vulnerable.

Once that is complete, we need to build our defense. The best strategy to prevent a SQL injection is to use input validation and parameterized queries everywhere across the codebase. It may be beneficial to break this task up into even smaller pieces. So fill out the details of the defense strategy via ticket(s) and start implementing them.

3. Verify

Use simple tools to ensure that non-parameterized queries are eliminated from the codebase. A deployed tool can be used to warn developers in their IDE if they violate this rule. It is also important to automatically re-verify during CI/CD as well as perform a final check prior to deployment.

4. Monitor and Defend

Finally, be aware of any attackers that target with SQL injection attacks. For visibility and protection in production, implement runtime application self-protection (RASP). This application then monitors requests and detects possible attacks.

5. Documentation

Now that these critical security steps are complete, it is time to document everything. Document this process from design to completion to ensure each task is visible to the rest of the team. Documentation provides insight into what has worked and what has not worked to come up with potential improvements.

Once each step is complete, it is time to start in on the next security breach or challenge. Each time we complete a challenge, we leave behind infrastructure to make sure it remains secure. Note that we don’t want to create a fragmented set of individual defenses, so it’s important to make good decisions over time about security architecture.

In DevSecOps, don’t let perfect be the enemy of good. We’re looking to improve our security story on every iteration. Your first attempt might be a partial solution with sampling instead of a rigorous defense and complete testing. The goal is to get this cycle running and make measurable progress over time. Again, do not forget to create documentation for each implementation to ensure we set a standard baseline for future applications.

Creating Your DevSecOps Pipeline

Let’s assume that you have implemented a CI/CD pipeline in your current DevOps practices. This pipeline includes all required steps to bring a piece of code from development to production. We need to integrate security into this pipeline as a part of the application delivery. Although implementing a separate pipeline for DevSecOps sounds interesting, this is not an efficient approach and creates overhead for teams. Additionally, DevSecOps aims to incorporate security earlier in your current CI/CD pipeline.

In each phase of the pipeline, there must be some tasks dedicated to security. Implementing DevSecOps within your pipeline requires adopting tools and processes that continuously perform security as code is written, integrated, tested, deployed, and operated. While there’s just one delivery pipeline, having a security “view” of your pipeline may help you understand the security value stream separately, revealing bottlenecks and inspiring confidence in the results. There are a lot of different components you can add to your pipeline. Many security tools can help to implement:

Unit Testing: This is the first opportunity to test any piece of code against its functionality, and therefore, it is the best moment to check if a small part of the software can operate as expected.
Dependency Scanning: Every project uses various libraries, and sometimes external libraries are used in the project. With dependency scanning, those libraries are scanned for vulnerabilities to ensure the libraries are safe to use.
Dynamic Application Security Testing (DAST): DAST is how you can test your applications for vulnerabilities without checking the code inside the application. This method is also known as black-box testing, which can detect vulnerabilities like SQL injections and cross-site scripting.
Static Application Security Testing (SAST): Unlike DAST, SAST focuses on the code. So the security vulnerabilities can be found earlier in the development stage without running the code. Tools like Synk can be integrated into the pipeline to scan application codes constantly. This approach is quite efficient and saves time and money for teams.
Container Scanning: Containers are great solutions, but they can become a black box for security. So it is crucial to constantly scan container images for vulnerabilities, malware, and other security measures to ensure nothing is left out of the radar.

Graphical user interface

Description automatically generated with medium confidence

The main cycle in the DevSecOps pipeline involves security tools, an analytics hub, and integrations with development and operations tools. The security tools at the bottom of the diagram identify vulnerabilities in applications and APIs across development, test, and production environments. In production, other tools monitor and prevent attacks. The telemetry from these tools feeds into an analytics system for historical tracking, analysis, and notifications.

Common events include:

Custom code vulnerabilities
Known vulnerabilities in libraries and frameworks
Attacks on custom code vulnerabilities
Attacks on libraries and frameworks
Application inventory, including all libraries and frameworks
Software architecture details

In general, DevSecOps favors the use of notifications (real-time integrations into normal development and operations tools) over PDF reports. However, for some purposes, such as compliance, PDF reports may be generated. Notification alert team members need to know about security events immediately through the tools they are already using as part of their everyday job. While a single analytics system would be ideal, today, you may need separate systems for vulnerabilities and threat events.

Essential DevSecOps Systems
System	Description
Smart alerting	It is vital to design a smart alerting system when it comes to monitoring — regardless of security or other goals. The term “smart” here means that our monitoring platform should raise an alert whenever an action needs to be performed. This requires the alerts to be designed and fine-tuned so that we avoid being spammed with any unimportant alerts. One example is getting an alert for a service that is slower than usual but not slow enough to make a considerable impact on the whole system. This incident is considered low severity and should be investigated later, but it should not interrupt workflows.
Intelligent security analysis	Use a platform that has implemented intelligent security analysis to process the information constantly and identify anomalies. This makes security monitoring much easier for security and DevOps engineers. One of the best examples of this is the threat protection on Azure, a collection of security services and functionalities. Azure threat protection provides an overview of what is happening at any time within your application. Therefore, choosing the right platform for your infrastructure and configuring the right tools can significantly impact security monitoring operations.
Centralize security monitoring	A critical aspect of security monitoring is centralizing all security issues and managing monitoring. Especially for enterprise organizations, having a birds-eye view is necessary. Without these, one might lose control of what is happening easily, resulting in huge costs for the organization. On the cloud, this is called “security posture management.” As an example, let’s look at the Sentinel product from Azure. Azure Sentinel connects to each different solution to collect data. Then, inside Azure Sentinel, you are notified of previously undetected incidents, investigating issues using artificial intelligence and other features. These help to respond to incidents rapidly and stay compliant. Another example is the AWS Security Hub, which provides cloud security posture management. Therefore, there are still options for security monitoring on any platform you choose. Having this knowledge upfront will help to plan your security roadmap accordingly.

Implementing a notification infrastructure encourages downstream security stakeholders (developers, testers, operations, audit, executives, etc.) to work closely with upstream providers (like security testers) to ensure that the work is optimized for them. Your DevSecOps pipeline should be designed for very tight feedback loops — think seconds, not hours, weeks, or months. The faster you can get feedback from the people that need it, the more secure and cost-effective your DevSecOps pipeline will be.

In the beginning, a DevSecOps pipeline will only verify a few simple things about the software. But over time, as challenges are addressed, you will automate verification of more and more of the security strategies and defenses. Over time, the goal is to migrate from manual security testing to a fully automated pipeline capable of deploying secure code directly into production without gates.

Choosing Security Tools and Technologies

Here are a few of the attributes to consider when choosing security tools and technologies to build your DevSecOps pipeline across the entire SDLC. Please note that there is no one set of best tools for DevSecOps. The tools you choose should match how you build software, your goals, your culture, and the other technologies you use.

Considerations for DevSecOps Tool Selection
Consideration	Description
Policy coverage	First and foremost, you must confirm that the tool actually covers the risks you need it to cover. Many products have surprising shortcomings in this area. See the OWASP Benchmark Project for help.
Accuracy	Accuracy (eliminating both false positives and false negatives) is critical. Inaccuracy means humans have to fix results, which will destroy your pipeline. You should carefully test your tools to be sure they accurately verify what you need.
Speed	You need to test whether tools are fast enough to work as a part of your DevSecOps pipeline. That may be microseconds, seconds, or minutes — but probably not hours and certainly not days.
Scale	Consider the size of your application portfolio and whether the tools you select can operate continuously, in parallel, and across the entire portfolio. Be sure to factor in the number of people you need to make that work.
Process fit	It is best to verify that the tools are useful without a complex installation process. When well-meaning security folks buy tools for development and operations teams, it can cause friction if they aren’t compatible with their technology stack or workflow. Engage the actual users in the evaluation process and conduct pilots to confirm they will be easy to install and use.
Integrations	Verify that the tool integrates with the tools that people in your DevOps toolchain are already using. Look for well-documented, supported REST APIs and SDKs in a variety of languages, IDE plugins, webhook support, ChatOps integrations like HipChat and Slack, notifications with PagerDuty and VictorOps, and SIEM integrations like Splunk.
Security on the cloud	Security becomes a shared responsibility between cloud providers and companies in cloud solutions. As the complexities of architecture increase on the cloud, controlling events is much easier than on-premise architectures. The interesting thing about cloud architectures is that you can add these options to your design before developing any line of code. So your solution design is already security-proof before the implementation. Additionally, every resource is created by code, and that’s how engineers can think of all security issues earlier on in the development process. Cloud providers already give best practices for various security practices such as using firewalls, network security groups, DDoS and identity management, etc. They also provide various tools and products to implement and monitor security measures.

Section 5

DevSecOps Core Practices

DevSecOps takes a very agile approach to security, breaking down massive security tasks into incremental improvements that are performed as everyday development tasks. These small batches of work include continuous verification so that security builds over time instead of repeatedly starting from scratch.

DevSecOps brings a culture of “security for everyone” to teams. Everyone has a significant role to play in security at their organization. Security specialists are critical in this by mentoring, advising, and leading teams to ensure quality and security.

Once we’ve identified the next security challenge, our normal engineering process can execute on the improvement. In this section, we explore four core practices to any DevSecOps initiative. Of course, your DevSecOps process might be considerably more complex. See the next section for more ideas, or add your own practices to this basic cycle.

Graphical user interface, text, application

Description automatically generated

Fundamentally, it is the constant tension between creating defenses and attempting to break them that actually makes organizations more secure. The faster you can repeat this DevSecOps cycle, the faster you can improve security. Over time, you’ll build a complete security story that will provide assurance both internally and externally.

1. ANALYZE: Identify Your Next Most Critical Security Challenge

Why should one always focus on the most critical security challenge? Generally, working on anything else won’t change your security posture very much. It doesn’t help to close the attic window when the garage and front door are wide open. In DevSecOps, we get the work flowing by creating small batch sizes. So, in most cases, we want to work on our enterprise's most critical security challenge first. Still, don’t be afraid to choose a partial measure or a tiny improvement to your most critical challenge. Working in small increments makes sure we stay on track.

When deciding what to work on next, the team looks at all the potential security “work” available and makes it visible. The team might add new features, pay down some technical debt, make an architectural improvement, fix defects/vulnerabilities, or do something to improve the team’s tools or practices to improve quality, security, or productivity.

It’s important that the team use its threat model and security architecture to make an informed decision about the next most critical security challenge. What is the cost to the company of certain kinds of attacks? What is the cost of implementing preventive measures for those attacks? Try to use both internal and external data sources to figure out the next thing to do that will most effectively reduce risk.

You may find yourself without a threat model or security architecture. Fortunately, in DevSecOps, these artifacts are created one step at a time. When you’re starting out, it’s easy to identify your top challenges: the problems that are the most likely to be found, exploited, and cause serious damage like injection, authorization problems, known vulnerabilities, etc.

Consider all the different layers of your application stack:

Security Considerations for Layers of the Application Stack
Layer	Consideration(s)
Applications and APIs	Do you have proactive controls in place? Are you susceptible to common vulnerabilities?
Libraries and frameworks	Are your libraries and frameworks up-to-date and properly configured? Do you have a complete, up-to-date inventory (with exact version numbers) of all the software you are running across all your servers?
Containers	Have you hardened your platform configuration and kept it up to date? Are your cloud environments configured correctly? Did you consider scanning container images against vulnerabilities? Did you add security scans to your pipeline?
Cloud resources	Are applications and customer data secure? Do we have an authorization matrix in place to make sure the right people have the right access? Are networking configurations configured properly for each resource? Do we have the penetration test results on public-facing resources?
Network	Do you have strong network security defenses in place and monitored for attacks?

Use a risk-based approach to decide what to work on next. Be sure to consider whether there’s a viable connection between a threat agent, attack vector, weakness, technical impact, and business impact in your enterprise. OWASP depicts this connection as follows:

Diagram

Description automatically generated

Going forward, you should practice a combination of threat intelligence and security research to continuously zero in on your next most critical security challenge. Note that this process is significantly different from assessing all your threats. Many organizations get overwhelmed trying to protect against everything at once.

DevSecOps Threat Intelligence, Security Research, and Security Architecture
Threat intelligence	External sources: ISACs (STIX/TAXII), OWASP, SANS, BlackHat, DefCon, LASCON, DevSecCon, CISO events, friendly peer companies, etc. Internal sources: Monitor your systems for attacks and learn from the data. Understanding actual attacks is a major factor in prioritizing.
Security research	Security research should focus on challenging security architectures and identifying new strategic ways to improve security. Where possible, work with development to enhance the DevSecOps pipeline with new testing methods.
Security architecture	There is a shortage of great threat modeling and security architecture tools. But some interesting projects include OWASP Threat Dragon, IriusRisk, and Chaos Engineering (Chaos Monkey, ChaoSlingr).

You’ll have to work out your own cadence for re-examining your threat model. So, increasing your cycle speed will directly affect the level of security you are able to achieve. And because DevSecOps adds to your security incrementally with continuous security verification (discussed below), you have protection against backsliding.

2. SECURE: Implement a Defense Strategy

Once you’ve decided on a security challenge to tackle, you’ll need to choose a defense strategy. A defense strategy isn’t a single security mechanism or product. A defense strategy can combine technical security mechanisms, secure coding practices, procedural controls, supporting processes, training, background checks, and more. We are using the term “defense strategy” broadly to include anything that you rely on to prevent a breach. Your defense strategy for a particular challenge can (and probably should) comprise one or more primary defenses and a set of supporting defenses as well.

In an agile way, you have defined stories and tasks for each sprint. The product owner is the person who defines new feature requests and also the priority level. This usually applies to application features and not security. In a DevSecOps approach, security is also considered a high-priority feature added to the backlog. Security specialists give consults on what needs to be done, and all team members can participate in the implementation, test, and delivery. This approach prevents teams from having security silos, and security expertise is shared between all team members from developers to testers and DevOps engineers.

You can capture each security strategy for implementation in a JIRA ticket that covers each security enhancement that you want to make, including:

DevSecOps Defense Strategies
Strategy	Description
Challenge description	The goal is to justify why this is an important security challenge concisely. This might take the form of a security story or misuse case. The description should cover the elements shown in the OWASP diagram above.
Defense story	This story should detail exactly how the defense should work. For technical defense mechanisms, the story should clearly detail how the threat is countered and why this defense is effective. For other defenses, how they work to provide protection should be argued.
Documentation	As part of the defense strategy, you should also consider how to configure, operate, and use these defenses. This guidance should be written as a tutorial to apply to end-users and operations staff, and even indicate how developers should use the defense effectively. It is vital to document all operations and implementations to prevent any rework in the future.
Security testing approach	It is important to define automated tests, which verify the correctness and effectiveness of the defense strategy.
Fail and test your system	Do not wait for disaster to strike in order to verify security effectiveness. Instead, try to fail the system and challenge implementations yourself. This can be done on environments other than production, where you may have a clone production environment like an acceptance environment. Simulate security issues to see if the defense strategy works as expected.
Security monitoring	The final part of the defense strategy is to detail how to will continuously monitor security and avoid possible attacks.

Your strategy is right when you can easily answer with confidence when anyone asks: “How do you protect against X?” Having a clear, concise, defensible answer to this kind of question can not only provide an easy path to compliance but can also provide business advantages over competitors.

Your defense strategy doesn’t have to be perfect from the very start. It’s far better to start with a basic defense and evolve it over time. After you implement a basic defense, you may choose to work on another, more pressing, threat. The key to DevSecOps is to continuously reprioritize based on the threat and existing defenses. The ability to respond quickly is critical for a world of continuously changing threats.

The work of implementing the security defense shouldn’t be any different than any other feature. It should — to the maximum extent possible — be delivered as code or configuration with everything in source control. Managing security in this way makes it possible to test and redeploy at any time, ensuring that defenses are in place, working correctly, and properly configured.

3. VERIFY: Automate Security Testing

A key part of DevSecOps is ensuring that the defense strategies have been properly implemented, configured, and operated. Security testing is how to verify that your actual security controls match your intended defenses. In DevSecOps, we focus on automating those tests by “turning security into code” so that we can run them frequently without requiring humans, particularly security experts, in the critical path.

There are many ways to verify the security of a system automatically. There is no possible way to list them all, but we provide a few examples of popular tools that have proven to be DevSecOps compatible.

Popular DevSecOps Automation Techniques
Technique	Description
Automate everything	It's not automated if you need humans in the loop. Don't fall into the trap of thinking that you've automated security when all you really did was automate the "scan" button. Think about the entire process. Does the tool require human expertise to configure or run? Does it require an expert to interpret and triage the results? We are looking to eliminate the involvement of humans in the critical path so that we can push code to production with both speed and assurance.
Avoid “tool soup”	Every tool that you adopt means additional process steps, a full set of integrations, and a team of people to configure, run, and interpret. Choose powerful platforms that will allow you to address many different types of security challenges using an integrated framework. Therefore, you avoid having overhead when maintaining multiple tools.
Test your testing tools	Security testing tools vary significantly in their ability to test real applications for a broad range of issues. The only way to know how well a particular tool will work on your applications and APIs is to try it. Consider temporarily adding “tool canaries” in your applications to verify that real vulnerabilities are being discovered and false alarms are not being flagged. See the OWASP Benchmark project for details.

Below is a DevSecOps security testing funnel to help you choose a security verification technique for a particular security challenge. This may seem obvious, but don’t blindly rely on the wrong tool. Take a minute to select the simplest, fastest, most accurate way to check that your defense implementation is correct, complete, and configured.

For example, testing for proper clickjacking protection is easy if you simply examine HTTP responses for the proper security headers. But it would be very difficult to verify this by looking at the source code, as there are so many ways to accomplish this.

DevSecOps Security Testing Funnel
What are we trying to test?	Think carefully about exactly what you want to test and the results you want. Direct, complete verification of application behavior is always best, but you can use sampling, fuzzing, design analysis, and other techniques.
Do we need to test it?	Probably, yes. But you should clearly prioritize things that are the most critical to security and the most likely to be discovered and exploited by an attacker.
Positive or negative testing?	Are we able to verify that the application always follows a known good pattern of execution (positive security), or will we have to resort to trying to verify that the application never follows any of the known bad patterns of execution (negative security)?
Do we already test for it?	You may be using a security testing tool that covers this risk. It's critical to confirm that your tool does a good job of accurately and efficiently verifying your security defense.
Do we already have a platform that will allow us to test this easily?	Perhaps you just need to enable a rule in a tool you're already using. Or maybe you can use an existing tool as a platform for creating a custom rule.
Can we test it by writing custom tests?	If we can't use a security testing platform, can we create a custom test case?
Is there another tool on the market that can help test this?	Network: nmap, sslyze, ssh_scan, Tenable, Qualys. Cloud/container: Aqua, Twistlock, Redlock, ThreatStack. Libraries/frameworks: OWASP Dependency Check, retire.js, Contrast Assess, Snyk, Sonatype, BlackDuck. Application: OWASP ZAP, Arachni, sqlmap, Burp, Contrast Assess, Micro Focus, CA Veracode, Synopsys, Checkmarx.
Do we need human experts to test it?	While the goal of DevSecOps is to minimize the number of things that you need human experts to test, bug bounties, red team exercises, and manual penetration testing can provide useful insight into defenses that are difficult to test automatically. Ensure that these efforts deliver rules, test cases, and other automation, not PDF reports. Examples: BugCrowd, HackerOne
Are we also testing for what we didn’t think of?	It’s impossible to know everything, so certain kinds of testing rely on volume and randomness to try to force applications to misbehave. Fuzz testing and chaos engineering tools can help here.

Any security issues discovered during testing should feed into the DevSecOps management infrastructure described above to notify all the people that need to know through the tools they are already using.

4. DEFEND: Detect Attacks and Prevent Exploits

DevSecOps organizations recognize that you can never test yourself securely, so they adopt a balanced approach that focuses on minimizing vulnerabilities during development and identifying and preventing vulnerabilities from being exploited in production. While these two activities have traditionally been separated, DevOps has brought them together by supporting the full software lifecycle.

DevSecOps Attack Detection and Prevention Tools
Tool	Description
Runtime Application Self Protection (RASP)	RASP uses application instrumentation to add attack detection and prevention directly to applications regardless of where or how they are deployed. DevSecOps projects can use RASP to achieve accurate attack-blocking and the flexibility to deploy in cloud/container environments easily. Examples: Contrast Protect, Prevoty, Immunio.
Web Application Firewall (WAF)	WAFs have been on the market since the early 2000s and have a history of complex configuration and spotty protection. Nevertheless, a DevSecOps project might be able to use a WAF for basic protection or as a platform for virtual patches. Examples: ModSecurity, Imperva, F5, Signal Sciences.
Network Intrusion Detection and Prevention (IDS/IPS)	There is a variety of network-, container-, and host-level protections against attacks. Seek out products that can be deployed and managed as part of your standard technology stack. Examples: Snort, Suricata, Bro, Kismet.
Security Information and Event Management (SIEM)	SIEM tools provide real-time analysis of security alerts generated by applications and network hardware and are important to handling attacks in DevSecOps. Examples: Splunk, ELK, SumoLogic, LogRhythm, ArcSight.

The threat and attack data gathered should feed directly into the next DevSecOps cycle to use security research to help choose the next most critical security challenge.

Section 6

DevSecOps Additional Practices

Additional sets of security challenges emerge when an enterprise has hundreds or thousands of applications in its portfolio. Doing security at this scale is far beyond what a small, dedicated security team can accomplish. DevSecOps is a technique for distributing this work effectively across development and operations.

It’s worth noting that in most organizations, only a small percentage of projects are very far along in their DevOps journey. So, managing the transition to DevSecOps across an entire application portfolio is a key part of the challenge.

Standard Defenses and Enterprise Security Architecture

Generally, DevSecOps organizations — particularly those with large application portfolios — prefer standard security defenses that are heavily tested for correctness and effectiveness.

Don’t reinvent authentication, authorization, for encryption for every application and API you build. This reduces the amount of security work that must be done and simplifies the organization’s security architecture. Achieving effective enterprise security architecture in a DevSecOps manner is beyond the scope of this document, but these guidelines are reasonable first steps.

Guidelines for Effective Enterprise Security Architecture
Strategy	Description
Security libraries	Generally, it is best to use popular, well-tested libraries for security features, as it is dangerously easy to make small but disastrous mistakes writing your own. Probably the best approach here is to assemble your own enterprise security API that implements, extends, wraps, or is based on existing security libraries like Spring Security, OWASP ESAPI, BouncyCastle, OWASP Encoder, AntiXSS, AntiSamy, Jasypt, etc.
Standardize on a standard software stack	To the greatest extent possible, you want a standard approach to security, and you should use the security features provided by your software stack. This makes security invisible or automatic and reduces the likelihood of mistakes. Don't assume they're working, though; you have to test these defenses continuously.
Security services	Consider the extent to which you can turn security defenses into high assurance services that your applications can invoke. This creates the possibility of upgrading security across many applications without having to recode, retest, and redeploy. In the spirit of DevOps, make sure there is a self-service way for your empowered engineering teams to consume them without needing central approval or provisioning.

Manage the Software Supply Chain

Libraries have serious vulnerabilities, most of which have not yet been discovered by the good guys. And attacks now start within a day or so of new vulnerabilities being disclosed. So, every DevSecOps project must pay attention to the security of its supply chain.

Your software stack is composed of thousands of libraries, frameworks, modules, and components written by unknown developers worldwide. This becomes an increasing concern when you have open-source libraries or tools in your stack. These days, it is very common for developers to adopt open-source libraries such as Bootstrap, Angular JS, OpenStack, etc.

Although using these libraries can allow you to create software much more rapidly, what many may not realize is this convenience comes at a risk. Not all of these libraries are tested via security best practices, and some libraries are constantly providing new upgrades that need to be verified. This means that you must take full responsibility for the security of all of that software.

In practice, this translates to a few important strategies:

Strategies for Secure, Effective Software Management
Strategy	Description
Show some restraint	Be cautious about the libraries and frameworks that you adopt. Stick to projects that demonstrate a solid security program, evidence of security testing, and effective response to new vulnerabilities.
Self-inventory	DevSecOps organizations must know the exact version of every library, framework, application, and API that is running on every server in every environment. The best approach is to enable all your systems to self-inventory by reporting their “bill of materials” to a central database. This “always-up-to-date” inventory will enable you to respond quickly to novel attacks.
Establish secure coding guardrails	When you decide to take on a new library, standardize how you will use it safely (both positive and negative rules) and turn them into code so they can be continuously tested.
Test for latent vulnerabilities	Before you trust your business to someone else’s code, you must verify that the defenses that are provided by the libraries and frameworks work as advertised and don't contain undiscovered vulnerabilities. Very few libraries receive adequate security testing.
Continuously monitor for new vulnerabilities and respond	Modern applications are only fully composed at runtime, as their dynamic dependencies, plugins, and injections are fully realized. RASP tools (discussed above) can proactively prevent both known and unknown vulnerabilities from being exploited.

Explode, Offload, Reload

While it’s not strictly necessary, many DevOps projects use the cloud, containers, and APIs. Many organizations have already discovered that this is, in fact, the fastest path to securely achieving digital transformation. DevSecOps projects should strongly consider Ed Amoroso’s advice to “explode, offload, reload.”

DevSecOps Advice: Explode, Offload, Reload
Advice	Description
Explode	Perimeters are no longer effective at stopping attacks, so the first step is to break up your monolithic internal infrastructure into smaller distributed workloads.
Offload	Second, take advantage of security and cost advantages by moving these new, smaller workloads to virtual cloud and container infrastructure.
Reload	Finally, re-establish continuous security for each of your newly virtualized workloads by selecting and deploying modern protection technology at the application, library, container, and network layers.

This strategy is very compatible with DevSecOps, as it allows for efficient centralized security management with distributed enforcement for speed and accuracy.

Create a Security Culture

Some companies simply seem to have the ability to take security seriously, focus, and do a great job. But others — even companies that seem to be doing all the same practices and using the same tools — are simply terrible at security. The difference is culture. And while culture is a difficult thing to change, there are a few key practices that have worked in organizations and should be part of a DevSecOps program.

DevSecOps Culture in Teams

When it comes to the DevSecOps culture implementation, teamwork is key. Communication is the most important part of creating a collaborative atmosphere within teams. The goal of DevSecOps is to get everyone involved so that security does not fall under the responsibility of one person or department, but it is distributed between everyone across the SDLC.

Now, let’s look at a few strategies to create a DevSecOps culture on your team.

DevSecOps Culture Strategies for Success
Strategy	Description
Executive sponsorship	You must have support from the executive level. Executives need to clarify that security is everyone’s responsibility — and that simply getting past the compliance audit is not the goal.
Micro-training	Everyone needs to understand exactly what their security responsibilities are. The best way to achieve this is by providing instant feedback while doing their job.
Accountability	Development and operations team members should be responsible for the security of what they produce and operate. Security specialists shouldn't be in the critical delivery path and should instead act as coaches and tool smiths.
Security in sunshine	Make security as visible as possible. Be sure that vulnerability and attack data are never used to shame people. Instead, celebrate security risks as the fastest path to learning and improving. Only when security is visible can you achieve a culture of making informed risk decisions.

Section 7

What’s Next for DevSecOps?

DevSecOps is still in the formative stages. The best way to get involved is to implement DevSecOps in your organization and publish your experiences. Here are some sources of additional information:

Section 8

Introduction

What Is DevSecOps?

Key DevSecOps Themes