DevOps represents the fundamental cultural shift in software engineering towards performance: high performing teams, and high performance code.

In DevOps, security was never a primary consideration.

DevSecOps represents the reality that DevOps must grow to encompass security. Eventually, performant code will mean secure code by default – but we’re not there yet. How do we get there?

Let’s start with where we are now. Earlier this year, Checkmarx ran a survey asking CISOs about their current AppSec programs. One of our questions specifically asked: “Where are you on your DevSecOps journey?” You can see the answers below:

On the surface, this doesn’t look too bad… until you consider the details. Based on our scaling, “medium” only represents the “definition and strategy” phase. The actual process of integration and automation is where companies actually start “doing” DevSecOps, and only 1 in 5 companies surveyed have reached that stage in some form.

And let’s be clear – integration and automation are the goals of DevSecOps. DevSecOps is about taking the needs and outcomes of application security and integrating them with the processes and culture of DevOps. In 10 years, there should be no difference between “DevOps” and “DevSecOps.” DevSecOps is just what DevOps needs to be when it grows up.

DevSecOps: The Path to Maturity

OK – how do we get there? If I were to create a rough sketch of DevSecOps maturity, it would look like this:

Let’s start on the bottom. This is traditionally where AppSec finds vulnerabilities, and essentially throws them over the wall to developers and says “here, fix these.” I have some bad news for you, this is actually “Shift Left” in action. Maybe that’s flippant and a bit unfair; but it is the base level of maturity that puts organizations on the road to DevSecOps.

The next level focuses on the developer experience. Here, AppSec teams and developers alike realize that “Shift Left” isn’t really working. Not because anyone is bad, uncaring, or unintelligent, but because it is only intended to be the first step. In that stage, AppSec got tools to find and triage vulnerabilities. Now developers need tools to manage those vulnerabilities themselves without breaking their workflow. The “developer experience” stage of maturity focuses on IDE-integrations, remediation guidance, and other ways to keep developers focused without greatly disrupting their flow.

But like “Shift Left”, focusing on the developer experience eventually hits diminishing returns. Organizations will get stuck, and then they will need to begin to move towards the third step of maturity. This is where you take the foundational understandings of the first two steps, and work to define a DevSecOps culture.

DevSecOps: Worth the Effort

Culture is hard to change, but luckily, DevOps people have done it before. If you go back to the early days of Agile and Scrum, teams would hold daily standups, and then go back to working exactly the way

they had before. But, as modern DevOps organizations can confirm, it’s worth the effort. For DevSecOps, it’s also worth the effort. Here is an example of a Checkmarx customer’s journey, and you can see them go through the stages and the results:

This is a chart showing the number of vulnerabilities remediated by a Fortune 100 company, and it’s a powerful representation of what things look like when teams integrate.

If you’re curious about the types of things this customer did to get from the left to right side of the graph above, here we’ve got some examples ready based on where you are from a maturity standpoint:

Shift Left: If you just need to get something in place to start getting vulnerabilities over to developers, the easiest way is to integrate your AppSec tools with your feedback tool (be careful here, you don’t want to suddenly shunt 10,000 JIRA tickets over to the devs, so set some policies around it). Click here to see a video showing how easy that the integration is to do with Checkmarx.

Developer Experience: The easiest way to start improving your developer experience is by integrating with their IDE of choice. This is also simple to do with Checkmarx, and here is a video showing how: Watch Now!

DevSecOps: We’ll explore the keys to DevSecOps in detail in the next blog, particularly culture, automation, and speed, but we mentioned the importance of policy management in our first bullet point. While designing policy is difficult – it relies on great communication between development teams and security teams – creating and implementing policy with Checkmarx is easy. Here’s a video showing how it’s done: Watch Now!

This blog is just our first in a series on DevSecOps. Our next blog will focus on how to change culture, the need for automation, and the true meaning of “speed” within the context of DevSecOps. In the meantime, the videos I just listed are only some a few of those you can check out over on YouTube showing how easy it is for platform engineers and developers to integrate and work with Checkmarx One. Check them out here!

There are a plethora of free and Open Source AppSec testing tools available on the internet and many of them are quite good – some top-of-mind shining examples (picked at random, of course) are KICS, DustiLock, and ChainAlert. With so many free and open source AppSec tools available, it may be tempting to question, “do enterprises really need to purchase Application Security Testing solutions?”  After all, most organizations have teams of in-house developers who are naturally inclined to the line of thinking of we can build that, or why pay for something we can get for free? And any CISO would be remiss if he or she didn’t consider at least some free tools, to help keep expenses to a minimum and organizational leadership happy.

However, as tempting as it may be to roll up your sleeves and follow an entirely free and open-source approach to AppSec testing solutions, CISOs and AppSec teams need to be aware of all of the considerations when it comes to building or selecting AppSec solutions, including total cost of ownership (TCO), opportunity cost of building, integrating, and maintaining custom AppSec solutions, and keeping abreast of the ever changing attack landscape. All of these factors culminate into overall risk—risk to an organization’s data, its customers, and its reputation.

Total Cost of Ownership (TCO) and Opportunity Cost

Most free or open-source tools are technology-specific (e.g., language, framework, technology). This means to adequately cover and help secure custom application code, third-party code, infrastructure as code, APIs, or other software attack vectors (e.g., software supply chain), many different tools are needed and often one suite of tools won’t work across multiple projects. This produces a disjointed approach to AppSec within DevSecOps and relegates control to individual developers with little oversight from security teams or executives.

Pragmatically, these tools all work differently, they do not integrate, they are rigid in terms of how they may be implemented, and they do not scale to an enterprise level. To integrate effectively, organizations will need to devote man hours building, plumbing, and operationalizing them along with their required supporting infrastructure and applications (whether on-premises, or in the cloud). This also means that organizations will need to devote time and resources to updating, maintaining, and troubleshooting these solutions, ultimately increasing the total cost of ownership (TCO) of these otherwise free tools.

Moreover, AppSec leaders and CISOs need to weigh the opportunity costs of Dev and DevOps teams not focusing on building or facilitating the production of the enterprise’s actual product. As an example, if your organization’s primary product is mobile applications, any developer, DevOps, or Security time spent customizing, integrating, or modifying AppSec testing software to meet the enterprise’s needs is time not spent building mobile apps.

Deploying, integrating, and maintaining custom solutions is only half the battle. Factor in the various results across multiple scan engines, such as SAST, SCA, SCS, API Security, and Infrastructure-as-Code, it becomes readily apparent that being able to consume and prioritize results within a single platform can save significant time and overhead in identifying and remediating risks in applications.

Commercial products also come with a simple, predictable cost structure, which makes budgeting and planning AppSec initiatives and calculating ROI much easier.

Siloed Knowledge and Threat Intelligence

Most AppSec professionals (or teams) are knowledgeable about pockets of vulnerabilities, application code weaknesses, runtime environments, developer operations, technology stacks in use, and available tooling; as such, they are only able to provide expertise that is narrowly focused and may not encompass the entire attack landscape of an organization. For this reason, many (if not most) companies (even the big ones) either need or want to leverage the expertise of application security companies—from a product or partnership perspective.

Building a completely bespoke or piecewise AppSec solution introduces a significant risk of “unknown unknowns,” where organizations can potentially suffer from tunnel vision, focusing only on security vulnerabilities or exploits they or their teams have observed. This has the potential to introduce significant blind spots and overlooked vulnerabilities with potential dire consequences – with the ever evolving threat landscape, relying on internal teams or unknown (hopefully) benevolent open source contributors may result in organizations being reactive rather than proactive, subjecting it to potential, existential risk.

One particularly illuminating example is without the benefit of supply chain security capabilities, developers and AppSec teams would have to continuously monitor social media or GitHub comments for all versions of the open source packages they’re using within their projects, both explicitly and as dependencies, which could easily number in the hundreds of thousands.

Support, Maintenance, and Updates

Another pitfall to consider when it comes to open source or free solutions is they often do not offer the same level of customer support relative to that of a commercial AppSec solution.  Software or feature support aside, consultation, planning, education, and developer adoption services are all lacking or non-existent for free or open-source solutions. Consumers of these kinds of solutions (vs. commercial) will have to navigate significant overhead to manage many disparate tools, slow reactions (by maintainers of the tool) as threats emerge or change, and incomplete or defective offerings where nobody can be held accountable.

Support and services aside, commercial AppSec solutions offer consistent and reliable updates and maintenance. Commercial AppSec vendors have quality-focused teams to test and validate their application and software updates prior to their release. In contrast, open-source projects typically rely on a community of engaged volunteers to perform quality assurance functions ad-hoc, then report findings back to project maintainers.

Checkmarx itself has several open-source application security testing projects that are very active and are updated on a daily or weekly basis. Today, a quick review of our very active projects shows dozens of open issues and pull requests that ostensibly fix defects or proliferate new capabilities that are needed by the community. This merely reiterates that the responsiveness in open-source projects to issues and challenges—even for projects that are highly crowd-sourced and supported diligently—are, in most cases, not sufficient (on their own) for enterprises.

Conclusion

Ultimately, the decision to approach AppSec testing using either open source and in-house developed tools or commercial AppSec solutions boils down to risk. Organizations and enterprises have a risk budget much in the same way they have an OPEX or CAPEX budget—and the two are inversely proportional. While open source or in-house developed AppSec solutions minimize upfront or short-term costs, it’s at the expense of operations, maintainability, flexibility, and risk exposure.

For enterprises, while free and open-source tools can augment an existing AppSec program or solution, they simply cannot meet the organization’s needs in terms of ease-of-use, maintainability, or service and support on their own. 

About Checkmarx

Checkmarx’s comprehensive and integrated Application Security Testing Platform provides a robust, flexible, and extensible AppSec solution that allows organizations to leverage our years of experience, research, and understanding in an easy-to-use platform.

Reach out for a live demo today to see for yourself!

Organizations with a DevOps framework should be looking at how they can shift towards a DevSecOps mindset, bringing individuals across all technology disciplines to a higher level of security proficiency. The goal is to ensure security is built-in to application development rather than bolted on afterward. However, there is often a perception that adding security testing into the mix will negatively impact DevOps speed. But by deploying AppSec tools that integrate security with every stage of the SDLC, businesses can reduce the cost of compliance and deliver secure software faster. 

So how can organizations go about making this shift and identifying the technical tools to support it? The first step is to determine where they sit on the DevSecOps maturity scale because despite the generally accepted definition of DevSecOps, in practice, it is a concept that means different things to different organizations based on their software development and security maturity. We typically see four levels:

Level One: Checking the Box

Organizations often have the mindset to focus mainly on meeting compliance requirements. However, simply answering yes or no questions in a security audit doesn’t necessarily make the business any more secure. Therefore, organizations should aim to mature onto other levels.

This level of DevSecOps maturity is typical of companies that wait to perform security assessments until just before deployment when there’s often no time or budget remaining at this point to address high-risk vulnerabilities adequately. These tests may or may not include automated vulnerability scanning.  The high-risk vulnerabilities will often be documented in the development’s team issue tracker, which may not allow for complete organizational visibility. As a result, vulnerabilities may be silently available for exploitation with no clear plan for remediation.

Level Two: Situational Security Awareness

The next level of maturity sees organizations start to gain more situational awareness. They begin to identify the risks living inside their applications, not about fixing code but understanding the risks. So, rather than scanning for compliance, they know that there are risks worth addressing within applications.  

Teams may identify vulnerabilities through manual testing, code reviews, or even automated vulnerability scanning and then deep dive into to assess risk with greater accuracy based on proven or perceived exploitability.  The organization will typically document these high-risk issues and implement a compensating control as a permitted policy exception with a future deadline for proper remediation.

These quasi-effective compensating controls may help organizations make software changes, but those changes would properly remediate the discovered risks. Instead, the stakeholders and development team involved in the original release may be completely different as future deadlines approach. As a result, historical vulnerability documentation sits there without a clear path for resolution. 

Level Three: Remediate Everything

If organizations tried to remediate every vulnerability, they would experience a massive slowdown in DevOps. This slowdown results from the heavy lift of combing through historical code and perhaps not using AppSec solutions that help developers prioritize fixes.

Instead, organizations typically develop an understanding of how to manage that risk by designing processes to support policies that match their compliance requirements and risk appetite. F For example, many organizations have the approach to remediate high-severity and OWASP Top 10 vulnerabilities within a specified timeframe.

A higher security posture or fallout from a recently exploited vulnerability may lead to procuring new automated vulnerability scanning tools.  Lack of engineering in integrating these tools into developers’ existing tools leads to a complete re-tooling of the organization’s software development practices. Along with the new tools, a higher level of scrutiny on detected results may lead non-technical managers to impose strict policies for deploying detected vulnerabilities that may be incompatible with business goals and, at times, technically infeasible to enforce.

Managers may incorrectly interpret the slowdown as developers being unwilling to adopt new tools and practices.  The reality is that slowness may be due to a combination of several factors:

• A subset of the organization’s developers may need the training to understand secure coding concepts, including how vulnerabilities manifest and are exploited.
• Developers are having difficulty in locating and consuming vulnerability results out-of-band of their existing development tools and workflows.
• Developers being overwhelmed with the frequency of reviewing vulnerability data produced from automated scanning tools when scans frequency is not tied appropriately to the development process.

Level Four: The Holy Grail of AppSec

At the highest level of code security maturity, organizations focus on preventing the introduction of new vulnerabilities and effectively managing technical debt by integrating continuous AppSec testing without stopping or slowing new development. Training developers to write secure code and remediate vulnerabilities and procuring developer-centric solutions helps further accelerate DevSecOps. Plus, if solutions support the developer with fine-tuned results and features like best-fix-location, remediation suggestions, and multi-scan engine results correlation, they’re further supported in making fixes faster.

A higher security posture or fallout from a recently exploited vulnerability may lead to procuring new automated vulnerability scanning tools. Management understands that any attempt to “boil the ocean” by stopping all work while identifying, accurately assessing, and fixing all high-priority vulnerabilities is not infeasible.  Instead, they:

• Identify the highest-profile applications and focus on identifying high-risk vulnerabilities in those applications.
• Focus application security training on teams that maintain these high-profile applications.
• Scale the training effectiveness by involving AppSec experts in the development teams’ vulnerability triage and remediation efforts.
• Engineer the integration of vulnerability result consumption into the developers’ tools and workflows.

How to Develop Your Organization’s AppSec Maturity

While seemingly simple in principle, how do teams move through the various maturity levels without being overwhelmed by thousands of new alerts while optimizing their AppSec program and tools? There is no one-size-fits-all approach, but here are some practical best practices our Professional Services teams have seen or put into practice.

Some organizations run baseline scans on their existing applications and put findings into technical debt backlogs, which they gradually reduce over time. Other organizations set a policy that any new releases won’t introduce new vulnerabilities, ensuring compliance with automated AppSec testing kicking off directly from their DevOps environments. Checkmarx’s Application Security Platform and AppSec Maturity Services help both DevOps and AppSec teams understand and assess where they are today and what they need to do to mature their environment.

Adoption Requires Top-Down Support and Bottom-Up Enablement

Most importantly, DevSecOps needs top-down executive directives, otherwise prioritizing AppSec won’t get buy-in from the wider team. Senior management support and exemplary technical leadership are critical because changing an existing DevOps approach to integrate security includes a degree of disruption as teams adjust to the new process. Managers might purchase AppSec tools, but individuals will have differing levels of experience in implementing and using them. While developers interested in security may embrace the new process, others will be reticent about what they see as an unhelpful change in their preferred methodology. Consequently, adoption is patchy, and the tools don’t deliver ROI. Even though developers want to write secure code, their priority is to ship feature sets and functionality quickly. Hence, leadership needs to support the technical adoption of tools through procuring developer-centric products and proper training.

Where does your organization land? In level one, two, three, or four?

Download Our Ultimate Guide to SCA Here

Pertaining to development tools, DevOps is also about automation of the different tooling in use that improves the speed of software delivery. Designed primarily for those embarking on DevSecOps initiatives, Checkmarx CxFlow integrates security into the existing tools so that secure software development can be achieved without requiring any extra tools. Using the existing tools already in place, CxFlow seamlessly operates in the background.

Checkmarx CxFlow is all about the automation of AST solutions into the tooling within today’s organizations. It was developed to address AST automation head-on, but there is more to CxFlow than meets the eye. Below is a list of the top three benefits organizations will experience when using CxFlow with their CxSAST and CxSCA deployments. Let’s delve a little deeper into what CxFlow is all about.

#1 The Most Shift-left Where Automation Can Occur

Traditionally, application security testing (AST) solutions on the market operate within the CI tooling in use and scans are normally performed after a merge/build has taken place – further to the right of the SDLC, so to speak. CxFlow on the other hand, allows organizations to shift that functionality to the left since CxFlow is an orchestration layer that simplifies the implementation and automation of AST in today’s modern development environments.
Using CI plugins to launch AST scans are still supported and even recommended, for example, launching scans from Jenkins, CircleCI, Travis CI, Bamboo, TeamCity, etc.

However, AST scans can now be integrated and launched directly from code management tools as well. CxFlow effortlessly integrates with application release orchestration and agile planning tools such as GitHub, GitLab, BitBucket, Azure DevOps, etc., enabling fully automated scanning of applications and the delivery of consumable results to the developers themselves.

Plus, CxFlow integrates directly with bug/defect tracking tools like Jira, Rally Software, ServiceNow, SonarQube, etc. This eliminates the need for time-consuming manual scan configurations and allows developers to publish and update findings based on policy to a defect/backlog management system for application teams to track.

#2 End-to-End Automation, from Scanning to Ticketing

Leveraging Checkmarx’s unique ability to scan uncompiled code, CxFlow automates the steps required to scan code earlier in the SDLC. This eliminates the need for time-consuming manual configuration of scans and allows developers to publish and update scan findings based on a pre-configured policy within the code management tools themselves. After the initial configuration, AST scan activity is performed hands-off with no human intervention required whatsoever beyond a pull request initiated by a developer.
With CxFlow, upon a pull request in the code management tools in use, the developer not only receives notifications from scans as comments that are related to functionality, but also as comments that are related to security, since CxFlow easily integrates with IDEs and code management tools in use. Just like that, the developer is able to have their code reviewed once for all bugs and would be able to close the full feedback loop with ticketing systems, all while the code is still fresh in their mind.  This allows developers to:

• Catch and fix vulnerabilities during the coding phase (earliest stage of development).
• Work as usual with no disruptions, no new tools, no additional security reviews needed, etc.
• Treat security bugs and functional bugs alike and allows them to immediately address those bugs within the code branch(es) they’re currently working on.
• Reduce the overhead of manually opening, validating, and closing security tickets without spending countless hours in bug tracking/ticketing management systems.

#3 Removes Friction Between Developers and DevOps/AppSec Teams

CxFlow eliminates the manual and time-consuming configuration per project within DevOps, thereby removing the friction between developers and DevOps teams when needing to add scanning steps into the jobs of all CI pipelines, since adding jobs/steps to scan code is challenging using the older CI-scan model. Today, CxFlow:

• Simplifies the AST lifecycle through automation into the tools already in use by taking a web listener approach, listening for events from the source code repositories and triggering AST scan actions upon such an event.
• Enables developers to have little if any intimate knowledge of CxSAST or CxSCA solutions since developers receive their automated scan reviews as comments/reports right from the repositories, not the AST solutions deployed.
• Works off the concept of a protected-branching strategy whereby you can configure master, develop, and security branches, for example, that are all deemed protected. This means that pull requests, push events, etc. will trigger scans and produce results when any code changes are made that are associated with those protected branches.
• Reduces TCO and improves ROI for scanning tools by reducing the need to manage and maintain multiple CI plugins concerning installation, updates, etc., when multiple CI solutions are in use.

Conclusion

CxFlow streamlines the configuration and orchestration between the development tool set and the Checkmarx Software Security Platform to drive AST automation. With this, organizations can instantly onboard their development, security, and operations teams and simplify the governance of their security policies and DevSecOps processes.

The traditional AST solution providers are leaving developers behind, because without the ability to scan source code directly, the traditional players can’t efficiently work within code management tools similar to Checkmarx.

Clearly, integration is key to automation and CxFlow enables the most shift left approach where automation can actually occur within the SDLC—changing the way AST solutions are integrated with in all DevOps environments.

Automation is Essential for DevSecOps

We think that automating the process, particularly using Continuous Integration (CI) and Continuous Delivery (CD) tools, is essential for successful adoption of DevSecOps.

• Continuous integration. Allows a developer to integrate changes into the source code mainline as they finish writing a piece of code.
• Continuous delivery. Allows system components to be updated as needed, rather than waiting to deliver component updates in the next full release.
• Continuous deployment (also known as CD). Allows applications to be continuously deployed, often to just part of the user base at first, then later to the entire user base if the deployment is successful.

“While enabling organizations to develop software with more efficiency and speed, the DevOps process also dramatically expands risk through software exposure,” said Emmanuel Benzaquen, CEO, Checkmarx.

Beyond the Acronyms Lies a Secure SDLC

For many organizations, DevOps, CI, CD (and CD again) just amount to a lot of acronyms and words that are difficult to turn into their ultimate goal—a secure software development lifecycle (aka an SSDLC). As is so often the case, the Open Web Application Security Project (OWASP) provides a model for integrating security into any existing SDLC, which they call the Software Assurance Maturity Model. Applying this model in an organization is comprised of six basic steps, as follows:

• Assess: Ensure a proper start of the project by defining scope, identifying stakeholders, and spreading the word so people understand what you’re doing and why.
• Assess: Identify and understand the maturity of your scope in each of the 12 software security practices.
• Set the target: Develop a target score to use to measure against, to guide you to act on the most important activities for your situation.
• Define the plan: Determine a change schedule and develop or update your roadmap plan. It’s important to have a realistic change strategy in terms of number and duration of phases. Identify quick wins you can make early on.
• Implement: Work the plan by implementing all activities in this period, considering their impact on processes, people, knowledge, and tools.
• Roll out: Make sure that improvements are available and visible for everyone involved. Organize training and communicate the improvements to the team, then measure the adoption and effectiveness of the improvements implemented.

Successful Implementation of an SSDLC

It’s easy to say you’re going to implement a secure SDLC, using automation and integration. However, each of those six steps above takes time, energy, evangelizing, training, and much more. CIOs and CISOs seek ways to accelerate the maturity of their DevOps programs, but need help to get started. One way to accelerate that process is with expert services for software security deployment and automation.
Implementing automation into DevSecOps processes is a critical challenge for most organizations. An outside expert can work closely with organizations help advance the automation capabilities in their organizations and throughout their SDLC. Successful automation is just one part of implementing a secure SDLC. As outlined by OWASP in the steps above, it’s an ongoing process. Checkmarx provides customers with professional services, who share the essential guidance to develop and deploy software security programs with automation. Let’s get your organization off to a running start with DevSecOps.