In particular, the regulations that will impact cyber and application security teams in 2024 are the EU Cyber Resilience Act (CRA), the Digital Operational Resilience Act (DORA), and the Network and Information Security Directive (NIS2).

• The CRA introduces mandatory cybersecurity requirements for hardware and software products, throughout their whole lifecycle, and is expected to come into force in 2024; manufacturers will have to apply the rules no later than 36 months afterwards.  

• DORA is a crucial legislative framework that mandates operational resilience for financial institutions within the EU. DORA comes into force in January 2025, and it requires organizations to prepare for and withstand operational disruptions, including cyberattacks and technology failures.

• The NIS2 Directive is the most comprehensive European cybersecurity directive to date. It has stricter requirements for risk management and incident reporting, wider coverage of sectors, and more hard-hitting penalties for non-compliance. This will impact hundreds of thousands of organizations who will need to reassess their cybersecurity posture. The EU introduced the NIS2 Directive in January 2023, and it becomes law in October 2024.

Like any new legislation, understanding the precise language used can be daunting. Here I examine what NIS2, the most imminent new regulation, means for application security teams.

NIS2 Explained 

The NIS2 Directive is an updated EU cybersecurity law that builds on the original NIS Directive (NISD). The goals of NIS2 are to boost cybersecurity, simplify reporting, and create consistent rules and penalties across the EU. By expanding its scope, NIS2 requires more businesses and sectors to take cybersecurity measures, with the goal of raising the standard of Europe’s cybersecurity performance in the long run. With stricter rules to overcome previous limitations, NIS2 impacts a wider range of industries. Entities under NIS2 are classified as essential or important, and the directive outlines security requirements as well as a process for incident reporting. It is estimated that 160K+ companies will be affected by NIS2, with a €10 million maximum fine for non-compliance. 

There were several factors that necessitated the replacement of the previous NISD. These factors primarily revolved around the consensus that the legislation needed to be more stringent, and that its implementation required a greater level of uniformity across the EU. This was based on evidence in a 2020 study by ENISA which found that EU organizations allocated 41% fewer resources to information security than their US counterparts, despite NISD being in place for four years.  The report also highlighted that there was unclear guidance around how to apply the Directive. Layer onto this a significant rise in cyberattacks, with organizations across Europe increasingly affected by ransomware and other types of cyberattacks. Additionally, there was a perceived lack of transparency in the reporting of cyberattacks.

Reporting Obligations and Risk Management

To this point, the NIS2 Directive mandates the reporting of “significant incidents” within 24 hours and less significant incidents within 72 hours. Effectively, if you are hacked and the impact will affect your customers and partners, disrupting the products or services you deliver, you must tell the relevant authorities through prescribed channels. If you fail to do this correctly, your organization and its directors can be publicly named as being non-compliant, and fines or other sanctions may be issued.

The directive requires organizations to take a risk management approach to cyber security. Organizations must identify and reduce risk as far as possible, then implement robust procedures to manage incidents. AppSec plays a critical role in risk reduction by providing visibility over vulnerabilities so they can be remediated before they are exploited. An effective AppSec program will contribute significantly to minimizing the number of incidents that have to be reported. In contrast, if you are regularly reporting incidents, you can expect to find your AppSec program under investigation by authorities.    

Therefore, AppSec managers must take appropriate technical, operational, and organizational measures to manage the risks posed to the security of their systems, and to prevent or minimize the impact of incidents on recipients of their services. Additionally, AppSec managers are responsible for making sure their developers are properly trained and that the quality of software development is being maintained. AppSec managers must be able to prove to authorities that they have robust processes for software development and that they deploy secure applications into production.

Every company is part of someone’s supply chain 

Today the regulatory environment is increasingly focused on supply chains, with Biden’s Executive Order 14028 introduced in 2021 now joined by NIS2. Even organizations that aren’t directly in the scope of these regulations will find they are affected if they want to sell to companies that are. Every company is part of someone’s supply chain. 

In part, that’s because open source software (OSS) has become integral to software development. Its use is widespread, making up on average 80% of a typical code base. However, open source packages bring inherent risks such as vulnerabilities and license non-compliance. So, having clear visibility over your open source libraries as well as knowing how your suppliers are protected will be paramount. A sobering thought: the US Securities and Exchange Commission (SEC) recently charged SolarWinds and its CISO with fraudulent internal controls for failing to disclose known material cybersecurity risks and vulnerabilities. While these were risks that were known but not disclosed, organizations are also liable for risks that they fail to identify due to monitoring and due diligence failures. 

NIS2 addresses supply chains in Article 22 and AppSec managers will need to pay close attention to this. Here at Checkmarx our Checkmarx One platform enables AppSec teams to better manage open source and software supply chain risk. It integrates a comprehensive suite of AppSec solutions including SAST, SCA, SCS, API Security, DAST, Container and IaC Security. We believe it’s not just about complying with this new Directive and finding risk but remediating it across the entire application footprint and software supply chain with one seamless process that simplifies compliance for everyone. 

So, what steps should AppSec managers take to get ready for NIS2 compliance?  If you want to learn more, register for our NIS2 webinar here.

For AppSec managers and CISOs it’s important to take reasonable action so that they and their board of directors can sleep well at night without having to worry about cyber incidents. Incidents will continue to happen – we all know that, and it’s part of the reason why regulations like NIS2 exist. The focus should be on doing what you can to prevent them, and preparing our environment so we can follow the rules if an incident happens.

Unfortunately, that’s what happens with organizations when they are faced with the barrage of information on what tools to use, what vulnerabilities need the most attention, and find themselves in a web of unstructured processes that keep them from moving forward. 

What good are the best AppSec tools, if you don’t have the right strategy, processes, and implementation in place to use the tools efficiently and effectively?  

This is why understanding your AppSec program’s maturity matters.  

You don’t know what you don’t know. If tools are not being used consistently, effectively, or at all, AppSec programs can fall into a dangerous pattern of working on the least critical aspects, while unknowingly creating larger blind spots and issues in the future.  

The more maturity an AppSec program has, the more continuously those programs build in security throughout their software development lifecycle (SDLC). The problem is, it’s difficult to keep all the moving parts working in tandem. Each tool used must be used consistently, and effectively, to remediate confirmed vulnerabilities, protect critical assets, and keep stakeholders in the loop of timelines and risk. A more mature AppSec program signifies a proactive, rather than reactive, stance to security. A mature organization doesn’t merely respond to threats; it anticipates and mitigates them, significantly reducing the window of opportunity for malicious users. This forward-thinking approach saves valuable time and resources, while safeguarding business continuity.   

The limitations of alternative AppSec maturity models 

Maturity models in AppSec and DevSecOps are not new. There are existing frameworks, however, they are not without their downsides. 

1. Information overload: Some assessments can be overwhelming for someone who just wants to get a high-level overview of the current situation and get an indication of where to get started. It can be difficult for AppSec managers to have to go into a great level of detail before they have the chance to make improvements to their current SDLC. Many of these assessments can take several days or even a week.  

2. Long and tedious timelines: Some existing models have too many results for the user, which might lead them to lose sight of what their immediate priority should be. Furthermore, these models tend to lead users far into the future typically building a detailed plan for 4 or 5 phases ahead of where they currently are. While looking into the future is great, these methodologies don’t work in modern development organizations. Because of constantly changing environment organizations have adapted to invest in only the right amount of planning: make detailed plans in the short term and longer-term plans only at a high level – this is aligned with enterprise Agile frameworks such as the Scaled Agile Framework (SAFe). 

3. Stakeholder concerns: SDLCs and AppSec programs have many different stakeholders. Each stakeholder has different responsibilities, and perspectives, on the overarching process and outcome. Developers are often focused on meeting their objectives, which means security can fall off their priority list since their focus is on getting the required functionality in place. However, to other stakeholders such as CISOs or AppSec managers, the software development organization should also be aware and manage application risks. Depending on the type of software, these risks can be substantial. Existing frameworks typically focus on one of these perspectives. For example, some other frameworks are focused on the management perspective, while others are focused on the developer perspective.  

So, how do we address these challenges, while still trying to give users a way to understand their AppSec maturity? Checkmarx developed the AppSec Program Methodology and Assessment (APMA) methodology – it takes a simpler, more pragmatic, and more agile, approach. 

How is the AppSec Program Methodology and Assessment (APMA) different? 

We introduced the AppSec Program Methodology and Assessment (APMA) methodology in a previous blog post. APMA is different in that it has a low barrier to entry, does not require a long planning horizon, and takes all stakeholders’ perspectives into consideration. It still has the benefit of a maturity model, in that it gives you a baseline of a current state as a starting point for improvements that help you see the progress you’re making, and how you quickly bite off chunks of the backlog to the reach desired state. 

How does it work and how does it differ from other application security methodologies? 

The steps of the full methodology are:   

1. Identifying gaps by speaking with your Checkmarx expert to get an overview of the current situation.  

2. Agreeing on a target or desired state, considering your goals and AppSec best practices. 

3. Working in short iterations with a few actions (sprints) to gradually close the gap and reach the desired state.  

With the APMA Premium Assessment, you will work with one of our AppSec advisors. The interview process is short – taking on average 1-2 hours. Based on the interview, our AppSec advisors will create an assessment report and come to an agreed plan. This report includes the recommended actions for the next sprint and a preview of the following sprint. It also includes an overview of the backlog of actions to reach the desired state. 

Another AMPA offering is the Comprehensive Assessment. This is an assessment where stakeholders from different functions are interviewed. The purpose of this is to break down organizational siloes and bring in multiple perspectives. This allows organizations to notice the differences in perceptions of the AppSec program, and gaps in communication, which can then be addressed in the APMA report following the assessment. 

Introducing APMA Digital – Try it for Free! 

In addition to the premium and comprehensive assessment, we are excited to introduce APMA Digital, a free and fully automated way to receive an APMA assessment that will give you actionable recommendations within minutes. 

And, best of all, this is open to everyone – whether you are a Checkmarx user or not – at no cost and the results are available within minutes. Get started now. 

In just 15 minutes, you can start your path to AppSec maturity. 

What’s involved? 

• A short self-service questionnaire with just a few quick questions. 

• Get an automatically generated report analyzing your AppSec posture and recommended areas of improvement for your next sprint. 
• Take the results and start improving your AppSec maturity. 

Field-proven with the world’s leading enterprises 

We recently completed our 100th APMA premium assessment. The largest enterprises have seen significant value from these assessments.  For example, a market-leading digital travel platform provider said: “The thorough examination of various aspects of our processes, along with identifying potential bottlenecks and inefficiencies, has given us a clear roadmap. With this newfound understanding, we can prioritize our efforts and allocate resources more effectively, enhancing our overall performance.” Or Christophe Piquet, AppSec Manager at Cdiscount  said “The APMA methodology elevated the discussion to the overall spectrum of an AppSec program and zoomed out from the day-to-day discussion that usually is driven by a tactical or operational issues to fix.”  

15 minutes to start your path to AppSec maturity 

Are you ready to find and focus on the most critical issues, maximize your return on investment into AppSec, and increase developer buy-in? Investing in AppSec maturity isn’t just about reducing risk and preventing business-critical incidents. It’s about fostering a culture of security within the organization, ensuring compliance, preserving brand reputation, and instilling confidence with internal and external stakeholders around your commitment to AppSec. 

All it takes is 15 minutes, and you’ll have an idea of where you can take immediate action to help your AppSec program become more efficient and impactful, and your organization more secure. Take the first step now and perform a self-assessment and get an APMA report within minutes here! 

Peter Drucker is usually credited saying “If you can’t measure it, you can’t manage it.” Yet others say that he never said it like that and also that it’s a fallacy[1]. We also believe that, like Drucker, that the first role of a manager is a personal one: “It is the relationship with people, the development of mutual confidence, the identification of people, the creation of a community. This is something only you can do.”[2]

However, this does not mean that measuring is not important, it just means that the inversion of the argument is not true, i.e., measuring is very important for management, but it is not the only important factor of management. Without measuring how would you be able to know if you are progressing towards your goals and success criteria? And, if you see that you are not progressing or not progressing as planned aren’t metrics at least one good option in trying to understand the issue and look into ways to improve the situation? So, measuring is important, and these questions also give us some indications about how metrics should be designed.

Link to goals and objectives

But let us come back to what this means for an AppSec program: When we design AppSec programs we view it as the foundation to know what the specific goals and objectives the program. By clarifying the goals and breaking them down them into objectives (or milestones) we can also plan better success criteria and then plan how to work towards them. For working towards the success criteria there can be soft factors like job satisfaction and organization culture, in our case secure development culture and the job satisfaction of the stakeholders of the software development organization, but there are also factors that boil down to questions about hard numbers such as “How many critical vulnerabilities do we have?”, “Is the number of vulnerabilities going up or down?” and “Is the risk from application level vulnerabilities[3] at an acceptable level to the business?”

Learning from other disciplines

Sometimes it is valuable to lean on shoulders of other disciplines who have the same or similar requirements. For a software application in production, it is quite similar if an outage is caused by a bug or a directly exploitable vulnerability. Therefore, it stands to reason that the ways to measure progress in managing bugs are similar to managing vulnerabilities. There are of course differences, for example, that risk to information disclosure or data integrity is usually caused by vulnerabilities and that the damage to the business from information disclosure or altered data can be even bigger than a system outage, but that mainly means that the problem is even bigger so there is even more motivation to address an exploitable vulnerability than a bug. Learning from long-standing background and experience in software quality assurance (QA) testing there are 3 important ways to measure this at a business level:

1.) Escaped defects

This metric refers to the number of defects that have “escaped” the QA process and are found in production, i.e., in software that is released. This is one of the most important metrics in QA since it is tied to the performance of the QA practice.[4]

2.) Defect distribution

Usually, this metric refers to when the defects are found in the development cycle, i.e., in unit testing, and of course the aim is to find them as early as possible, which means to shift left (note to the AppSec experts among the readers: sounds familiar?). Defect distribution measures how many defects were found earlier compared to later in the testing cycle, for example, unit testing vs. integration testing vs. escaped defects.

3.) Defect density

This metric refers to the number of defects in relation to the size of the release or software. The problem is however, how do you measure the size of a release or software? There are different ways. All of them are not perfect but the best available today is still to count the number of lines of code (loc). Another option is to count the amount of time that went into developing the software or to count story points[5]. But it is not always easy to get reliable data for other measures of density. LOC data is normally reliable to obtain but it can depend on whether some code in auto-generated and how much custom code is needed may depend on the programming language.

How to design KPIs for the business/strategic perspective of AppSec

As stated above the challenge of bugs/defects in software in production is quite similar or almost the same as vulnerabilities in software in production. Therefore, we can learn from the best practices from QA testing also for AppSec testing. Some of the key requirements for designing and using KPIs are:

1. Measure only what is part of the goals and objectives
2. Be able to “drill-in”
3. Create a baseline and track values over time
4. Make the metrics available to all stakeholder and as live as possible
5. Indicators should not fluctuate unnecessarily
6. Combine measures to give executive leadership one score to track

(1) The business goals and objectives for AppSec typically revolve around ensuring that the risk from application-level attacks is at an acceptable level to the business, i.e., the metric should endeavour to measure and quantify risk. What constitutes risk, e.g., if it is compliance risk, reputational risk, legal risk, or risk from a loss of revenue directly tied to a production outages due to software being unavailable requires a more detailed analysis. When you design your metrics or key performance indictors (KPIs) you should make sure that you only measure what is included in your goals and objectives and nothing else. The aim should be to measure as much as possible of the goals and objectives in as few as possible KPIs, ideally in just one metric at executive level.

You should be able to “drill-in” to the metric (2). This mean you need to be able to break down the number into individual components. For example, when using the defect density of the whole software development organization as a metric, you should be able to drill down into the defect density of each application that the organization in developing, so that areas with lower performance can be looked into, addressed, and improved. This is the best and sometimes the only way to work towards overall improvements. This also requires (3) to baseline and track values over time, otherwise you cannot compare the performance of one area (e.g., one application) with the performance of the same area in a previous time period and therefore don’t know if the area has progressed or regressed. In order for the organisation to work together on improvements it is crucial that everyone is able to monitor the metrics (4) therefore be able to act if the values decline. This can also help in driving some level of healthy competition between teams because no team normally wants to be at the bottom of the list in terms of performance or even worse than another team they relate to.

Furthermore, indicators should not fluctuate unnecessarily (5). This can maybe best be explained by an example of a KPI that has a serious flaw in this sense: Mean time to remediate (MTTR) is a common KPI that measures how much time (usually measured in days) it takes to remediate an issue after it was found. However, this metric only captures issues that are actually remediated. This may mean that when a software organisation or team starts remediating issues that were identified a long time ago, the value will actually temporarily go up. Therefore, this penalizes good behaviour if it is not at least viewed in conjunction with the number of vulnerabilities that are still open. The amount of time that a vulnerability is open can of course be relevant, but our recommendation is to look first at the vulnerability age (independently of whether the vulnerability was already fixed or not). Only in practices that have a high maturity it can be interesting to also look at the MTTR, but it should not be the first and most important KPI.

Lastly (6), it is a good practice that has proven to be effective to combine measurements into one weighted risk score value. For example, relevant measurements from different testing tools such as SAST and SCA and findings of different criticalities should be combined into one weighted score. That way executive leadership can review this score on a regular basis, e.g., monthly and see if it is developing in the right direction. The metric only needs to be agreed once and afterwards can be tracked and reported on a regular basis which should mean that the business can rest assured that the risk from application-level attacks is appropriately managed. If it does not develop in the right direction, you can drill-into it (see above) understand the root cause and address it.

KPIs for different levels – leadership/strategic and management/tactical

In QA as well as in AppSec there are KPIs/metrics that are relevant for different levels of the organization. For example, in QA testing metrics such as test coverage and code coverage are important on a tactical level and should be improved over time, but they are more important at the level of the QA management and less for the executive leadership. The number of escaping defects will already reflect if the test coverage is sufficient. Therefore, executive leadership does not have to track test coverage whereas as a QA manager this metric is important to work towards reducing escaping defects. Similarly, in AppSec testing there are findings that are detected in code that is already in production and findings that are found in development branches that are not yet in production (feature branches). The latter are not causing risk to the business, but the density of such issues is relevant from a tactical perspective to make sure fewer new vulnerabilities are introduced in newly developed source code.

Conclusion for AppSec KPIs

In conclusion our recommendation is to only measure defect density vulnerabilities from different testing types such as SAST and SCA for code that is already in production (i.e., release branches). The defect density should be weighted by the criticality of the finding, i.e., higher criticality findings should have a higher weight in the calculation. This metric is the equivalent to a combination of “escaped defects” and “defect density” in QA and can serve as the only value to track at executive level.

Other metrics, such as vulnerability aging and defect distribution are also important on a AppSec management level (but not necessarily at executive level). Defect distribution can compare how many defects were identified in code that is not yet in production compared to vulnerabilities detect in source code that is already in production and the ratio should be reduced over time. Vulnerability aging is a metric that should be reduced over time, and it will contribute to reducing the vulnerability density over time: The shorter the time vulnerabilities are open the fewer of them there will be. It is therefore a tactical level KPI that will help to improve the long term strategic KPI.

Conclusion

Finding the right KPIs for AppSec testing can be difficult but it is crucially important as one handle to manage application-level risk over time and this risk is one of the biggest risks to a lot of organizations whose main business process relies on software or is software. In our opinion it is valuable to lean on other related disciplines and learn from the past to inform decisions about KPIs. We hope this article gave some insights from our practice into which metrics and KPIs to use. Please contact the authors if you have comments or further questions. The considerations presented in this article is one of a number of best practices in AppSec management and practice that we gather and use in the Checkmarx AppSec Program Methodology and Assessment (APMA) Framework. For more information, please see our APMA page.

This article originated from joint work of Yoav Ziv and Carsten Huth at Checkmarx, Yoav with a long-standing background and experience in QA management, Carsten with a similar background and experience in application security. Working together in the customer-facing application of the topics discussed in this article, we gained insights which we find valuable to share with practitioners in application security and information security.

[1] Anne-Laure Le Cunff: “The fallacy of ‘what gets measured gets managed’”, see https://nesslabs.com/what-gets-measured-gets-managed

[2] Peter Drucker, according to Paul Zak, see https://www.drucker.institute/thedx/measurement-myopia/

[3] Despite the esoteric introduction we talk about AppSec in this article.

[4] https://www.testim.io/blog/qa-metrics-an-introduction/

[5] Story points are used in Agile and Scrum methodologies for expressing an estimate of the effort required to implement a product backlog item or any other piece of work.