Secure Code Warrior Unraveled: Impact of Secure Developers on Software Metrics

As we slowly come back to reality after a flurry of RSA-related activity in San Francisco, I have started to ponder some of the recurring conversations I’ve had with CISOs and security professionals on the ground. Many lament juggling increasingly sharp budgets, where data-backed justification of every tool and service is becoming the norm, not to mention the deluge of both AI-powered security tooling, and the new wave of AI-driven vulnerabilities they expose unsuspecting enterprise security teams to, who are scrambling to contain their attack surfaces.

We’re in a precarious, delicate period of technological advancement and change, and many of us are doing our best to navigate it despite near-daily modifications to the landscape. The industry continues to rally around Secure by Design initiatives as the future of safer, higher-quality software, in spite of ongoing CISA setbacks. One of the best aspects of RSA is the community spirit driving the partnerships, collaboration, and innovation that underpin our thriving industry, and ultimately, the immensely talented people serving as its backbone. 

When reflecting on our own technology, SCW’s Developer Risk Management platform, I am buoyed by recent metrics from one of our core enterprise clients, and going down the proverbial rabbit hole into this data tells a tale of a high-adoption, potent Secure by Design initiative that has been proven to considerably reduce risk in their organization.

The ROI of Secure Coding Engagement 

It can be difficult to quantify the true impact of security upskilling among the development cohort, and whether part of a Secure by Design commitment, compliance-driven implementation, or simply a design to modernize and uplift overall software security and quality, the raw data to prove it’s actually beneficial can be elusive. 

Through this initiative, we aimed to assess the impact of Secure Code Warrior (SCW) training on application security (AppSec) metrics, and to quantify its return on investment (ROI) in terms of time and cost savings for our client. We’ve executed about 20 ROI studies with our customers, each time achieving similar results, so I thought it was important enough to share.

SCW ingested vulnerability data from SonarQube and Apiiro, primarily from Q2 and Q4 of 2024: Two periods where vulnerability scanning and remediation efforts were actively driven. Additionally, SCW Trust Agent was deployed to correlate flaw data (typically tracked by application or repository) with secure coding proficiency data of developers (e.g., using the SCW Trust Score).

Some of our significant findings were:

• Return on Investment (ROI): 3000 software engineering hours saved, as proficient developers generated way fewer security issues and were able to fix the issues that did arise much faster, thanks to mastering secure coding. Depending on how expensive your developers are per hour, this means an estimated ROI of $94,000 (using $30 per developer hour) to $210,000 (using $70 per developer hour) 
• Strong AppSec Tooling Adoption: Our client demonstrated a 45% reduction in critical and high-risk vulnerabilities introduced and an 85% decrease in mean time to resolve (MTTR), signaling successful adoption of scanning tools across engineering teams. Most critical issues are now resolved promptly, with near-zero remaining open.
• Training in Secure Coding Correlates with Quality: Repositories with one or more unskilled developers (less proficient in secure coding) introduced 5x more security flaws than those with only skilled, secure code proficient developers, emphasizing the value of secure code skill uplifting.

Traditionally, security (and, indeed, understanding secure coding best practices) has been hindered by an image problem among the development cohort: It’s seen as a blocker to feature delivery and innovation, rather than a necessary step to have the most robust, airtight code possible. These results show that it is possible to enact secure coding at speed, reaping the benefits without the burden.

Technical Debt Running at (Almost) Zero?

Technical debt remains a bugbear of most security and engineering teams, and there are few avenues to alleviate the issue in a meaningful way, especially when many enterprises carry the burden of legacy code, monolithic codebases, and tool sprawl, adding to the overall complexity.

However, one of our most prominent findings with this enterprise is a considerable reduction in their technical debt as a result of their commitment to developer-level security proficiency and risk management:

This Vulnerability Debt Chart shows a positive movement in ‘Critical’ vulnerability discoveries starting from July 2024.

The months of July through to October 2024 saw more detections closed vs. those opened in the same month. What’s more, this trend is becoming the norm, with critical vulnerability debt remaining close to zero in recent months.

Developers With Low Security Skills Were 5x More Likely to Introduce Vulnerabilities vs. Secure Developers

With SCW Trust Agent, we are afforded granular insights into the efficacy of upskilling initiatives. When comparing our client’s repositories with commits where developers were skilled vs. unskilled, we can see a significant impact:

Looking at the data, we get a 95% certainty that what we see in the repos of both the cohort of Secure Developers and regular Developers is not just by chance, and if we repeat this exercise again, the intervals we can expect to see the same values would be:

• For “Secure Developers” repos (175): We should get anywhere between 4-7 flaws per 100 commits.
• For “Developers” (191): We should get anywhere between 26-29 flaws per 100 commits.

This is the tangible proof that is so often missing when assessing developer-driven security initiatives, pointing to a marked reduction in committed vulnerabilities as a result of precision, hands-on and high-impact, verified learning being an integral focus of the security program. These results are expected to improve over time, as more developers adopt upskilling programs that align with business objectives, and their pathways are adapted to address knowledge gaps and threat vectors of focus.

Next Steps

Developer risk management will be a key success factor for any organization looking to seriously adopt Secure by Design initiatives, as well as measure, mitigate and manage the growing risk profile of emergent AI technologies. 

Using (agentic) AI, and any AI coding assistant, will result in a massive increase in lines of code being written, at an incredibly fast speed. The role of a developer is changing, and they will need to be proficient in critical thinking, code reviews (from AI) and secure prompt engineering. Secure developers with AI will provide a major benefit for enterprises, producing 10x more code, 10x faster, and with fewer security issues. Unfortunately, the opposite is also true: Unskilled developers will produce 10x more code, 10x faster with 10x the number of security issues, resulting in a massive technical debt afterwards. 

Our case study with this enterprise has proven that this is not only possible, but also produces favorable results and improved software quality in a relatively short time. 

‍

How are you managing developer risk?

‍

> Download our all-new white paper, divulging our findings from over twenty interviews with top security professionals discussing enterprise Secure by Design initiatives.

> Learn more about coding with AI, and the learning pathways available for your developers to safely leverage the productivity gains of this technology.

> Book your demo today.