DedSec Application Security ·
Picture a quarterly review with a leaderboard on one slide. The team at the top has no open high-severity findings. The team at the bottom has the largest backlog in the organisation, and its champion is asked what went wrong.
Now look underneath, and in this hypothetical the picture inverts. The top team has scanning switched on for two of its fourteen repositories. The bottom team spent the quarter turning on secret detection across full git history and a second static analyser, triaged what came out, and closed more findings than anyone else. The leaderboard ranked them on the one number their good work was guaranteed to make worse.
This post is for the AppSec lead who runs a security champions programme and has to say whether it works. It skips recruiting and training, and covers the measurement underneath: how a finding comes to belong to a team, what "clean rate" can mean and how each meaning is gamed, how to normalise, why ranking on level rewards the wrong thing, and how to tell whether the champions caused an improvement.
It builds on finding identity, remediation time, suppression at scale and AI false-positive triage. Every per-team number inherits the problems those posts describe, and adds one: the team.
Sources: GitHub Docs, "About code owners" (docs.github.com); C. A. E. Goodhart, "Problems of Monetary Management: The U.K. Experience", Papers in Monetary Economics, Reserve Bank of Australia, 1975; and M. Strathern, "'Improving ratings': audit in the British University system", European Review 5(3), 1997. Every number in a worked example below is illustrative arithmetic, not a measurement of any organisation or tool, ours included.
A per-team metric is a per-finding metric grouped by owner. If the owner is wrong, every number is computed correctly and describes the wrong team. So start with the question nobody puts on the slide: how did this finding get assigned to this team?
There are usually three sources, and they disagree:
Pick one source as primary, say which on the dashboard, and record which source attributed each finding.
If you use GitHub CODEOWNERS for attribution, its matching rules become your attribution rules, and some of them are easy to miss. From GitHub's "About code owners" documentation:
Two consequences. First, a catch-all line, * @org/platform, makes every path resolve to somebody. That looks like full attribution and hides unowned code inside the platform team's numbers, so report catch-all attributions separately. Second, evaluate the file at a stated revision. A skipped line or a mis-cased pattern moves findings between teams without anyone editing an owner. Other source hosts have their own ownership rules; read theirs rather than assuming GitHub's.
In a monorepo, repository ownership is meaningless and path ownership carries the whole load. Two cases go wrong regularly.
Shared libraries. A helper in libs/http-client/ is owned by one team and called by twenty. A finding in the library's own code is the owner's; charging callers inflates their numbers with work they cannot do. A finding that exists only because a caller passes tainted data into the library has a trace through both. Attributing by sink charges the library team, by source the caller. Pick one, write it down, and keep the trace on the finding so a human can re-route it.
Generated and vendored code. Code in vendor/ or third_party/, or generated from a schema, is still scanned, but the team above it in the tree rarely wrote it and often can only fix it upstream. Mark these paths so their findings form a class of their own.
A composition-analysis finding is attached to a package version in a resolved dependency graph, possibly several hops from anything a team chose. The practical rule: it belongs to whoever controls the version. Usually that is the team whose manifest declares the direct dependency that pulls it in. With a single root lockfile in a monorepo, or a base image maintained by a platform team, control sits elsewhere. An operating-system package finding belongs to the base image's owner, not to the forty services built on it; charge all forty and one upstream fix shows up as forty resolutions. Record the dependency path on the finding so someone can tell who was able to fix it.
A dynamic scan reports against a URL, a parameter and a response, not a file. Mapping that to a team needs a route-to-service table and a service-to-owner table, both maintained by hand behind gateways and shared ingress. Some findings belong to no application: a header set at a shared edge, or a TLS setting on a load balancer, belongs to whoever owns that configuration, not to the application behind the route.
If the dashboard recomputes ownership from today's CODEOWNERS or catalogue, every historical chart is quietly redrawn after a reorg: a team that did good work in March can lose it in June. Store the owner at detection, the owner at resolution and the current owner. The current owner is accountable for today's backlog; the historical owners say whose behaviour produced the history, and trend charts should use them.
Some findings match no pattern, catalogue entry or route. Do not drop them or spread them across teams. Report them as their own row, with a count, on every per-team view; otherwise nobody fixes them and the total looks smaller than it is. A rising unattributed share means attribution is breaking (an unregistered repository, a moved CODEOWNERS file, a renamed team), and is worth an alert.
"Clean rate" sounds like one number. It is at least four, and they reward different behaviour.
This is the closest thing to a leading indicator of developer behaviour, because it measures what is being introduced rather than what was inherited. It is also the easiest to inflate without changing behaviour:
Intuitive and easy to explain. Its weaknesses are structural:
Resolved over detected, or open over total, for a period. These are less binary, but they inherit everything the remediation-time post describes: what counts as resolved, whether a closure as false positive or accepted risk counts as a fix, and what happens to findings that never close.
Every clean rate is a ratio, and the denominator is where it gets misleading.
No scanning looks perfect. A team with no analysers configured, or with scanning on only its newest repository, has no findings and therefore a perfect clean rate. In a leaderboard sorted by clean rate, coverage gaps rank at the top.
A new analyser makes a team look worse overnight. Turn on secrets detection across full git history, or add a second static analyser, and a team's open count jumps on the day it happens. The code did not get worse. The organisation can now see more of it. If the team that volunteered for the new analyser drops down the leaderboard, the next team will not volunteer.
The fix for both is the same: report coverage next to every clean rate, and never as a footnote. Coverage here means which analysers ran successfully against which repositories in the period. A clean rate with no coverage figure beside it is not a statement about security.
Every definition improves when findings close, and the cheapest closure is declaring a finding not a problem: a false-positive label, an accepted-risk decision, a suppression comment, a broad ignore rule. So clean rate cannot be read without the closure mix. For each team and period, report how findings left the open state: fixed by a code change, closed as false positive, closed as accepted risk, suppressed by rule, or gone because the file or repository disappeared. A team whose clean rate rose because it closed two hundred findings as false positives needs a different conversation from one whose rate rose because it shipped fixes. The suppression post argues that each of these is a decision record to audit, and the AI triage post explains how to measure an automated false-positive label before trusting it. Both apply here, more strongly, because a leaderboard gives teams a reason to close.
Two controls help: require a second person to approve false-positive and accepted-risk closures, at least at higher severities, and spot-audit a sample of closures per team per period.
Any metric with a threshold invites movement across it. If only high and critical findings count, a finding re-rated from high to medium has left the metric. Sometimes the re-rating is right: reachability analysis shows the path cannot be reached, or the asset is internal. Sometimes it is the cheapest way off the leaderboard. Either way it should be recorded as a decision with a reason, and the dashboard should show how many findings crossed the threshold downwards in the period. A team that downgraded many findings in the quarter it improved deserves a closer look.
Raw counts reward small teams and inactive teams. A team with one repository that nobody has touched since last year introduces nothing, because it writes nothing. A team shipping every day introduces more findings in absolute terms even if it writes safer code per change. So per-team numbers need a denominator, and each choice has a cost.
There is no correct choice, only one that fits the question. As general practice, normalise introduction by changes, normalise backlog by code size, and show the raw count next to every normalised figure so readers can see when a dramatic rate rests on a small number.
Illustrative arithmetic, not data: Team A merged 400 changes and introduced 20 findings, 5 per hundred changes. Team B merged 20 and introduced 3, or 15 per hundred. On raw counts B looks far better; per change, three times worse. But one finding more or less moves B's rate by 5 per hundred. Neither view supports a ranking.
In 1975 Charles Goodhart, writing about monetary policy in the United Kingdom, observed that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." The version usually quoted comes from Marilyn Strathern's 1997 paper on audit in British universities, which states it as "When a measure becomes a target, it ceases to be a good measure", in a passage drawing on Keith Hoskin's discussion of Goodhart's law. A leaderboard is how a measure becomes a target. Once a team's rank depends on a number, the team will find the cheapest way to move it, and the earlier sections list most of the cheap ways.
That is not a reason to have no comparison across teams. It is a reason to choose carefully what is compared.
A ranking on open backlog, the level, mostly measures history: how old the codebase is, how many analysers are on, and how long ago scanning started. It punishes the team that onboarded more scanners and rewards the one that onboarded none. Measures of flow, which describe what happened in the period, are harder to game by doing nothing:
Separate the backlog a team inherited from what it introduced since a fixed date. A team can be held to a high standard on new code and given a realistic schedule for old code. One number covering both blames current engineers for past decisions and lets old debt hide new carelessness.
Lagging indicators measure outcomes: vulnerabilities introduced, findings that escaped to later testing or production, incidents. They are what the programme exists to change, and they move slowly and noisily. Leading indicators measure activity expected to cause those outcomes: threat models a champion joined, security review comments, time from assignment to triage. They move fast and are the easiest to game; ten perfunctory comments outscore one careful one. Use leading indicators to check the programme is running and lagging ones to judge whether it works, and never rank teams on activity counts.
Here is a pattern that looks like success. At the end of the quarter, the team with the worst numbers gets a champion and extra attention. Next quarter its numbers improve. The programme takes the credit.
Part of that would have happened anyway. Every measurement has a random component: a burst of findings in one release, a legacy import, a backlog dumped on the team by a reorg. A team picked for scoring worst was probably also unlucky, and next quarter its luck is likely to be closer to average. The same effect makes the best team look complacent. Selecting on an extreme score and then measuring change makes an apparent effect likely even when nothing worked.
The defence is a comparison group, covered next, and a longer baseline: select on several periods of data rather than one, and look at the trend before the champion arrived, not just the last point.
A team with three findings that closes two has a 67% resolution rate; so does a team that closes two hundred of three hundred. Only one of those means much. Show the count beside every rate, grey out rates below a minimum count, and never rank on a percentage of a handful of findings.
Even with honest metrics, "the numbers improved after we launched the programme" does not show that the programme did it.
A simple before-and-after comparison credits the programme with everything else that changed in the same period. In application security, a lot changes:
Any of these moves the metrics with no champion involved.
If teams join in waves, the teams not yet joined are a natural comparison group, measured over the same periods with the same scanner versions. Compare the change in joined teams with the change in not-yet-joined teams. That is the reasoning behind difference-in-differences in policy evaluation, and it removes anything that affected everyone equally, such as an organisation-wide rule-pack update.
It does not remove differences between the groups. Champions are often volunteers, and teams that volunteer may already care more about security. If the first wave is made up of keen teams, its improvement partly reflects the keenness. Where you can, decide the order of rollout by something unrelated to performance, such as an alphabetical or calendar order, and write it down before you look at results.
To keep a rule-pack change from being read as a change in behaviour, every scan should record which analysers ran, their versions, and the version of the rule pack or configuration. Then:
If champions were trained on injection and access control, look at the introduction rate of those classes at the pull-request gate in their teams. A broad "total open findings" figure mixes targeted classes with everything else, diluting a real effect and hiding the absence of one.
On it:
Off it, or carefully caveated:
AppSecD gives teams a posture leaderboard and per-team attribution, with clean-rate tracking and champion enablement, and security champions get a dashboard that tracks the finding lifecycle. Underneath, findings are deduplicated across scans, move through new, reopened and fixed states, and carry ageing, SLA and assignment. The lifecycle supports maker-checker approval, so a finding marked fixed or false positive can require a second person to confirm it.
That is the whole claim. The definitions, normalisations and methods in this post are general practice, not a description of how AppSecD computes anything. Nothing here claims how AppSecD defines clean rate, what it normalises by, which ownership source it attributes from, how it handles CODEOWNERS, reorgs, dependencies or unattributed findings, or what its leaderboard ranks on. A posture leaderboard is a ranking, and everything above about rankings applies to ours. Put the questions above to us as you would to anyone else.
Back to the slide. Before anyone asks the bottom team's champion what went wrong, put coverage beside the backlog, split introduced from inherited, show how each team's findings closed, and mark the week the new analysers went on. The team at the bottom may turn out to be the one the programme is working for. The team at the top may turn out to have nothing switched on.