DedSec Application Security · 2026-10-14
A vulnerability ships. Someone pulls up the pipeline for the commit that introduced it, and the security stage is green. The obvious conclusion is that the scanner missed it, and sometimes that is true. Just as often, the scanner never really looked, or it looked and the result was lost before it reached anyone.
A green check means two narrow things: the job exited in a way the pipeline accepted, and the list of findings that reached the end was empty. Neither says the analysis covered the code you think it covered. This post is about that gap. We build a scanner, so these are failures we have to think about too, and the questions at the end apply to us as much as to anyone.
A security stage that reports "no findings" could mean any of these:
Most pipelines show all four in the same colour.
The simplest silent failure is a process that dies: out of memory on a large file, a job timeout, a crash on unexpected input. What happens next depends on the wrapper around it.
Exit codes are where this goes wrong, because tools disagree about what they mean. Some scanners exit non-zero precisely because they found something. Others exit zero regardless unless told otherwise, and reserve non-zero codes for errors. An orchestration layer that runs many tools usually normalises this, and normalisation can swallow either signal:
The multi-analyzer arrangement makes this sharper. AppSecD runs static analysis through a farm of more than fifteen analyzers with Semgrep as the primary engine. Any arrangement like that needs to report each analyzer's status separately from its findings, because one engine failing while the others return normal results produces a scan that looks entirely ordinary. That is a fair thing to ask of any multi-engine scanner, including ours.
A scanner can complete successfully, with a clean exit and well-formed output, against a fraction of what you meant it to read.
Language and framework detection. Analyzers decide what to read by file type and project markers. A file extension the detector does not map, a template language embedded in another language, or a framework that the rule packs do not model means no rules ran for those files. From the tool's point of view that is not an error.
Parse failures. A static analyzer has to parse a file before it can reason about it. Code using syntax newer than the parser understands, or a file with an encoding the parser rejects, is commonly skipped with a warning and the scan continues. The warning goes into a log. The summary says completed.
Exclusions that catch real code. Excluding vendored, generated and minified code is right. The trouble is the glob. A pattern meant for generated client stubs also matches a hand-edited directory of the same name. A size limit meant to skip minified bundles also skips the oldest, largest controller in the repository, which is usually where the interesting bugs are.
Analysis budgets. Engines that trace data across functions commonly have time or depth limits. When one is hit, the engine stops following the path or falls back to shallower analysis, and nothing in the result says the boundary was reached.
Output format drift. A tool is upgraded, and a field in its JSON output is renamed or moved, or a SARIF producer changes how it nests results. An ingest parser that looks for the old field does not find it. If that parser defaults a missing results field to an empty list, which is a very natural thing to write, the upgrade turns every subsequent scan into zero findings without a single error being raised.
Truncated output. A process killed mid-write leaves invalid JSON. A parser that catches the exception and moves on records nothing for that analyzer.
Unmapped rules and severities. Ingest usually maps each tool's severity vocabulary onto one scale, and each rule onto a known catalogue entry. A new rule shipped in a rule-pack update, or a severity label the mapping table does not contain, has to go somewhere. If it is dropped, or defaulted to the lowest severity and then filtered out by a display threshold, the finding existed and nobody saw it.
Fingerprint collisions. Deduplication decides whether a finding is new or one already known. It works by computing a fingerprint from things like the rule, the file and the code involved. Make the fingerprint too coarse, for example rule plus file, and a genuinely new instance in the same file merges into an old one. If the old one was marked false positive or accepted, the new one inherits that state and never appears. AppSecD tracks findings in one deduplicated model with new, reopened and fixed states. Any deduplicating system should surface a match against a suppressed finding rather than silently absorb it. Ask any vendor, including us, what happens in that case.
Suppressions and baselines are deliberate and necessary, and they are the most common way a real finding is hidden by a decision nobody remembers making.
The useful number here is not how many findings a scanner produces. It is the suppression rate: the share of raw findings hidden by suppressions, baselines and false-positive decisions, per analyzer, over time. A sudden rise usually means someone wrote a broad rule. A steady rise means the triage process is absorbing things rather than resolving them.
Two AppSecD features bear on this. AppSecD's lifecycle supports maker-checker approval, so marking a finding false positive can require a second person to confirm it. And its AI triage layer labels likely false positives with the label itself tracked, so triage quality is measurable rather than asserted.
Secrets against a shallow clone. Many CI systems check out a shallow clone by default, and the standard GitHub checkout action fetches a single commit unless told otherwise. A secret scanner asked to read history in that checkout reads the history that is there, which is almost none, and reports cleanly. AppSecD runs TruffleHog and Gitleaks over the working tree and the full git history, because a credential removed in the latest commit is still in the repository. For any tool, a full-history claim is only as good as the history the scan was actually handed, and the same applies to code that arrives as an archive without its repository metadata.
Dependencies without a lockfile. A manifest declares version ranges. The build resolved those ranges to specific versions at a specific time. A composition scanner without the lockfile has to resolve the ranges itself, and may pick different versions than the build did, or skip what it cannot resolve. AppSecD's composition analysis uses Trivy and OSV across npm, PyPI, Go, Maven, RubyGems, Cargo, Packagist and NuGet, but ecosystem coverage describes what can be read, not whether the file that pins what you actually shipped was present.
Dynamic testing that tested the login page. When an authenticated scan's session expires part-way through, subsequent requests are typically redirected to a login form or rejected with a 401, payloads come back unreflected, and checks against authenticated pages report not vulnerable. The scan completed. It spent its second half testing a login page.
Blocked is not refuted. A web application firewall or a rate limiter that answers payloads with 403 or 429 produces responses that do not match any vulnerability signature. A blocked payload has not been shown to be harmless; it has not been tested.
Out-of-band checks with no way home. Blind injection is detected by getting the target to call back to an interaction server. AppSecD's dynamic testing includes Interactsh for out-of-band detection alongside Nuclei and SQLMap. For any out-of-band check, the absence of a callback only means "not vulnerable" if the target could have reached the callback host. A target whose outbound traffic is filtered, including external DNS resolution, can return no callback whether it is vulnerable or not.
The breach-monitoring post on this site made the same point about feeds: a monitor has to distinguish "nothing found" from "source unavailable". Application scanning needs the same discipline, applied everywhere above. Whatever tools you run:
Of any scanner, ours included:
A scanner that can answer those is telling you what it looked at. One that can only show you an empty list is telling you the list is empty, and you will not know which of the four outcomes that was until it matters.