DedSec Security Research ·
Two services ship on the same day. One image scan reports a short list of CVEs; the other, built from a different base, reports several times as many. Someone asks which team has more work to do. Neither number answers that.
An image scan's CVE count is the product of three things: which packages the scanner could see, which advisory data it matched them against, and how it counted. The Dockerfile decides most of the first, the distribution decides most of the second, and the third is a reporting choice. None of them is a direct measure of how exposed the running container is, and several of the things that matter most about an image do not appear in the count at all.
The sources here are the Docker and BuildKit documentation, the Kubernetes documentation and kubelet source, the Debian security tracker documentation, and Trivy's coverage documentation, which is unusually explicit about what it detects and what it does not.
An image CVE scan is not a code review or a malware scan. It is an inventory followed by a lookup: find the packages in the image filesystem, then match each name and version against advisories. The inventory comes from metadata that package managers and build tools leave behind.
OS packages. On Debian and Ubuntu, dpkg records every installed package in /var/lib/dpkg/status. Alpine's apk keeps /lib/apk/db/installed. RPM-based images have an rpm database under /var/lib/rpm. Google's distroless images have no package manager, but they keep per-package dpkg status records, which is how scanners that support them see what is installed.
Language packages, from what is installed. The image case differs from the repository case. Trivy's documentation splits targets into pre-build and post-build: for a code repository it reads lockfiles, and for a container image it reads installed package metadata. In an image it reads the package.json files under node_modules rather than package-lock.json, Python's installed egg and wheel metadata rather than requirements.txt or poetry.lock, and JAR, WAR and EAR files for Java. An image scan therefore reports what was actually installed, which can differ from the lockfile, for example when a RUN pip install adds something the lockfile never mentioned.
Binaries that describe themselves. The Go toolchain embeds module and Go version information in the binaries it builds in module mode, and go version -m prints it. Stripping with -ldflags="-s -w" removes the symbol table and debug information, not the build info, so a stripped Go binary is still identifiable. Scanners read that build info for the binary's dependencies and for the Go version it was compiled with, which means Go standard library CVEs are reported against the binary. Trivy's documentation notes that it cannot know how you use the standard library, so those can be false positives, and points to govulncheck for reachability. It also notes that Go binary scanning does not work on UPX-compressed binaries. Rust binaries carry a dependency list only if they were built with cargo-auditable.
What it cannot name. Trivy's Debian page puts it plainly: apart from exceptions such as Go binaries and JAR files, binaries custom-built with make or tools installed via curl are generally not detected. A curl ... | sh installer, a tarball unpacked into /opt, a C library compiled from source, a C or C++ binary copied out of another image with COPY --from: none of these writes an entry into a package database, so none of them enter the lookup. An image can carry outdated software and still show a short scan, because the outdated parts are the invisible ones.
The sharpest case is an image built FROM scratch around a statically linked C binary. It has no package database and no embedded build info, and a scan of it reports zero. Zero there means the scanner found nothing to look up, not that nothing is vulnerable. The post on scan passed versus scan ran covers the general version of this. For images, the practical check is to look at the scan's package inventory before its vulnerability list, and ask whether the inventory is plausible for what you know is in the image.
Once the scanner has a name and a version, it needs to know whether that version is affected. For OS packages the right source is the distribution's own security data, not the upstream project's fixed-in version.
Backports. Stable distributions generally fix security issues by backporting the patch into the version they already ship, rather than moving to the new upstream release. Trivy's documentation uses CVE-2023-3269, a Linux kernel issue: the mainline fix landed in 6.5, Debian 12 lists the fix as 6.1.37-1 (DSA-5448-1), and Ubuntu 23.04 as 6.2.0-26.26. For CVE-2023-0464 in OpenSSL, the RHEL 9 fix is 3.0.7-16.el9_2, while upstream fixed releases include 3.0.9 and 3.1.1. NVD typically lists only the upstream information. A scanner that compares installed versions against upstream fixed-in versions reports every one of those patched packages as vulnerable. That is the most structural false positive in image scanning, and the fix is advisory data that understands the distribution's version strings.
Severity. Distributions rate an issue's severity in their own build and configuration. Trivy uses Debian's urgency, Ubuntu's priority and Red Hat's impact where they exist, and falls back to NVD otherwise. Its documentation gives CVE-2019-15052 as the example: Critical in NVD, Low by Debian urgency, Medium by Ubuntu priority. Two scanners can disagree about the severity of the same finding in the same image simply because one prefers vendor ratings and the other prefers NVD.
States with no fix to install. Debian's tracker documentation defines the states. <unfixed> means no fixed version exists yet. <no-dsa> means the security team decided the issue does not warrant a security advisory, for example because it is minor; such issues can still be fixed in a point update or alongside a later, more severe fix. <ignored> means no update will be provided. <postponed> means the issue deserves an advisory fix but not one of its own, and will be included in a later advisory. Red Hat's statuses include Affected, Fix deferred, Will not fix and Under investigation. None of these can be resolved by rebuilding today. Scanners usually offer a filter, such as Trivy's --ignore-unfixed, and that is a reasonable default for a pull request gate, provided the unfixed findings stay visible somewhere else, because a fix that appears later should become actionable.
Where unfixed issues are not reported. Trivy's coverage pages each carry a feature table. On the Debian, Ubuntu and Red Hat Enterprise Linux pages, the "Unfixed vulnerabilities" row is marked ✓. On the Alpine page and the Google Distroless page, the same row is marked "-". Neither page says why. The distroless page states that distroless is based on Debian and defers to the Debian page for vulnerability details, so the difference is not that Debian lacks unfixed-issue data; it is in how this scanner handles those images. Whatever the cause, an unfixed issue that the scanner does not report for an image cannot appear in its count, so comparing an Alpine or distroless image with a Debian image of the same application compares two different kinds of report, not only two different sets of packages.
Counting. Distribution advisories are tracked against source packages; images contain binary packages. In Debian 12, the openssl source package builds libssl3 and openssl, among others (Debian 13 renames the library package libssl3t64), so one CVE in the source package is a finding against each binary package installed. linux-libc-dev, the Linux userspace API headers that libc6-dev depends on, is built from Debian's linux source package, so kernel CVEs are matched against it, although a container runs on the host's kernel and not on these headers. A report that does not group by source package and CVE counts one fix several times.
Layers. Red Hat's container images carry content manifests per layer, which Trivy maps to the CPEs Red Hat's advisories use. Its documentation warns that when layers are merged, for example by docker export, the correct CPE can no longer be determined, which may lead to false detection. Flattening an image to make it smaller can make its scan less accurate.
A typical application Dockerfile adds a runtime and a handful of libraries on top of its base. Every package in the base is in your image, so the base sets the floor of the OS inventory and of its findings, and choosing it is the largest single lever on the count. Every option has a cost.
Smaller bases lower the count partly by removing software and partly by removing the scanner's ability to see it. A review has to tell the two apart.
Everything below is visible in the Dockerfile and either invisible to an image CVE scan or visible only indirectly. These are the things to read in the pull request.
A single-stage build that installs a compiler, headers and development dependencies ships them. A multi-stage build compiles in one stage and copies only the artefacts into the final stage with COPY --from=build. Docker's best-practice guide makes the same point: a small image with minimal dependencies can considerably lower the attack surface. The final image's scan then reflects the runtime rather than the build environment, and packages such as linux-libc-dev leave the count.
Two caveats. First, the builder stage still exists in the build cache. Docker's cache documentation says the default mode=min caches only the layers exported into the resulting image, while mode=max caches all layers, including those of intermediate steps. A registry cache exported with mode=max holds the builder stage's layers, so anything sensitive there is in the registry as well. Second, a binary copied out of the builder that links dynamically against the builder's libraries needs the same libraries in the runtime base.
Each instruction that changes the filesystem creates a layer, and an image is the stack of all of them. RUN rm /root/.npmrc in a later instruction adds a deletion marker in a new layer. The file is still in the earlier layer, and anyone who can pull the image can extract it from the image's layer archives. The same is true of COPY . . bringing in a .env file that a later step removes.
Build arguments and environment variables are no better. Docker's build secrets documentation says they are inappropriate for passing secrets because they persist in the final image, and BuildKit's SecretsUsedInArgOrEnv build check reports ARG and ENV keys that indicate sensitive data.
The fix is a secret mount. RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci, built with docker build --secret id=npmrc,src=$HOME/.npmrc, makes the file available to that one RUN instruction only. By default a secret is mounted at /run/secrets/<id>; the target option moves it and the env option exposes it as an environment variable instead. For git over SSH, --mount=type=ssh forwards an agent socket or key. The limit is that the mount keeps the secret out of the layer only if the command does not write it somewhere itself. A tool that copies its credentials into a config file has put that file in the layer.
This is the git history problem in a different store: deletion adds a new state and leaves the old one in place. A credential that has been in a pushed layer should be rotated, not just removed.
COPY . . copies whatever the build context contains that the ignore file does not exclude. Without exclusions that includes .git, and so the repository's full history, plus .env files, local credential files and a node_modules built on someone's laptop. Exclude them. Docker also supports a Dockerfile-specific ignore file named after the Dockerfile, such as build.Dockerfile.dockerignore, which takes precedence over the .dockerignore at the root of the context when both exist, so check which file actually applies. Copying named paths instead of the whole context avoids the question.
An image without a USER instruction runs as root unless the runtime overrides it. Switch to an unprivileged user in the final stage, and give it as a number: USER 10001 rather than USER app.
The number matters on Kubernetes. With runAsNonRoot: true, the API reference says the kubelet validates the image at runtime and fails to start the container if it would run as UID 0. The kubelet's check shows what happens with a name. If the pod sets runAsUser, the kubelet uses that value and does not consult the image. If it does not, and the image's user is a name rather than a number, the kubelet cannot verify it and refuses the container with an error saying the image has a non-numeric user and it cannot verify the user is non-root. The Restricted Pod Security Standard requires runAsNonRoot: true, so an image with a named user fails in a restricted namespace unless every pod that uses it also sets runAsUser.
COPY creates files owned by UID and GID 0 unless --chown says otherwise. For application code that is usually right: the process can read its code but cannot modify it. Use --chown for the directories the process genuinely writes, not for everything.
Tags are mutable. Docker's documentation notes that a tag such as ubuntu:24.04 may resolve to a different underlying image over time, as the publisher rebuilds it with security patches and updated libraries. Pinning by digest, as in FROM alpine:3.21@sha256:..., guarantees the same image on every build even if the publisher replaces the tag, or a failed pull if that digest is ever deleted from the registry.
The trade-off fits in one sentence: a pinned digest gets no silent changes and no silent patches. An unpinned tag picks up base patches on the next build that pulls, but you cannot say which base a given image was built from, and builds are not reproducible. A pinned digest is only safe with an update path, typically an automated pull request that moves the digest when the tag moves, so each patch arrives as a reviewed change with its own scan. If you build for more than one architecture, pin the digest of the multi-platform index, not one platform's manifest.
The build cache interacts with this. Docker's guide distinguishes --pull, which checks for and downloads a newer base image, from --no-cache, which reruns every step and so fetches current packages from apt or npm, but does not pull a fresh base. A cached RUN apt-get install layer can be reused long after the packages it installed were patched. A rebuild that reuses cache is not necessarily a patched rebuild.
COPY copies from the build context or another stage. ADD does more. Given a URL it downloads the file, given a Git repository it clones it, and given a local tar archive it decompresses and extracts it into the destination. A remote archive is not extracted by default; recent Dockerfile syntax has an --unpack flag for that. The local extraction is the review concern: ADD vendor.tar.gz /opt/ looks like a copy and unpacks an archive whose contents are not in the diff.
For remote sources, ADD --checksum verifies an HTTP resource against its SHA-256 content digest. The Dockerfile reference lists 1.6 as the minimum Dockerfile syntax version for --checksum. The current reference also accepts a commit SHA as the checksum for a Git source, without giving a separate version for that, so check the Dockerfile frontend you build with before relying on it. Use COPY for local files, and require a checksum for anything downloaded, whether through ADD --checksum or a RUN step that verifies one. Either way, downloaded software falls in the "cannot name" category above unless it brings package metadata with it.
Because every RUN is a layer, cleanup only helps inside the instruction that made the mess. Docker's guide recommends apt-get update and apt-get install in a single RUN, followed by rm -rf /var/lib/apt/lists/*, and notes that official Debian and Ubuntu images run apt-get clean automatically. --no-install-recommends keeps recommended packages, and their advisories, out of the image. apk add --no-cache and pip install --no-cache-dir do the same for their caches. A cleanup in a separate RUN makes the image no smaller.
EXPOSE does not publish a port or restrict one. Docker's reference describes it as documentation between the person who builds the image and the person who runs it; docker run -P publishes every exposed port to a high host port. List only the ports the process listens on, so the documentation is true, and check real exposure in the runtime configuration.
HEALTHCHECK tells Docker how to test that a container still works. Docker reports the result as the container's health status; Compose can make a service wait for a dependency with depends_on and condition: service_healthy; and Swarm replaces tasks that become unhealthy. A plain docker run does not restart an unhealthy container. Kubernetes does not run it; it uses the liveness, readiness and startup probes defined in the pod spec. A HEALTHCHECK that calls curl also forces curl into the image: one more package to scan, and one more tool for anyone who gains code execution.
Every package with a CVE in the final image is a finding. That is correct: presence is a fact, and a scanner should not guess about use. Priority is a separate question, and general practice is to order it by evidence.
A CVE in a library the application loads, on a path it takes, with input an attacker controls, belongs at the top of the queue. A CVE in a command-line tool the application never runs, such as perl in a full base image, belongs lower, but not at zero: after a code execution bug, the tools in the image are what an attacker uses next, and a shell, a package manager or a network client shortens that step. Kernel CVEs matched against a headers package say nothing about the host's kernel, which may be a different version or come from a different vendor; kernel exposure is assessed on the nodes, by whoever patches them. "Never executed" is also harder to show than it sounds. Dynamic loading, plugins and subprocess calls mean code can be reached by something you did not expect. The post on reachability covers how reachability is established and where it goes wrong.
The cheapest triage is removal. A package that is not in the image needs no priority.
Docker's guide describes images as immutable and a build as a snapshot taken at that moment. The remediation model follows from that: change the Dockerfile or move the base, rebuild, rescan, redeploy. Running apt-get upgrade inside a running container changes one container until it is replaced, and makes what is running differ from what was scanned.
Two operational consequences.
Both have limits worth stating.
AppSecD's infrastructure and container coverage includes Dockerfile hardening with image CVE scanning, alongside Terraform, CloudFormation, Helm, Ansible and Pulumi rules and Kubernetes RBAC, network policy and pod security context analysis. Its software composition analysis runs Trivy and OSV across npm, PyPI, Go, Maven, RubyGems, Cargo, Packagist and NuGet. Container results land in the same deduplicated finding model as the platform's other scanning, so a vulnerability is tracked once no matter which engine found it, and pull requests and commits can be gated by policy. The scanner behaviour described above is taken from Trivy's documentation, and the checklist below is general practice; neither is a description of AppSecD's checks.
An image scan's CVE count is a precise statement about the packages a scanner could name and the advisories it matched them against. Whether the image is hardened is mostly written in the Dockerfile, and that has to be read.