We need a custom SAST rule for our own bug pattern. How do we write one that does not drown the team? Semgrep search and taint mode, sanitisers, metavariable constraints, rule tests, and measuring a rule before it gates a pull request

DedSec Application Security ·

Suppose a team wants to catch raw SQL built with SQLAlchemy's text() function, and somebody writes the obvious rule: a regular expression for text\(. We ran exactly that rule, as a Semgrep pattern-regex, over the Python 3.12 standard library that ships with Ubuntu 24.04. Semgrep scanned 515 files and reported 389 matches. Not one of them was a call to a function named text. The standard library does not contain SQLAlchemy, so a hit on the real sink was impossible. The interesting part is what the 389 were:

What the regex matchedMatches
Names ending in context or Context, such as getcontext(, get_context( and SSLContext(182
Other names ending in text, such as read_text(, itertext( and add_text(98
The gettext family: gettext(, ngettext(, pgettext(, npgettext(39
splitext( and _splitext(37
Text inside a comment, docstring or string literal32
A definition, def text(einfo, context=5) in cgitb1

"Context" contains "text(" once you add the parenthesis, and nobody writing the rule thought of that. That is the substring false positive from the SAST false positives post, reproduced in one line of YAML. A custom rule is a small analyser that you now own, and it can have every bug a vendor's analyser can have. Anyone can write one in ten minutes. The work is in making it precise enough that the team keeps reading its findings.

This post works through that, using Semgrep as the example because its rule format is public and documented. It covers:

How we tested. Every claim about Semgrep's behaviour below is either something we saw in a local lab or a quotation from Semgrep's documentation, and the text says which. The lab used the open-source Semgrep CLI, version 1.172.0, which Semgrep's documentation calls Community Edition (CE). We ran it without logging in and with metrics off, against about a dozen deliberately small Python files and one JavaScript file written to contain each case discussed here. Semgrep's commercial engine would differ where noted. The lab code is a fixture, and its precision figures describe these rules on these files, not Semgrep in general.

Related posts cover the parts around this one. The reachability post introduces taint analysis. The suppression rate post is about what happens to findings after they are raised. The stale baseline post is about the pull-request gate a new rule eventually feeds.

When a custom rule is the right tool

A public rule set knows that cursor.execute with a concatenated string is dangerous. It cannot know anything about your own code. Custom rules are worth writing where the dangerous thing is specific to you:

A custom rule is the wrong tool when:

Before writing a pattern, write the message. One sentence on what is wrong and one on what to do instead. If you cannot say what the fix is, the rule is not ready, because every finding it produces will need a conversation.

Search mode: what a pattern actually matches

Semgrep's default mode matches code structure. A pattern is written as code in the target language, with metavariables such as $X that bind to any expression, and an ellipsis, ..., that stands for any sequence of arguments, statements or characters in a string. $C.execute(...) matches a call to a method named execute on anything, with any arguments.

Because it matches parsed code rather than text, a structural pattern does not see comments and does not match a fragment of a longer identifier. On the 515 standard-library files, a structural rule for sqlalchemy.text($Q) with a non-literal argument found nothing, which was correct. A regular expression anchored on a word boundary, \btext\(, found one match, the def line above, which is still not a call. In our lab, which does use SQLAlchemy, the structural rule found the one real call and ignored the gettext calls, the comments and a helper named plaintext.

Structural matching also understands imports. In the lab:

So write patterns against the fully qualified name, and aliases come for free. This does not extend to every way of reaching a function. A get call on a session object, whether passed in as a parameter or created with requests.Session(), did not match requests.get(...), which is correct, since it is a different method. Your rule needs a separate pattern for it if sessions matter.

The operators, and the order they apply in

A single pattern is rarely enough. Semgrep's rule syntax combines patterns with a small set of operators:

OperatorMeaning
patternsAll of the listed conditions must hold (AND)
pattern-eitherAny of the listed patterns may match (OR)
pattern-notRemove matches that also match this pattern
pattern-insideKeep only matches that lie inside code matching this pattern
pattern-not-insideRemove matches that lie inside code matching this pattern
pattern-regexMatch text with a regular expression, the operator to avoid for sinks
metavariable-regexKeep matches where a bound metavariable's text matches a regex
metavariable-patternKeep matches where a bound metavariable matches a further pattern
metavariable-comparisonKeep matches where a comparison on a metavariable's value is true
focus-metavariableReport only the part of the match bound to a metavariable

The documentation describes the evaluation order inside patterns. Positive patterns, including pattern-inside, pattern-either and pattern-regex, are evaluated and their ranges intersected. Negative patterns, pattern-not, pattern-not-inside and pattern-not-regex, are then used to filter those ranges. The metavariable conditionals run after that and can only see metavariables bound by the positive patterns. Focusing is applied last. One consequence: a metavariable that appears only inside a pattern-not cannot be tested by a conditional.

Over-matching, one fix at a time

Here is the rule a team writes first for SQL built from strings: $C.execute(...). On the lab code it reported 14 matches. Six were real injections. The other eight were:

That last group is the receiver problem: the method name is right and the object is wrong. A pattern-inside fixes it by requiring the receiver to have come from a database connection. Written out, the second version of the rule is a patterns block containing:

The pattern-inside removed the job runner. Without it, the same four shapes also matched job.execute(f"retry-{n}") in the job module. The four shapes removed the parameterised query. This version reported six matches, and only two of them were real.

It did worse than the naive rule because it now missed things, and each miss is a shape nobody listed:

It also still matched the f-string built from a constant and the int() conversion, because a shape says nothing about where the data came from. Search mode describes what code looks like. It cannot say where a value has been. You can keep adding shapes, and each one adds a little recall and a little noise, which is the point at which taint mode is the better tool. The taint section below runs the same cases through it.

Exemptions are patterns too, and they over-match the same way

The mandated-wrapper rule is the easiest custom rule to write. The policy is to call app.http_client.fetch(), never requests.get() directly. Version one is pattern: requests.get(...). It found the two direct calls in the integration code, including the aliased one, and it also flagged the wrapper itself, whose implementation has to call requests.get.

There are two common ways to exempt the wrapper, and in the lab both had a flaw:

Both are worth a test case: a file that should be exempt and a look-alike that should not.

Constraining metavariables, and the regex that is anchored on one side

The three metavariable operators turn a structural match into a precise one.

metavariable-regex is anchored on the left only. The documentation says so: "Regex matching is left anchored." A rule for calls to eval or exec, written as pattern: $F(...) with metavariable-regex on $F and the regex (eval|exec), matched four calls in the lab. Two were eval and exec. The others were evaluate(model, data) and executor_for("pool"), because the regex matched a prefix. Anchoring both ends, ^(eval|exec)$, left the two real calls. This is the substring bug again, inside a structural rule. Any regex applied to a name needs an end anchor unless a prefix match is what you mean.

metavariable-comparison evaluates values, including constants. The token rule is pattern: issue_token(..., ttl=$T, ...) with metavariable-comparison $T > 3600. It flagged ttl=86400 * 30, so the arithmetic was evaluated. It also flagged ttl=LONG_TTL, where LONG_TTL is a module-level constant defined at the top of the file, so constant propagation supplied its value. It correctly ignored ttl=900 and ttl=DEFAULT_TTL, where that constant is 900. It also ignored ttl=ttl, a function parameter whose value is unknown, and it said nothing about that call. A comparison that cannot be evaluated does not match, so silence from this rule means "not provably over an hour", not "under an hour". If unknown values matter, write a second rule, at a lower severity, for the non-literal case.

metavariable-pattern matches a further pattern against what a metavariable bound. A rule for subprocess.run($CMD, ..., shell=True, ...) with metavariable-pattern requiring $CMD to be an f-string or to contain a concatenation matched the two commands built at runtime. It ignored a constant command string and a list-form call without shell=True.

focus-metavariable narrows what is reported. With focus-metavariable: $T the token finding pointed at the ttl argument, column 34, rather than the start of the call. With focus-metavariable: $CMD the shell finding pointed at the command string. This matters beyond display. The reported location is what a reviewer reads first, it is what an inline suppression sits next to, and it is one of the inputs a tool uses to decide whether two findings on different scans are the same one. The finding identity post covers how fingerprints are built from it.

Taint mode: where the data came from

A taint rule, mode: taint, describes data flow rather than shape. The documentation's operators are:

The engine reports a sink when tainted data reaches it without passing through a sanitiser. By default a sink must match exactly, while any subexpression of a source or sanitiser match counts. The documentation lists these as exact defaults: true for sinks, false for sources and sanitisers.

Focusing the sink on the query argument is not decoration. Without it, the whole call is the sink, and a parameterised query whose second argument carries the tainted value is reported, which is the precise case a SQL rule must not flag. With focus-metavariable: $Q, the bound parameter in cur.execute("... = ?", (term,)) was correctly ignored.

The same cases, four rules

The table compares the naive search rule, the refined search rule, a first taint rule (sources, two sinks, int() as the only sanitiser) and a final taint rule that adds the guard and propagator described below. "Real" means request data reaches the query in a way that can change its meaning. All cases are in one handler module except case 7, where the handler passes request data to a helper in another module that builds and executes the query.

CaseReal?Naive searchRefined searchFirst taint ruleFinal taint rule
1. f-string of request data in executeYesHitHitHitHit
2. Query built in one statement, executed in the nextYesHitMissHitHit
3. Bound parameterNoHit–––
4. f-string of a constantNoHitHit––
5. HTML-escaped value in an unquoted integer positionYesHitHitHitHit
6. Value converted with int()NoHitHit––
7. Helper in another module executes its parameterYesHitMissMissMiss
8. Helper returns a string built from its argumentYesHitMissHitHit
9. gettext call with request dataNo––––
10. sqlalchemy.text() built from request dataYesMissMissHitHit
11. Inline isdigit() guard that returns earlyNoHitHitHitHit
12. Validator function called for its side effectNoHitHitHit–
13. Query assembled with list.append and joinYesHitMissMissHit
execute on job-runner objects (3 calls)No3 hits–––
Reported / of which real7 real14 / 66 / 27 / 57 / 6

Case 10 is a second sink rather than a different shape. The naive and refined rules only looked at execute, and the taint rules listed both sinks. The rest of this section explains the cells that are not obvious, because each is a mechanism you will meet again.

Case 8 is caught by an assumption, not by analysis

The documentation is explicit about the scope of the open-source engine. Its comparison page lists "Single function taint analysis" for Community Edition and says it "can't track data beyond a single function or file". The taint documentation says "Interprocedural taint analysis is a Semgrep Pro feature". The advanced taint page says that in Community Edition, "where taint analysis is intraprocedural, all function calls are opaque", and that "by default, Semgrep assumes that an opaque function call propagates any taint passed through any of its arguments to its output."

That explains cases 7 and 8. In case 8 the handler calls build_filter(name) and passes the result to execute. The engine does not look inside build_filter. It assumes that tainted data in means tainted data out, so it reports the sink, which happens to be right. In case 7 the tainted value goes into run_report(conn, name) and is executed inside that function. Nothing comes back out to a sink in the handler, and inside run_report the parameter is not a source, so nothing is reported.

To separate "different file" from "different function", we put three helpers in the same file as the handler:

The --dataflow-traces text output shows the difference if you know to look. For case 8 the trace runs from the source line to the sink call and has no step inside the helper. A trace that ends at a call into your own code means the finding rests on the assumption, not on a path the engine actually followed.

There is an option to flip the assumption. taint_assume_safe_functions: true makes opaque calls clean their arguments. In the lab it removed case 8, which is real. It also removed case 5, which is real, because html.escape is an opaque function call too. The option traded two true positives for the absence of C-style false positives. Whether that is a good trade depends on how your code is structured, which is exactly what a measurement on your own code will tell you.

The commercial engine's interprocedural modes, --pro-intrafile and --pro, are documented as following taint across functions and across files. We did not test them: the open-source CLI reported that it needed the proprietary engine and a login, and exited.

Covering case 7 without interprocedural analysis

There are two ways to catch a helper that executes its parameter using only single-function analysis, and the lab tried both:

Both are reasonable as separate, lower-severity rules that say "check the callers" in their message. Neither should share a severity or a gate with a rule that has seen the source.

The sanitiser that silences a real finding

Case 5 is the one to remember. The handler HTML-escapes an id from the query string and concatenates it into WHERE id = with no quotes. Escaping for HTML replaces <, >, & and quote characters. A payload such as 1 OR 1=1 contains none of those, and in an unquoted integer position it needs none, so it passes through html.escape unchanged and changes the meaning of the query.

The first taint rule reported it. We then added html.escape(...) to the rule's sanitisers. That is what happens when someone keeps one shared list of "functions that clean input" and pastes it into every rule. The finding disappeared. Nothing about the code had changed and nothing told anyone a finding had been removed. A wrong sanitiser does not produce noise. It produces silence, and silence is never triaged.

Semgrep's structure helps here if you let it. Sanitisers are declared per rule, and a rule should cover one sink class, so a sanitiser is only ever credited against the sinks in its own rule. The failure comes from sharing. A sanitiser list is a claim of the form "this function makes data safe for this sink". Review each entry against the sink class it sits next to, and never copy the list between a SQL rule, a shell rule and an HTML rule. Semgrep also has taint labels, which track different kinds of taint within one rule. The documentation covers them, and we did not test them here.

Guards the engine cannot see, and validators it can be told about

Cases 11 and 12 are both safe and look alike. They are handled differently.

In case 11 the handler checks if not item_id.isdigit(): abort(400) and then concatenates item_id into the query. Every taint rule reported it. The data-flow overview lists the reason among its design trade-offs: "No path sensitivity: All potential execution paths are considered, even though some may not be feasible." The engine does not learn anything about item_id from the condition. You cannot fix this with a sanitiser, because there is no expression to match. The options are to restructure the code so the conversion is the guard, for example int(item_id) inside a try block, or to accept and record the finding.

In case 12 the handler calls require_digits(item_id), a function that aborts on bad input, and then uses item_id. Adding require_digits(...) as an ordinary sanitiser did not help. The finding stayed, because, as the documentation explains, only that occurrence of item_id inside the call is sanitised, not the variable afterwards. Declaring the sanitiser with by-side-effect: true, and using focus-metavariable: $X on require_digits($X) as the documentation requires, removed it. After that call the variable itself is treated as clean.

That sanitiser is a claim about require_digits: that it always stops execution on bad input. If someone later edits the function to log a warning and continue, the rule keeps crediting it, and real findings disappear the way case 5's did. Put the validator's contract in a test, or in a second small rule that checks its body still raises or aborts.

Propagators: data the engine does not know has moved

Case 13 builds the query by appending the tainted term to a list and joining it. Neither taint rule without a propagator reported it. Adding a propagator, pattern: $TO.append($FROM) with from: $FROM and to: $TO, made the list tainted once the term was appended, and the joined string reached the sink tainted. The documentation notes that propagators only work within a function. In the lab, that one propagator was the difference between missing and catching a real injection. Collections, builders and string buffers are where to look for missing propagators in your own code.

When search mode is the better choice

Taint mode is not automatically better. Search mode catches things that need no source: mandated wrappers and banned calls, where "never call requests.get directly" holds wherever the URL came from; configuration shapes such as a long token TTL, verify=False or shell=True with a built string; and sinks whose inputs arrive from callers you cannot model as a source, like case 7. Many custom rule sets end up with both for one bug class: a taint rule that blocks, and a shape rule that informs.

Testing a rule

Semgrep has a test runner built around annotated example files. The documentation lists four annotations, each written as a comment on the line before the code it describes:

AnnotationMeaning
ruleid: <id>The rule must match the next line
ok: <id>The rule must not match the next line
todoruleid: <id>The rule should match here but does not yet: a known false negative
todook: <id>The rule matches here but should not: a known false positive

The test file sits next to the rule with the same base name: http-wrapper-required.yaml and http-wrapper-required.py. semgrep --test on the directory runs every rule against its file and compares.

For the wrapper rule, the test file contained:

With the second version of the rule, the runner printed 2/2: ✓ All tests passed (two rules in that directory) and exited 0. With version one, which flags the wrapper, the same test file failed. The runner printed missed lines: [], incorrect lines: [7], where line 7 is the wrapper's call, and exited 1. When we changed the todoruleid on the session call to a plain ruleid, it failed with missed lines: [22].

One behaviour to know about: when we put a todoruleid on a line that the rule does match, the suite still passed. The documentation says tests "automatically avoid failing on lines marked with # todoruleid or # todook". The runner does not tell you a known gap has closed, so search the test files for todo annotations whenever the rule changes and promote the ones that now hold.

A useful test file for a security rule contains every vulnerable shape, including alias imports and the multi-statement variant; the safe twin of each, written the way your team writes it; the look-alikes, such as a same-named function in an unrelated module; each known limit as a todoruleid or todook; and the code from the incident that prompted the rule, if there was one. The test file is the rule's specification, and reviewing it is faster than reviewing the YAML.

Validating syntax, and what a broken rule does to a scan

semgrep --validate --config <rule> checks a rule's structure. The documentation says it runs Semgrep's own lint rules, pulled from the p/semgrep-rule-lints registry pack, together with built-in checks. We infer from that, without having tested it offline, that it needs network access; our runs had it. In the lab it rejected, with exit code 2:

The more important observation is what a broken rule does to an ordinary scan. We put the misspelled rule in a directory next to a valid one and ran semgrep scan --config on the directory. The scan reported "rule validation failed", exited with code 7 and produced no results, not even from the valid rule. That is the right behaviour, because it fails loudly. But a pipeline that only distinguishes zero from non-zero may treat 7 as "findings present". One that runs the scan with || true, or uploads whatever results file exists, would show a green check from a scan that never ran. The scan-passed post is about this class of failure.

Exit codes for a successful scan matter too. With findings present, semgrep scan exited 0 by default and 1 with --error. A gate built on the exit code needs --error, or needs to read the results file.

So the minimum for a repository of custom rules is a CI job that runs --validate and --test on every change and fails on either.

Measuring a rule before it gates anything

Tests show the rule does what its author intended on examples its author wrote. They say nothing about what it does to the codebase the team actually maintains. That needs a measurement, done before the rule can block a merge.

Run it alone, over everything

Run the new rule by itself over every repository it will apply to, on the default branch, with JSON output. Running it alone means its count is not mixed with anything else's. Record:

One detail matters if you track findings over time. In our runs, the rule ID Semgrep reported was prefixed with the directory the rule file was loaded from: rules.lab.http-wrapper-required when loaded from rules/, and nosem2.lab.http-wrapper-required when the same rule was loaded from another directory. Moving a rule file therefore changes the reported ID, and any stored triage history keyed to that ID stops matching. Fix the location before the rule's findings start accumulating history.

Label a sample, and estimate precision

Precision is the share of a rule's findings that are real. To estimate it, label findings by hand. If the rule produces a few dozen hits, label all of them. If it produces hundreds, label a random sample, not the first page. The first page is ordered by file path and will over-represent whichever directory sorts first.

Use labels that keep the information you will need later. The false positives post argues for three outcomes, not two:

The mechanism is the valuable part. Ten false positives that all say "job runner execute" are one pattern-inside away from zero.

The sampling arithmetic is ordinary statistics, not a property of any tool. If you label n findings and k are true positives, the estimate is k/n, and a rough 95% interval is plus or minus 1.96 times the square root of p(1−p)/n. As a worked example, 30 true out of 40 labelled is 0.75, plus or minus about 0.13. That is precise enough to decide whether a rule is close to gate quality, and not precise enough to compare two rules within a few points of each other. Label more if the decision depends on the difference.

Measure recall where you can. Precision says nothing about what the rule misses. You have ground truth for recall in fixed bugs. Check out the commit before each relevant fix and confirm the rule fires on it. A rule written after an incident that does not fire on the incident's own pre-fix code is not finished.

In the lab, the four SQL rules came out at 6 of 14, 2 of 6, 5 of 7 and 6 of 7, against 7 real cases. That shows how much the numbers move between versions of one rule on the same code, and why a rule should be measured again after every change, not once.

Turn precision into cost

A rule's cost is roughly its false positives multiplied by the time each takes to dismiss. As illustrative arithmetic only: 400 hits at 25% precision means 300 dismissals before the 100 real findings are worth anything, while 12 hits at 90% precision means one. Detection count would rank the first rule higher, which is why the suppression rate post calls it the wrong number for judging a rule set.

Roll it out in stages

General practice for introducing a rule without the team learning to ignore it:

  1. Report only, on the default branch. Findings go to whoever owns the rule, not to developers. This is where the labelling happens.
  2. Comment on pull requests, without blocking. Authors see findings in their own changes while they still have the context, and the rule owner watches how often comments are dismissed.
  3. Block on new findings only. The gate fails when a change introduces a finding, not when it touches a file with an old one. Semgrep's --baseline-commit does this in the CLI. In a lab git repository with two existing direct requests.get calls, a branch that added a third reported only the new one. Run with the first version of the rule, which also flags the wrapper, the same comparison still reported only the new call, because the wrapper's finding existed at the baseline commit too. The baseline was computed with the same rule at both commits. A gate that stores a baseline and compares a later rule version against it behaves differently, and the stale baseline post covers what that does to blame.
  4. Burn down the existing findings separately, owned and scheduled, not left as a precondition that never arrives. Once the count reaches zero the rule can block unconditionally.

Severity and confidence belong in the rule's metadata, alongside the CWE identifier, an OWASP category where one fits, and a reference to the internal guidance. The documentation describes metadata as arbitrary data that does not affect matching, and it appeared in the lab's JSON output for each finding. Treat severity as how bad a true positive is, and confidence as how likely a finding is to be true, and set confidence from the measured precision, not from the author's optimism. A gate can then block on high severity and high confidence while lower-confidence rules only comment.

After it ships: inline suppressions are a measurement

Semgrep's inline suppression is a comment containing nosemgrep. The documentation says it goes "at the first line or preceding line of the pattern match". In the lab:

By default, suppressed findings did not appear in the CLI's JSON output at all. Running the same scan with --disable-nosem brought them back. The difference between the two runs, per rule, is that rule's suppression count. That is the cheapest way to measure suppressions without a platform.

That per-rule count is the health signal. A rule that is suppressed far more often than its siblings is either wrong, which means a rule fix, or finding things nobody will fix, which means a policy conversation. Either way, the suppression comments are a list of labelled examples from your own code, often with a reason written next to them. Read them before changing the rule.

Bare nosemgrep comments deserve a rule of their own, because they suppress every rule on the line, including ones added later. We wrote one: a pattern-regex for a nosemgrep comment with no rule ID after it. On a file containing a bare # nosemgrep it reported nothing, because the finding was on a line carrying nosemgrep, so the comment suppressed its own detector. With --disable-nosem it reported the line. Any check for suppression hygiene has to run with suppressions disabled.

Where AppSecD fits

AppSecD's static analysis runs across more than twenty languages through a farm of over fifteen analyzers with Semgrep as the primary engine, against OWASP Top 10, CWE Top 25 and custom rule packs. It includes taint analysis in v1 and v2, with interprocedural dataflow that traces source to sink across function boundaries, and policy-driven gating that blocks a pull request or commit when it introduces a violation. Its AI layer labels likely false positives with the label itself tracked, so triage quality is measurable. The lifecycle supports maker-checker approval, so a finding marked false positive can require a second person to confirm it.

This post does not describe how AppSecD loads, tests or measures a custom rule, and the rule-authoring failures above (over-matching, sanitisers credited to the wrong sink, unanchored regexes, over-broad exemptions, untested rules) apply to any engine running a rule someone wrote, ours included. This post also makes no claim that AppSecD's interprocedural analysis applies to custom Semgrep rules, so nothing here should be read as saying it catches the cross-function cases that Semgrep Community Edition misses. The AI triage measurement post covers measuring a false-positive label against ground truth, which applies to a human labelling a new rule's sample as much as to a model.

A checklist for a custom rule

QuestionIf the answer is no or unknown
Can the message say what is wrong and what to do instead?The rule is not ready
Could the dangerous API be removed instead, or is it already covered by a maintained rule?Remove it, or use the existing rule
Are sinks matched structurally against fully qualified names, with the receiver constrained where a method name is shared?Expect substring, comment and wrong-object matches
Does every name regex have both anchors?metavariable-regex will match prefixes
Is each exemption keyed to something more specific than a function name or bare filename?Look-alikes inherit the exemption
Is each sanitiser correct for this rule's sink class specifically?Real findings disappear silently
Are side-effect validators declared with by-side-effect, and their contract tested?False positives now, silent misses later
Are list and builder operations covered by propagators?Flows through collections are missed
Does the rule need data followed across functions or files, and does your engine do that?Add a shape rule or parameter sources
Is there a test file with shapes, safe twins, look-alikes and todo annotations, run with --validate in CI?Regressions and typos go unnoticed
Does the scan job fail on a non-zero exit that is not "findings"?A broken rule can switch the scan off
Has it been run alone over every repository, a random sample labelled with mechanisms, and precision estimated?You do not know its volume or noise
Does it fire on the pre-fix commit of the bug that motivated it?You do not know its recall
Is it rolled out report-only, then comment-only, then blocking on new findings?The team learns to ignore it
Are severity, confidence and CWE in metadata, with confidence from measurement?The gate cannot tell certain from speculative
Is its suppression count tracked, with bare nosemgrep checked with suppressions disabled?Quality decays unseen
Is the rule file's location final, so its reported ID is stable?Triage history is orphaned on a move

Back to the regex that matched getcontext(. It was syntactically valid, it would have passed --validate, and with a test file written by the person who wrote the regex it would probably have passed --test too. Only running it over code its author did not write showed what it really matched. That is the general lesson for custom rules. A rule's author sees the cases they were thinking of, and the codebase contains the rest. Measure on the codebase before anyone else has to read the findings.

More about AppSecD · All posts