DedSec Application Security ·
Suppose a team wants to catch raw SQL built with SQLAlchemy's text() function, and somebody writes the obvious rule: a regular expression for text\(. We ran exactly that rule, as a Semgrep pattern-regex, over the Python 3.12 standard library that ships with Ubuntu 24.04. Semgrep scanned 515 files and reported 389 matches. Not one of them was a call to a function named text. The standard library does not contain SQLAlchemy, so a hit on the real sink was impossible. The interesting part is what the 389 were:
| What the regex matched | Matches |
|---|---|
Names ending in context or Context, such as getcontext(, get_context( and SSLContext( | 182 |
Other names ending in text, such as read_text(, itertext( and add_text( | 98 |
The gettext family: gettext(, ngettext(, pgettext(, npgettext( | 39 |
splitext( and _splitext( | 37 |
| Text inside a comment, docstring or string literal | 32 |
A definition, def text(einfo, context=5) in cgitb | 1 |
"Context" contains "text(" once you add the parenthesis, and nobody writing the rule thought of that. That is the substring false positive from the SAST false positives post, reproduced in one line of YAML. A custom rule is a small analyser that you now own, and it can have every bug a vendor's analyser can have. Anyone can write one in ten minutes. The work is in making it precise enough that the team keeps reading its findings.
This post works through that, using Semgrep as the example because its rule format is public and documented. It covers:
How we tested. Every claim about Semgrep's behaviour below is either something we saw in a local lab or a quotation from Semgrep's documentation, and the text says which. The lab used the open-source Semgrep CLI, version 1.172.0, which Semgrep's documentation calls Community Edition (CE). We ran it without logging in and with metrics off, against about a dozen deliberately small Python files and one JavaScript file written to contain each case discussed here. Semgrep's commercial engine would differ where noted. The lab code is a fixture, and its precision figures describe these rules on these files, not Semgrep in general.
Related posts cover the parts around this one. The reachability post introduces taint analysis. The suppression rate post is about what happens to findings after they are raised. The stale baseline post is about the pull-request gate a new rule eventually feeds.
A public rule set knows that cursor.execute with a concatenated string is dangerous. It cannot know anything about your own code. Custom rules are worth writing where the dangerous thing is specific to you:
raw() escape hatch, a token service whose ttl should never exceed an hour, a template helper's unsafe=True flag.A custom rule is the wrong tool when:
Before writing a pattern, write the message. One sentence on what is wrong and one on what to do instead. If you cannot say what the fix is, the rule is not ready, because every finding it produces will need a conversation.
Semgrep's default mode matches code structure. A pattern is written as code in the target language, with metavariables such as $X that bind to any expression, and an ellipsis, ..., that stands for any sequence of arguments, statements or characters in a string. $C.execute(...) matches a call to a method named execute on anything, with any arguments.
Because it matches parsed code rather than text, a structural pattern does not see comments and does not match a fragment of a longer identifier. On the 515 standard-library files, a structural rule for sqlalchemy.text($Q) with a non-literal argument found nothing, which was correct. A regular expression anchored on a word boundary, \btext\(, found one match, the def line above, which is still not a call. In our lab, which does use SQLAlchemy, the structural rule found the one real call and ignored the gettext calls, the comments and a helper named plaintext.
Structural matching also understands imports. In the lab:
from sqlalchemy import text followed by a bare text(...) matched a pattern written as sqlalchemy.text(...)from requests import get as http_get followed by http_get(url) matched requests.get(...)from flask import request followed by request.args.get("q") matched flask.request.args.get(...)So write patterns against the fully qualified name, and aliases come for free. This does not extend to every way of reaching a function. A get call on a session object, whether passed in as a parameter or created with requests.Session(), did not match requests.get(...), which is correct, since it is a different method. Your rule needs a separate pattern for it if sessions matter.
A single pattern is rarely enough. Semgrep's rule syntax combines patterns with a small set of operators:
| Operator | Meaning |
|---|---|
patterns | All of the listed conditions must hold (AND) |
pattern-either | Any of the listed patterns may match (OR) |
pattern-not | Remove matches that also match this pattern |
pattern-inside | Keep only matches that lie inside code matching this pattern |
pattern-not-inside | Remove matches that lie inside code matching this pattern |
pattern-regex | Match text with a regular expression, the operator to avoid for sinks |
metavariable-regex | Keep matches where a bound metavariable's text matches a regex |
metavariable-pattern | Keep matches where a bound metavariable matches a further pattern |
metavariable-comparison | Keep matches where a comparison on a metavariable's value is true |
focus-metavariable | Report only the part of the match bound to a metavariable |
The documentation describes the evaluation order inside patterns. Positive patterns, including pattern-inside, pattern-either and pattern-regex, are evaluated and their ranges intersected. Negative patterns, pattern-not, pattern-not-inside and pattern-not-regex, are then used to filter those ranges. The metavariable conditionals run after that and can only see metavariables bound by the positive patterns. Focusing is applied last. One consequence: a metavariable that appears only inside a pattern-not cannot be tested by a conditional.
Here is the rule a team writes first for SQL built from strings: $C.execute(...). On the lab code it reported 14 matches. Six were real injections. The other eight were:
int() before it reached the queryexecute method on job-runner objects, which have nothing to do with SQLThat last group is the receiver problem: the method name is right and the object is wrong. A pattern-inside fixes it by requiring the receiver to have come from a database connection. Written out, the second version of the rule is a patterns block containing:
pattern-inside: $CUR = $CONN.cursor() followed by ...pattern-either of four shapes: $CUR.execute(f"...", ...), $CUR.execute("..." + $X, ...), $CUR.execute("..." % $X, ...) and $CUR.execute("...".format(...), ...)The pattern-inside removed the job runner. Without it, the same four shapes also matched job.execute(f"retry-{n}") in the job module. The four shapes removed the parameterised query. This version reported six matches, and only two of them were real.
It did worse than the naive rule because it now missed things, and each miss is a shape nobody listed:
sql = "..." + term followed by cur.execute(sql) is not any of the four shapes, because the argument to execute is a plain variable."SELECT ... '" + name + "'" parses as ("SELECT ... '" + name) + "'". The outermost operator's left operand is not a string literal, so "..." + $X does not match it. We confirmed this with that pattern alone. It matched the two-operand concatenations in the lab and not the three-operand one. The deep-expression operator, <... "..." + $X ...>, which matches the pattern anywhere inside the argument, does catch it.It also still matched the f-string built from a constant and the int() conversion, because a shape says nothing about where the data came from. Search mode describes what code looks like. It cannot say where a value has been. You can keep adding shapes, and each one adds a little recall and a little noise, which is the point at which taint mode is the better tool. The taint section below runs the same cases through it.
The mandated-wrapper rule is the easiest custom rule to write. The policy is to call app.http_client.fetch(), never requests.get() directly. Version one is pattern: requests.get(...). It found the two direct calls in the integration code, including the aliased one, and it also flagged the wrapper itself, whose implementation has to call requests.get.
There are two common ways to exempt the wrapper, and in the lab both had a flaw:
pattern-not-inside on def fetch(...): with a ... body. This removed the wrapper's own call. It also removed a requests.get inside an unrelated function that happened to be called fetch, in a different module. The exemption is keyed on a function name, and a name is not an identity. Constraining it further, for example by adding the module path, narrows it.paths: exclude entry. The documentation describes these as glob patterns "relative to the project root". Our lab directory was not a git repository, and there the result depended on how Semgrep was invoked. With app/http_client.py as the exclude glob, semgrep scan . excluded the wrapper and semgrep scan app did not. The bare filename http_client.py excluded it in both cases, and would also exclude any other file with that name anywhere in the tree.Both are worth a test case: a file that should be exempt and a look-alike that should not.
The three metavariable operators turn a structural match into a precise one.
metavariable-regex is anchored on the left only. The documentation says so: "Regex matching is left anchored." A rule for calls to eval or exec, written as pattern: $F(...) with metavariable-regex on $F and the regex (eval|exec), matched four calls in the lab. Two were eval and exec. The others were evaluate(model, data) and executor_for("pool"), because the regex matched a prefix. Anchoring both ends, ^(eval|exec)$, left the two real calls. This is the substring bug again, inside a structural rule. Any regex applied to a name needs an end anchor unless a prefix match is what you mean.
metavariable-comparison evaluates values, including constants. The token rule is pattern: issue_token(..., ttl=$T, ...) with metavariable-comparison $T > 3600. It flagged ttl=86400 * 30, so the arithmetic was evaluated. It also flagged ttl=LONG_TTL, where LONG_TTL is a module-level constant defined at the top of the file, so constant propagation supplied its value. It correctly ignored ttl=900 and ttl=DEFAULT_TTL, where that constant is 900. It also ignored ttl=ttl, a function parameter whose value is unknown, and it said nothing about that call. A comparison that cannot be evaluated does not match, so silence from this rule means "not provably over an hour", not "under an hour". If unknown values matter, write a second rule, at a lower severity, for the non-literal case.
metavariable-pattern matches a further pattern against what a metavariable bound. A rule for subprocess.run($CMD, ..., shell=True, ...) with metavariable-pattern requiring $CMD to be an f-string or to contain a concatenation matched the two commands built at runtime. It ignored a constant command string and a list-form call without shell=True.
focus-metavariable narrows what is reported. With focus-metavariable: $T the token finding pointed at the ttl argument, column 34, rather than the start of the call. With focus-metavariable: $CMD the shell finding pointed at the command string. This matters beyond display. The reported location is what a reviewer reads first, it is what an inline suppression sits next to, and it is one of the inputs a tool uses to decide whether two findings on different scans are the same one. The finding identity post covers how fingerprints are built from it.
A taint rule, mode: taint, describes data flow rather than shape. The documentation's operators are:
pattern-sources: expressions whose values are untrusted, such as flask.request.args.get(...)pattern-sinks: places where untrusted data must not arrive, such as the query argument of $CUR.execute($Q, ...), focused on $Qpattern-sanitizers: expressions that make data safe, such as int(...)pattern-propagators: expressions that move taint from one value to another that the engine would not otherwise connect, such as $TO.append($FROM)The engine reports a sink when tainted data reaches it without passing through a sanitiser. By default a sink must match exactly, while any subexpression of a source or sanitiser match counts. The documentation lists these as exact defaults: true for sinks, false for sources and sanitisers.
Focusing the sink on the query argument is not decoration. Without it, the whole call is the sink, and a parameterised query whose second argument carries the tainted value is reported, which is the precise case a SQL rule must not flag. With focus-metavariable: $Q, the bound parameter in cur.execute("... = ?", (term,)) was correctly ignored.
The table compares the naive search rule, the refined search rule, a first taint rule (sources, two sinks, int() as the only sanitiser) and a final taint rule that adds the guard and propagator described below. "Real" means request data reaches the query in a way that can change its meaning. All cases are in one handler module except case 7, where the handler passes request data to a helper in another module that builds and executes the query.
| Case | Real? | Naive search | Refined search | First taint rule | Final taint rule |
|---|---|---|---|---|---|
1. f-string of request data in execute | Yes | Hit | Hit | Hit | Hit |
| 2. Query built in one statement, executed in the next | Yes | Hit | Miss | Hit | Hit |
| 3. Bound parameter | No | Hit | – | – | – |
| 4. f-string of a constant | No | Hit | Hit | – | – |
| 5. HTML-escaped value in an unquoted integer position | Yes | Hit | Hit | Hit | Hit |
6. Value converted with int() | No | Hit | Hit | – | – |
| 7. Helper in another module executes its parameter | Yes | Hit | Miss | Miss | Miss |
| 8. Helper returns a string built from its argument | Yes | Hit | Miss | Hit | Hit |
9. gettext call with request data | No | – | – | – | – |
10. sqlalchemy.text() built from request data | Yes | Miss | Miss | Hit | Hit |
11. Inline isdigit() guard that returns early | No | Hit | Hit | Hit | Hit |
| 12. Validator function called for its side effect | No | Hit | Hit | Hit | – |
13. Query assembled with list.append and join | Yes | Hit | Miss | Miss | Hit |
execute on job-runner objects (3 calls) | No | 3 hits | – | – | – |
| Reported / of which real | 7 real | 14 / 6 | 6 / 2 | 7 / 5 | 7 / 6 |
Case 10 is a second sink rather than a different shape. The naive and refined rules only looked at execute, and the taint rules listed both sinks. The rest of this section explains the cells that are not obvious, because each is a mechanism you will meet again.
The documentation is explicit about the scope of the open-source engine. Its comparison page lists "Single function taint analysis" for Community Edition and says it "can't track data beyond a single function or file". The taint documentation says "Interprocedural taint analysis is a Semgrep Pro feature". The advanced taint page says that in Community Edition, "where taint analysis is intraprocedural, all function calls are opaque", and that "by default, Semgrep assumes that an opaque function call propagates any taint passed through any of its arguments to its output."
That explains cases 7 and 8. In case 8 the handler calls build_filter(name) and passes the result to execute. The engine does not look inside build_filter. It assumes that tainted data in means tainted data out, so it reports the sink, which happens to be right. In case 7 the tainted value goes into run_report(conn, name) and is executed inside that function. Nothing comes back out to a sink in the handler, and inside run_report the parameter is not a source, so nothing is reported.
To separate "different file" from "different function", we put three helpers in the same file as the handler:
The --dataflow-traces text output shows the difference if you know to look. For case 8 the trace runs from the source line to the sink call and has no step inside the helper. A trace that ends at a call into your own code means the finding rests on the assumption, not on a path the engine actually followed.
There is an option to flip the assumption. taint_assume_safe_functions: true makes opaque calls clean their arguments. In the lab it removed case 8, which is real. It also removed case 5, which is real, because html.escape is an opaque function call too. The option traded two true positives for the absence of C-style false positives. Whether that is a good trade depends on how your code is structured, which is exactly what a measurement on your own code will tell you.
The commercial engine's interprocedural modes, --pro-intrafile and --pro, are documented as following taint across functions and across files. We did not test them: the open-source CLI reported that it needed the proprietary engine and a login, and exited.
There are two ways to catch a helper that executes its parameter using only single-function analysis, and the lab tried both:
$CUR.execute(<... "..." + $X ...>, ...), flagged the helper's concatenation regardless of where name came from. It cannot tell a parameter that carries request data from one that never does.pattern-inside: def $F(..., $P, ...): with a ... body, plus focus-metavariable: $P, made every parameter of every function a source. It found the cross-module helper and the same-file helper. It also found the two job-runner execute calls, because that rule's sink was any method named execute. Making every parameter a source moves the precision burden onto the sink, so the sink needs the receiver constraint from the search section.Both are reasonable as separate, lower-severity rules that say "check the callers" in their message. Neither should share a severity or a gate with a rule that has seen the source.
Case 5 is the one to remember. The handler HTML-escapes an id from the query string and concatenates it into WHERE id = with no quotes. Escaping for HTML replaces <, >, & and quote characters. A payload such as 1 OR 1=1 contains none of those, and in an unquoted integer position it needs none, so it passes through html.escape unchanged and changes the meaning of the query.
The first taint rule reported it. We then added html.escape(...) to the rule's sanitisers. That is what happens when someone keeps one shared list of "functions that clean input" and pastes it into every rule. The finding disappeared. Nothing about the code had changed and nothing told anyone a finding had been removed. A wrong sanitiser does not produce noise. It produces silence, and silence is never triaged.
Semgrep's structure helps here if you let it. Sanitisers are declared per rule, and a rule should cover one sink class, so a sanitiser is only ever credited against the sinks in its own rule. The failure comes from sharing. A sanitiser list is a claim of the form "this function makes data safe for this sink". Review each entry against the sink class it sits next to, and never copy the list between a SQL rule, a shell rule and an HTML rule. Semgrep also has taint labels, which track different kinds of taint within one rule. The documentation covers them, and we did not test them here.
Cases 11 and 12 are both safe and look alike. They are handled differently.
In case 11 the handler checks if not item_id.isdigit(): abort(400) and then concatenates item_id into the query. Every taint rule reported it. The data-flow overview lists the reason among its design trade-offs: "No path sensitivity: All potential execution paths are considered, even though some may not be feasible." The engine does not learn anything about item_id from the condition. You cannot fix this with a sanitiser, because there is no expression to match. The options are to restructure the code so the conversion is the guard, for example int(item_id) inside a try block, or to accept and record the finding.
In case 12 the handler calls require_digits(item_id), a function that aborts on bad input, and then uses item_id. Adding require_digits(...) as an ordinary sanitiser did not help. The finding stayed, because, as the documentation explains, only that occurrence of item_id inside the call is sanitised, not the variable afterwards. Declaring the sanitiser with by-side-effect: true, and using focus-metavariable: $X on require_digits($X) as the documentation requires, removed it. After that call the variable itself is treated as clean.
That sanitiser is a claim about require_digits: that it always stops execution on bad input. If someone later edits the function to log a warning and continue, the rule keeps crediting it, and real findings disappear the way case 5's did. Put the validator's contract in a test, or in a second small rule that checks its body still raises or aborts.
Case 13 builds the query by appending the tainted term to a list and joining it. Neither taint rule without a propagator reported it. Adding a propagator, pattern: $TO.append($FROM) with from: $FROM and to: $TO, made the list tainted once the term was appended, and the joined string reached the sink tainted. The documentation notes that propagators only work within a function. In the lab, that one propagator was the difference between missing and catching a real injection. Collections, builders and string buffers are where to look for missing propagators in your own code.
Taint mode is not automatically better. Search mode catches things that need no source: mandated wrappers and banned calls, where "never call requests.get directly" holds wherever the URL came from; configuration shapes such as a long token TTL, verify=False or shell=True with a built string; and sinks whose inputs arrive from callers you cannot model as a source, like case 7. Many custom rule sets end up with both for one bug class: a taint rule that blocks, and a shape rule that informs.
Semgrep has a test runner built around annotated example files. The documentation lists four annotations, each written as a comment on the line before the code it describes:
| Annotation | Meaning |
|---|---|
ruleid: <id> | The rule must match the next line |
ok: <id> | The rule must not match the next line |
todoruleid: <id> | The rule should match here but does not yet: a known false negative |
todook: <id> | The rule matches here but should not: a known false positive |
The test file sits next to the rule with the same base name: http-wrapper-required.yaml and http-wrapper-required.py. semgrep --test on the directory runs every rule against its file and compares.
For the wrapper rule, the test file contained:
ruleid on a direct requests.get call and on the aliased http_get callok on the call inside the wrapper itselfok on a requests.post call, which this policy does not covertodoruleid on a get call made through a session object passed in as a parameter, which the rule does not match yetWith the second version of the rule, the runner printed 2/2: ✓ All tests passed (two rules in that directory) and exited 0. With version one, which flags the wrapper, the same test file failed. The runner printed missed lines: [], incorrect lines: [7], where line 7 is the wrapper's call, and exited 1. When we changed the todoruleid on the session call to a plain ruleid, it failed with missed lines: [22].
One behaviour to know about: when we put a todoruleid on a line that the rule does match, the suite still passed. The documentation says tests "automatically avoid failing on lines marked with # todoruleid or # todook". The runner does not tell you a known gap has closed, so search the test files for todo annotations whenever the rule changes and promote the ones that now hold.
A useful test file for a security rule contains every vulnerable shape, including alias imports and the multi-statement variant; the safe twin of each, written the way your team writes it; the look-alikes, such as a same-named function in an unrelated module; each known limit as a todoruleid or todook; and the code from the incident that prompted the rule, if there was one. The test file is the rule's specification, and reviewing it is faster than reviewing the YAML.
semgrep --validate --config <rule> checks a rule's structure. The documentation says it runs Semgrep's own lint rules, pulled from the p/semgrep-rule-lints registry pack, together with built-in checks. We infer from that, without having tested it offline, that it needs network access; our runs had it. In the lab it rejected, with exit code 2:
message ("One of these properties is missing: 'message'")pattern-not-insde ("unexpected key pattern-not-insde")requests.get(... with no closing parenthesisSEVERE. The error listed the accepted values: ERROR, WARNING, INFO, INVENTORY, EXPERIMENT, CRITICAL, HIGH, MEDIUM and LOW. The documentation maps the older ERROR, WARNING and INFO to HIGH, MEDIUM and LOW.pattern-either with the same alternative twiceThe more important observation is what a broken rule does to an ordinary scan. We put the misspelled rule in a directory next to a valid one and ran semgrep scan --config on the directory. The scan reported "rule validation failed", exited with code 7 and produced no results, not even from the valid rule. That is the right behaviour, because it fails loudly. But a pipeline that only distinguishes zero from non-zero may treat 7 as "findings present". One that runs the scan with || true, or uploads whatever results file exists, would show a green check from a scan that never ran. The scan-passed post is about this class of failure.
Exit codes for a successful scan matter too. With findings present, semgrep scan exited 0 by default and 1 with --error. A gate built on the exit code needs --error, or needs to read the results file.
So the minimum for a repository of custom rules is a CI job that runs --validate and --test on every change and fails on either.
Tests show the rule does what its author intended on examples its author wrote. They say nothing about what it does to the codebase the team actually maintains. That needs a measurement, done before the rule can block a merge.
Run the new rule by itself over every repository it will apply to, on the default branch, with JSON output. Running it alone means its count is not mixed with anything else's. Record:
One detail matters if you track findings over time. In our runs, the rule ID Semgrep reported was prefixed with the directory the rule file was loaded from: rules.lab.http-wrapper-required when loaded from rules/, and nosem2.lab.http-wrapper-required when the same rule was loaded from another directory. Moving a rule file therefore changes the reported ID, and any stored triage history keyed to that ID stops matching. Fix the location before the rule's findings start accumulating history.
Precision is the share of a rule's findings that are real. To estimate it, label findings by hand. If the rule produces a few dozen hits, label all of them. If it produces hundreds, label a random sample, not the first page. The first page is ordered by file path and will over-represent whichever directory sorts first.
Use labels that keep the information you will need later. The false positives post argues for three outcomes, not two:
The mechanism is the valuable part. Ten false positives that all say "job runner execute" are one pattern-inside away from zero.
The sampling arithmetic is ordinary statistics, not a property of any tool. If you label n findings and k are true positives, the estimate is k/n, and a rough 95% interval is plus or minus 1.96 times the square root of p(1−p)/n. As a worked example, 30 true out of 40 labelled is 0.75, plus or minus about 0.13. That is precise enough to decide whether a rule is close to gate quality, and not precise enough to compare two rules within a few points of each other. Label more if the decision depends on the difference.
Measure recall where you can. Precision says nothing about what the rule misses. You have ground truth for recall in fixed bugs. Check out the commit before each relevant fix and confirm the rule fires on it. A rule written after an incident that does not fire on the incident's own pre-fix code is not finished.
In the lab, the four SQL rules came out at 6 of 14, 2 of 6, 5 of 7 and 6 of 7, against 7 real cases. That shows how much the numbers move between versions of one rule on the same code, and why a rule should be measured again after every change, not once.
A rule's cost is roughly its false positives multiplied by the time each takes to dismiss. As illustrative arithmetic only: 400 hits at 25% precision means 300 dismissals before the 100 real findings are worth anything, while 12 hits at 90% precision means one. Detection count would rank the first rule higher, which is why the suppression rate post calls it the wrong number for judging a rule set.
General practice for introducing a rule without the team learning to ignore it:
--baseline-commit does this in the CLI. In a lab git repository with two existing direct requests.get calls, a branch that added a third reported only the new one. Run with the first version of the rule, which also flags the wrapper, the same comparison still reported only the new call, because the wrapper's finding existed at the baseline commit too. The baseline was computed with the same rule at both commits. A gate that stores a baseline and compares a later rule version against it behaves differently, and the stale baseline post covers what that does to blame.Severity and confidence belong in the rule's metadata, alongside the CWE identifier, an OWASP category where one fits, and a reference to the internal guidance. The documentation describes metadata as arbitrary data that does not affect matching, and it appeared in the lab's JSON output for each finding. Treat severity as how bad a true positive is, and confidence as how likely a finding is to be true, and set confidence from the measured precision, not from the author's optimism. A gate can then block on high severity and high confidence while lower-confidence rules only comment.
Semgrep's inline suppression is a comment containing nosemgrep. The documentation says it goes "at the first line or preceding line of the pattern match". In the lab:
# nosemgrep: lab.http-wrapper-required at the end of the line suppressed that rule, and a second rule matching the same call was still reported# nosemgrep on the preceding line suppressed both rules# nosemgrep: some-other-rule did not suppress our ruleBy default, suppressed findings did not appear in the CLI's JSON output at all. Running the same scan with --disable-nosem brought them back. The difference between the two runs, per rule, is that rule's suppression count. That is the cheapest way to measure suppressions without a platform.
That per-rule count is the health signal. A rule that is suppressed far more often than its siblings is either wrong, which means a rule fix, or finding things nobody will fix, which means a policy conversation. Either way, the suppression comments are a list of labelled examples from your own code, often with a reason written next to them. Read them before changing the rule.
Bare nosemgrep comments deserve a rule of their own, because they suppress every rule on the line, including ones added later. We wrote one: a pattern-regex for a nosemgrep comment with no rule ID after it. On a file containing a bare # nosemgrep it reported nothing, because the finding was on a line carrying nosemgrep, so the comment suppressed its own detector. With --disable-nosem it reported the line. Any check for suppression hygiene has to run with suppressions disabled.
AppSecD's static analysis runs across more than twenty languages through a farm of over fifteen analyzers with Semgrep as the primary engine, against OWASP Top 10, CWE Top 25 and custom rule packs. It includes taint analysis in v1 and v2, with interprocedural dataflow that traces source to sink across function boundaries, and policy-driven gating that blocks a pull request or commit when it introduces a violation. Its AI layer labels likely false positives with the label itself tracked, so triage quality is measurable. The lifecycle supports maker-checker approval, so a finding marked false positive can require a second person to confirm it.
This post does not describe how AppSecD loads, tests or measures a custom rule, and the rule-authoring failures above (over-matching, sanitisers credited to the wrong sink, unanchored regexes, over-broad exemptions, untested rules) apply to any engine running a rule someone wrote, ours included. This post also makes no claim that AppSecD's interprocedural analysis applies to custom Semgrep rules, so nothing here should be read as saying it catches the cross-function cases that Semgrep Community Edition misses. The AI triage measurement post covers measuring a false-positive label against ground truth, which applies to a human labelling a new rule's sample as much as to a model.
| Question | If the answer is no or unknown |
|---|---|
| Can the message say what is wrong and what to do instead? | The rule is not ready |
| Could the dangerous API be removed instead, or is it already covered by a maintained rule? | Remove it, or use the existing rule |
| Are sinks matched structurally against fully qualified names, with the receiver constrained where a method name is shared? | Expect substring, comment and wrong-object matches |
| Does every name regex have both anchors? | metavariable-regex will match prefixes |
| Is each exemption keyed to something more specific than a function name or bare filename? | Look-alikes inherit the exemption |
| Is each sanitiser correct for this rule's sink class specifically? | Real findings disappear silently |
Are side-effect validators declared with by-side-effect, and their contract tested? | False positives now, silent misses later |
| Are list and builder operations covered by propagators? | Flows through collections are missed |
| Does the rule need data followed across functions or files, and does your engine do that? | Add a shape rule or parameter sources |
Is there a test file with shapes, safe twins, look-alikes and todo annotations, run with --validate in CI? | Regressions and typos go unnoticed |
| Does the scan job fail on a non-zero exit that is not "findings"? | A broken rule can switch the scan off |
| Has it been run alone over every repository, a random sample labelled with mechanisms, and precision estimated? | You do not know its volume or noise |
| Does it fire on the pre-fix commit of the bug that motivated it? | You do not know its recall |
| Is it rolled out report-only, then comment-only, then blocking on new findings? | The team learns to ignore it |
| Are severity, confidence and CWE in metadata, with confidence from measurement? | The gate cannot tell certain from speculative |
Is its suppression count tracked, with bare nosemgrep checked with suppressions disabled? | Quality decays unseen |
| Is the rule file's location final, so its reported ID is stable? | Triage history is orphaned on a move |
Back to the regex that matched getcontext(. It was syntactically valid, it would have passed --validate, and with a test file written by the person who wrote the regex it would probably have passed --test too. Only running it over code its author did not write showed what it really matched. That is the general lesson for custom rules. A rule's author sees the cases they were thinking of, and the codebase contains the rest. Measure on the codebase before anyone else has to read the findings.