The DAST scan was clean on the gift-card endpoint. Can the card be redeemed twice at once? Race conditions and limit overrun as a DAST finding: check-then-act and TOCTOU, why one request at a time cannot see them, last-byte synchronisation, what evidence confirms one, and the fixes that hold across workers

DedSec Security Research ·

Here is a hypothetical. An application sells gift cards, and a card can be redeemed once: POST /api/giftcards/redeem moves the card's value into the user's wallet. Four results about that endpoint reach the security team in the same month.

The first is a dynamic scan that reported nothing on the endpoint, and the endpoint is recorded as scanned and clean. The second is from a tester who sent twenty redemptions of one card in parallel, got twenty 200 OK responses, and filed a critical: "race condition, card redeemed 20 times". The third is from another tester who ran the same test against staging, got one success and nineteen "already redeemed" responses, and closed the question as "not vulnerable to race conditions". The fourth is the developer's fix for the critical. The handler is wrapped in a lock, the retest on a laptop passes, and the ticket is closed as fixed.

As in the earlier posts in this series, every one of those decisions rests on something other than what the evidence can prove. In our lab, a scanner-style sequence of one request at a time never redeemed a vulnerable card more than once, in any configuration we ran it in. Twenty 200 responses came back from an endpoint that credited the card exactly once, because nineteen of them were replays from an idempotency layer, and the same endpoint still credited the card up to fourteen times when each request carried its own key. A vulnerable handler running behind a server that processes one request at a time redeemed the card once in all 120 trials, delay or no delay. A per-process lock that worked perfectly on one process let up to four redemptions through when the same code ran as four processes.

This post is for whoever is holding a race-condition finding, or a clean result on an endpoint that enforces a limit. It covers:

It belongs to our series on confirming DAST findings. The series guide sets out the evidence classes the series uses; the guide predates this post and does not list it. The evidence here is closest to the guide's last class, an effect seen by a second party, where the second party is a follow-up read of the ledger. The nearest neighbours are the GraphQL post, which measured a rate limiter that counted HTTP requests while each request carried a hundred login attempts, and the object-level authorisation post, which is about another business-logic bug that no single response reveals. The file upload post listed races between writing and validating a file as something it did not test, and this post does not test that either.

How we tested. We ran everything on 16 November 2026 on a virtual machine with two vCPUs of an AMD EPYC 9354P, running Ubuntu 24.04.4 with kernel 6.8.0. The lab application is a hand-written HTTP/1.1 server on Python 3.12.3, using only the standard library's http.server and sqlite3 modules, with SQLite 3.45.1 in WAL mode. Each request opens its own database connection with a 10-second busy timeout, which is roughly how an application with a connection pool behaves. The server listened only on 127.0.0.1, and every request travelled over loopback.

The scenario is one gift card worth 100 credits. A successful redemption writes one row to a ledger of credits, and the business rule is that the ledger may gain exactly one row per card. Every handler variant implements the same operation differently, and each has an optional delay between deciding and writing, passed as a query parameter, so we could widen the race window on purpose. We ran three deployments of the same code: a thread-per-connection server in one process, the same server as four processes sharing the port through SO_REUSEPORT, and a server that handles one request at a time.

The client sent twenty identical requests per trial, three ways, described below, and we ran twenty trials of each configuration, resetting the card before each one: 1,660 trials and 33,200 requests in all. After every trial the harness read the card's balance and the user's wallet through the API, and counted ledger rows directly in the database.

These are lab results from one machine, one language runtime and one database engine. They show mechanisms. They are not rates you should expect elsewhere: a different machine, a real network, a different database or a different server would produce different numbers, and some of the differences are the subject of the section on negative results. We did not test HTTP/2, any web framework, any production database server, any DAST tool or any proxy tool's race feature. Statements about PortSwigger's research, the OWASP Web Security Testing Guide and the CWE entries describe their published content in our own words.

What a race finding is, in mechanism

A race condition is a bug in which the outcome depends on the order or timing of operations that run concurrently. MITRE's CWE-362 names the general class, "Concurrent Execution using Shared Resource with Improper Synchronization ('Race Condition')". In a web application, the concurrent operations are usually requests, and the shared resource is usually a database row.

The common shape is check-then-act. The handler reads some state, decides whether the action is allowed, and then writes. Between the read and the write there is a gap. If a second request reads the same state inside that gap, it reaches the same decision, and both requests act. CWE-367, "Time-of-check Time-of-use (TOCTOU) Race Condition", is the specific entry for a check that is no longer true by the time its result is used.

Our vulnerable handler is about as plain as the shape gets:

  1. SELECT balance FROM cards WHERE code = ?
  2. if the balance is under 100, return 409 "already redeemed"
  3. UPDATE cards SET balance = ? WHERE code = ?, with the value computed in step 1 minus 100
  4. insert a ledger row crediting the wallet, and commit

Nothing in that code is wrong when requests arrive one at a time. The bug is a property of two executions interleaving, which is why it does not show up in a code review that reads the handler top to bottom.

Two variants of the same mechanism are worth naming, because they produce different evidence:

Beyond a single endpoint, the same research describes races across endpoints and across the steps of a workflow: a cart changed between the moment its total is checked and the moment the order is confirmed, or a request that arrives while an object is half-built and briefly sits in a state no single request should ever see. These are harder to find and harder to confirm, because the requests that race are different requests, and one of them may need to land in a specific few milliseconds. Our lab tested only the single-endpoint limit overrun, which is the simplest case and the one most likely to be tested at all.

Why a scanner that sends one request at a time cannot see it

The control in our lab was the scanner's usual behaviour: twenty redemptions sent one after another, each waiting for the previous response. In all 420 sequential trials, on both servers we ran it against and at every delay, that sequence credited the card exactly once, and the other nineteen requests got "already redeemed". Each request sees the state the previous one left behind, so the check is always right.

That is a structural limit, not a tuning problem. A typical dynamic check takes one request, varies a parameter, sends it and reads one response, then decides. Injection, reflection, misconfiguration and most of what DAST does fit that model, because the vulnerability is visible in the relationship between one request and its response. A race is visible only in the relationship between several requests and the state they leave behind together. A check designed around one exchange has nothing to look at.

Several other things stand between a scanner and this class of bug, and none of them is about cleverness:

So when a DAST report is clean on an endpoint that enforces a limit, the useful reading is "this was not tested for races" unless the tool's documentation says otherwise. The scan-passed post makes the same argument about scans that did not run. Here the scan ran, and the test that would have found the bug was not part of it.

Getting requests to arrive together

To observe a race you need two or more requests inside the window between check and act. How wide that window is depends on the handler, and how close together your requests arrive depends on how you send them. The lab compared two ways of sending, both over HTTP/1.1, with one connection per request:

That research also describes the technique for HTTP/2, the single-packet attack. HTTP/2 multiplexes many requests over one connection, so a client can hold back the final frame of each request and then send all of the final frames in one TCP packet. The requests then arrive in the same packet and are not separated by network jitter. We did not test it: Python's standard library has no HTTP/2 client. The point to take from it is that whether the target speaks HTTP/2 changes how tightly requests can be synchronised, and therefore what a test can prove.

Here is the vulnerable handler on the single-process threaded server, with twenty requests per trial and twenty trials per row:

Delay between check and writeHow sentTrials that overranCredits per trial, median (range)Total credits from 400 requests
nonesequential0 of 201 (1 to 1)20
noneparallel connections18 of 2010 (1 to 20)216
nonelast-byte sync20 of 2011 (2 to 20)251
1 mssequential0 of 201 (1 to 1)20
1 msparallel connections20 of 2015 (4 to 20)281
1 mslast-byte sync20 of 2018 (3 to 20)302
10 mssequential0 of 201 (1 to 1)20
10 msparallel connections20 of 2020 (5 to 20)347
10 mslast-byte sync20 of 2020 (12 to 20)391

Three things are visible in the table. First, a wider window means more overrun, and with no artificial delay at all, the handler's own gap (opening a connection, a SELECT, a Python decision) was wide enough on this machine to overrun the limit in most trials. Second, last-byte synchronisation produced more credits in total than parallel connections at every delay, and it never produced a trial with a single credit. Third, both methods produced trials with very different counts: the same configuration ranged from one credit to twenty.

We also recorded, on the server's clock, the time between the first and the last request in a trial finishing being read. On the single-process server the last-byte runs had the tighter median spread at every delay: 16 ms against 52 ms with no delay, 8 against 28 at 1 ms, and 9 against 18 at 10 ms. On the four-process server that did not hold with no delay, where parallel connections had the tighter median. Treat those spreads as a property of this runtime on two vCPUs, not as typical values. They include the time Python took to schedule the server's threads, which on a quieter runtime would be much smaller.

What evidence confirms a race

A race is confirmed by one thing: the effect happened more times than the rule allows, counted somewhere that records each effect. Everything else is a proxy, and in the lab both obvious proxies failed.

The read of the limited resource looked correct

After every one of the 1,660 trials, the card's balance, read through the API, was 0. That includes the trials where twenty redemptions succeeded and the wallet received twenty credits, 2,000 in value. Because step 3 writes a computed value, every racing request wrote "0", and the last writer's 0 looks exactly like the correct answer.

So a confirmation that checks "the card is now empty" proves nothing. The right follow-up read is of the side that accumulates: the wallet, the ledger, the list of orders, the count of redemptions. In our lab, the wallet balance read through the API matched the ledger count in every trial. Before a test, decide which read counts each effect separately, take it before and after, and make sure nothing else can change it during the test.

The status codes can be honest or not

On the vulnerable handler, the number of 200 responses matched the number of credits in every trial. That is because the handler returns 200 only when it writes. It is a property of this handler, not something a tester can assume.

The idempotency-key variant shows the other case. It wraps the same vulnerable handler in a layer that claims the request's Idempotency-Key in the database before doing anything, and replays the stored response if the key has been seen. With all twenty requests carrying the same key, every response was 200 OK in all 120 trials across three sending methods and two delays, which is 2,400 responses, and the ledger gained exactly one credit per trial. Nineteen of each twenty were replays of the first response. Count responses and you report a twenty-fold overrun that never happened. That is the second hypothetical tester's critical, and the tester was looking at the right endpoint with the wrong evidence.

Other common designs break the link between status code and effect too. An API may return 200 with an error in the body, accept a request with 202 and process it later, or return a success and then roll the transaction back after a downstream failure. The count of effects is the evidence. Responses tell you which requests to look at.

Timing is not evidence

Responses that arrive at the same time, or a handler that slows down under parallel load, show that requests were concurrent. They do not show that the check was bypassed. In the lab, the variants that held the limit also received tight bursts of requests, and they told us nothing about the bug.

What to record

A confirmed finding should carry:

The last item matters because a race is probabilistic. A finding that says "overran in 18 of 20 trials" tells the reader both that the bug is real and how reliably it reproduces.

Why a negative result is weak

A race test that does not overrun the limit shows that these requests, sent this way, to this deployment, at this moment, did not land inside the window. It does not show that the window is absent. The lab produced every way a negative result misleads:

So record a negative result as "not reproduced", with the method, the protocol, the deployment and the number of attempts, and never as "not vulnerable". If the code has the check-then-act shape, the code is the stronger evidence, and the fix is cheap.

The side-effect problem

Every confirmed race in this lab changed data: credits that should not exist, written to a wallet. That is the nature of the finding. Unlike a reflected marker or an out-of-band callback, there is no way to confirm limit overrun without overrunning the limit.

General practice follows from that:

The same reasoning applies to automated testing. A scanner configured to send bursts of state-changing requests is making changes, and it needs the same scoping and consent as a person doing it.

Fixes, and what they cost

The general pattern for a fix is to stop making the decision in application memory between two database round trips, and let the database make it as part of the write. We tested five fixes and two designs that look like fixes. Every row is twenty trials of twenty requests per configuration. "Held" means exactly one credit in every trial.

VariantOne process, threadedFour processes
Vulnerable check-then-actOverran in 58 of 60 parallel trials and 60 of 60 last-byte trialsOverran in 80 of 80 trials
Atomic conditional updateHeldHeld
UNIQUE constraint on the redemptionHeldHeld
Write lock taken before the readHeldHeld
Snapshot transaction, no retryHeld, with 114 to 345 errors per 400 requests under parallel loadnot run
Snapshot transaction, with retryHeld, no errorsnot run
Idempotency key, every request a different keyOverran in 72 of 80 parallel and last-byte trialsnot run
In-process mutexHeldOverran in 79 of 80 trials, at most 4 credits

Atomic conditional update

UPDATE cards SET balance = balance - 100 WHERE code = ? AND balance >= 100, then check how many rows changed. If the answer is zero, the card was already spent, and the handler returns 409. The check and the write are one statement, so there is no gap between them, and the row count is the verdict. It held in every configuration, including with a 10-millisecond delay placed before the statement.

This is the cheapest fix and the right default for counters, balances and quotas. It needs the rule to be expressible as a condition on the row being written. It also needs the handler to read the row count, which is easy to forget: an UPDATE that matches no rows is not an error.

Unique constraint

Our variant kept the check-then-act shape and the delay, but inserted a row into a redemptions table with a UNIQUE constraint on the card code, in the same transaction as the credit. The second insert fails, the transaction rolls back, and the handler returns 409. It held everywhere.

The constraint expresses "at most one", per card or per card and user, and nothing else. It cannot express "balance must not go negative" for a card that can be partly redeemed, or "at most five per day". Its advantage is that it holds whatever the application code does, including code paths written later by someone who never saw the race.

Locks and transactions

Our lock variant began with BEGIN IMMEDIATE, which in SQLite takes the database's write lock before the read. That is a much coarser lock than other databases use. SQLite has no row locks, so this serialises every writer to the whole database. The equivalent elsewhere is a row lock, such as SELECT ... FOR UPDATE in PostgreSQL or MySQL. The cost is contention on the locked rows, and in larger transactions, the possibility of deadlocks that must be handled.

The two snapshot-transaction variants are the instructive ones. Both read and wrote inside one deferred transaction. With no retry, the limit held, but in SQLite's WAL mode many of the transactions that had read the balance and then tried to write failed at once with "database is locked", rather than waiting. Under parallel load, between 114 and 345 of every 400 requests ended in a 500 error, more with the wider window. The race had become an error instead of an overrun, and a user who happened to be second would have seen a server error. With the whole transaction retried from the start, the retry re-read the balance, found the card spent, and returned 409. That held with no errors.

This generalises, although our numbers do not. A stricter isolation level, such as serialisable, turns some races into errors that the application is expected to catch and retry. Without the retry, users see failures. Not every isolation level does this. The default level in many production databases does not detect this pattern, and a read-then-write handler there may simply lose the update. Check what your database does at the isolation level you actually run, rather than assuming the transaction keyword is enough.

Idempotency keys

An idempotency key deduplicates a request that is sent more than once, which is how a client safely retries after a timeout. With every request carrying the same key, our variant credited the card once in every trial. With every request carrying a different key, which is what an attacker sends, the vulnerable handler underneath raced as before. It overran in 14 of 20 trials with parallel connections and 18 of 20 with last-byte sync with no delay, and in all 40 trials with the 10-millisecond window, with up to 14 credits in a trial.

An idempotency key solves a client's retry problem. It does not enforce a business limit, because the client chooses the key. The key layer also needs the same care as any other limit: it claims the key with an insert into a primary key before doing the work, so two requests with the same key cannot both pass the "seen this key?" check. A key layer built as check-then-act has its own race.

Application-level locks

The in-process mutex wrapped the vulnerable handler in a lock held in the process's memory. On one process it held in every trial. On four processes sharing the port, it overran in 79 of 80 trials. The credits in a trial never exceeded the number of processes that served the trial: four credits in 66 trials, three in 8, two in 5 and one in 1. Each process's lock serialised its own requests and knew nothing about the other three, so each process could redeem the card once.

That is the fourth hypothetical: a fix that passes a retest on a single-process development server and does not hold in a deployment with several workers, several containers or several hosts. A lock that coordinates more than one process has to live outside the processes, in the database or a shared lock service. A shared lock service brings its own questions about what happens when the lock holder dies or the lock expires mid-operation. Where the database can enforce the rule, that is usually simpler.

Writing a regression test

A race fix deserves a test, and a race test is easy to write badly. Ours runs in-process, without HTTP: twenty threads wait at a barrier, call the handler with the same card, and then the test counts ledger rows and asserts there is exactly one. We ran four variants for 50 rounds: the vulnerable handler and three fixes. The vulnerable handler failed all 50 rounds both with and without a 5-millisecond delay. With no delay it produced between 3 and 20 credits per round, with a median of 16.5. The atomic, unique-constraint and write-lock variants passed all 50 rounds, with and without a 5-millisecond delay.

What made it a useful test:

Even a well-built test only shows that the fix held for the cases it ran. Keep it in the suite, so a later refactor that reintroduces a read-then-write is caught.

The static side

Static analysis has a hard time with races in web handlers, for the same reason review does. The vulnerable handler is a sequence of individually correct statements, and the bug is a property of two executions interleaving. A data-flow rule has no source or sink to follow.

What a static rule can do is flag the shape: a read and a later write of the same table in one function, with no lock, conditional WHERE or constraint in between. That rule will be noisy, because most read-then-write code is not guarded by a business limit, and it is a candidate for an informational rule with a "check whether a limit applies here" message rather than a blocking one. The custom rule post covers how to measure a rule like that before it reaches anyone's pull request.

The more productive place to start is design. Every business limit in a system is a candidate race: one redemption per code, one vote per user, a balance that cannot go negative, a seat that can be booked once, a free-tier quota, an invitation that can be accepted once. A threat model that lists those limits, and asks for each one where it is enforced, produces a short list of endpoints worth a race test and a code review. It is the same kind of question as "who is allowed to read this object?", which the object-level authorisation post argues a scanner cannot answer on its own.

A triage checklist for a race-condition finding

QuestionIf the answer is no or unknown
Is the business limit written down as a number?You cannot say whether it was exceeded
Was the effect counted on the accumulating side (ledger, wallet, orders), before and after?A read of the limited resource can look correct after an overrun
Did the count exceed the limit?Status codes alone do not confirm a race
Do the response codes match the counted effects?Replays, async processing or rollbacks are breaking the link
Was the test repeated, and is the overrun rate recorded?One trial says little either way
Were the requests synchronised (last-byte on HTTP/1.1, single-packet on HTTP/2)?A negative result may reflect the sending method
Does the tested deployment run as many workers and processes as production?A single-worker environment can hide the race entirely
Was anything in front of the application (rate limiter, firewall, queue) spreading requests?A negative result may reflect the infrastructure
Was the test run outside production, on test resources, with the owner's consent and a clean-up list?The test itself is an incident
Does the fix make the database decide (conditional update, constraint, row lock, retried transaction)?An application-level check may still race
Is the fix an idempotency key or an in-process lock?Test with distinct keys, and with several processes
Is there a regression test that counts effects, widens the window, runs many rounds and uses the production database engine?The race can come back unnoticed
Is a clean DAST result on this endpoint being read as "tested for races"?Probably not, unless the tool documents a race test

What this post did not measure

We did not test HTTP/2 or the single-packet technique, any real network path, or any proxy or testing tool's race feature. We did not test PostgreSQL, MySQL or any database other than SQLite 3.45.1, and SQLite's locking is coarser than theirs, so our lock and transaction results describe SQLite. We did not test any web framework, application server, ORM or load balancer, a multi-endpoint or multi-step race, a partial-construction race, a race in a cache or in an external payment provider, or any DAST product's behaviour. The sending-method comparison is from one client runtime on two vCPUs. The numbers measure our lab. None of them measures AppSecD or any other product.

The hypothetical findings, resolved

Where AppSecD fits

AppSecD's dynamic and API testing runs Nuclei, Dalfox, SQLMap, Commix, Katana, Arjun, FFuf and Interactsh for out-of-band detection, plus 150+ native checks and API discovery across 50+ web frameworks. Its policy-driven gating blocks a pull request or commit when it introduces a violation. Findings are deduplicated across scans and move through new, reopened and fixed states, with maker-checker approval so a finding cannot be closed unilaterally.

That is the whole claim. data/products.ts is the single source of truth for what these products do, and it does not mention race conditions, concurrency testing, limit overrun or business-logic testing. So this post makes no claim that AppSecD detects race conditions. Nor does it claim anything about:

The mechanisms, measurements, confirmation procedure, fixes and checklist above are general practice that applies to any scanner's output and any team's code, ours included. The scenario in this post is hypothetical and does not describe any deployment, user or finding.

More about AppSecD · All posts