On a payments platform handling billions in volume, the fraud team had built something genuinely good. Gradient boosting, hundreds of features, lift numbers that made the data scientists proud. It caught fraud the old rules missed, but it sat outside the authorization path, scoring transactions a beat too late, and by the time it flagged anything the money had already moved. We did the obvious thing and put it inline, in front of the authorization decision. It fell over, because nobody had counted the milliseconds.
A fraud model that takes 400ms to score isn’t a fraud model, it’s a customer-service problem.
Everyone building inline fraud detection eventually runs into the same constraint, and hardly anyone plans for it. The authorization call has a hard deadline. The card network expects an answer in a couple of seconds, the payment processor wants its own margin inside that, and the merchant on the other end is watching a spinner. By the time the request reaches your fraud check, what you actually own is a slice of those two seconds. On a good day the slice was around 200ms, and fraud scoring had to fit inside it without starving everything else that also needed time in that window.
Where the 200ms actually goes
The mistake is to think the budget is “model inference time.” Inference is usually the cheapest part. The budget gets eaten by everything around the model.
Model inference is the small box, and feature fetch is the one that blows the budget.
Walk it through. The authorization request lands. Before the model can do anything, you have to assemble the features it was trained on, and those features do not live in the request. The card’s recent velocity, how many times this device has been seen this hour, the merchant’s chargeback rate, the distance between this transaction and the last one: all of that is state, and state lives in a store you have to go ask. Every feature you fetch is a network hop or a cache lookup, and you pay for the slowest one, not the average. Only after that does the model run, and then you serialize a decision and hand it back. Wrapping all of it is the network time getting in and out of your own service, which you do not control and cannot wish away.
When we instrumented it, the split was not close. Feature fetching was the overwhelming majority of the spend. The model itself ran in single-digit milliseconds. We had been optimizing the wrong thing for weeks, shaving the model when the model was never the problem.
The features that survive the budget
Once you accept that feature fetch is the budget, the whole design flips. The question stops being “what features make the model better” and becomes “what features can I get my hands on in time.” Those are different questions, and the gap between them is where the engineering lives.
We sorted every feature into three buckets by how expensive it was to retrieve at decision time.
Features already in the request were free: amount, currency, merchant category, the card BIN, the rough shape of the transaction. No fetch, no hop, and the model got all of it without spending a millisecond.
Features in a hot in-memory store were cheap. We precomputed the expensive aggregations offline and continuously, then kept the latest values in a low-latency cache keyed by card, device, and merchant. Rather than compute the card’s seven-day velocity at decision time, the model read a number a streaming job had already computed and parked there.
Features that needed a real query were the ones we cut. Anything that meant joining across tables, hitting a warehouse, or calling another service synchronously. Some of those were genuinely predictive. We dropped them anyway, or we moved their computation offline so that what remained at decision time was a single fast read. (The graph feature the data scientists loved, the one that traced money between linked accounts, was exactly this kind. It was good, and it took 90ms to compute live, so it lost.)
A model with twenty features it can actually fetch in time beats a model with two hundred features it cannot. The second model is better in the notebook and useless in production, because a fraud score that arrives after the deadline does not arrive at all. The network timed out, the authorization defaulted to its fallback, and your beautiful model contributed nothing except latency. Offline accuracy is a vanity metric if the model cannot answer the phone.
Make the timing impossible to ignore
The way you keep this honest is to measure every feature fetch the same way you measure the model, and to enforce a hard ceiling rather than hope. We wrapped feature retrieval so that anything blowing its slice of the budget got dropped and the model scored without it, instead of letting one slow lookup take the whole transaction down with it.
# Every feature carries its own time budget. A feature that can't answer
# in time is not a feature, it's a liability. We score without it.
async def gather_features(txn, budget_ms=120):
deadline = monotonic_ms() + budget_ms
feats = {}
# free features: already in the request, no fetch, no excuse
feats.update(extract_inline(txn))
# cached aggregates: precomputed offline, we only read here
for name, key in HOT_FEATURES:
remaining = deadline - monotonic_ms()
if remaining <= 0:
metrics.incr("feature.dropped.budget", tag=name)
continue
try:
feats[name] = await cache.get(key(txn), timeout_ms=remaining)
except TimeoutError:
# a missing feature is fine. a stalled authorization is not.
metrics.incr("feature.dropped.timeout", tag=name)
return featsThe model has to be trained to expect this. If a feature can go missing under load, the model cannot treat it as always present, so we trained with the same features randomly dropped, which is also just good regularization. A model that falls apart the moment one cache read times out is not production-ready, however well it scored in the offline test.
def score(txn):
feats = run(gather_features(txn)) # bounded, may be partial
risk = model.predict_proba(feats) # single-digit ms, the easy part
# the threshold is a business call, not a model call: declining a real
# customer costs more than letting one bad txn through, up to a point.
if risk >= DECLINE_AT:
return Decision.DECLINE
if risk >= REVIEW_AT:
return Decision.STEP_UP # 3DS challenge, push the cost to the rare case
return Decision.APPROVEThat step-up path mattered more than any model improvement. Most transactions are fine and you approve them without friction. A thin slice are clearly fraud and you decline. The interesting band in the middle is where a better model stops helping and a second, slower check takes over, one you can afford precisely because you only run it on the few percent the fast model was unsure about. The expensive graph feature we cut from the inline path lives here, in the step-up flow, where it has time to run because we are no longer holding an authorization hostage.
The architecture that won wasn’t the most accurate model but a fast, slightly dumber one that fit the budget, backed by a slower, smarter check that fired only when the fast one hesitated. The fraud team hated giving up features at first, and they came around when the decline rate on good customers dropped, because the inline model finally answered before the network gave up on it.
The lesson generalizes past fraud. Any model you put on the critical path of a real-time decision is competing for the same milliseconds as the thing it is trying to protect, so it never gets to spend the whole budget on being right. The model that wins is the one that does the most with the slice it’s handed.
On any hot path I own, I measure where the budget actually goes before I trust the model. My money is on the feature fetch.