The inspector was standing in a pump room with no signal, holding a cracked gauge in one hand and a phone in the other, talking to it. That image is the whole design brief. Not the demo where someone in an office types a clean question into a clean box. The real user is outdoors, gloved, in a hurry, and the network left an hour ago.
The pitch was simple enough to draw on a whiteboard. An inspector walks a site, says what they see, snaps a photo of the thing, fills in whatever the form demands, and the system turns that mess into a structured action: a work order, a flag, a sign-off. No laptop, no back-office data entry, no transcribing a paper sheet two days later. Everyone in the room nodded. Then we went to build it and the connectivity assumption fell over before anything else did.
Here is the part nobody tells you about field AI. The model quality is not your risk. The risk is that the field is hostile to every assumption a cloud-first architecture quietly makes: a stable connection, low latency, a clean single input, and a user with both hands and full attention. You lose all four at once, and you lose them in the exact moment the work happens.
Three inputs, one intent
Voice, image, and form are not three features. They are three angles on a single intent, and the job is to fuse them, not collect them.
Voice carries the narrative. “Number two pump is leaking from the seal, sounds like cavitation, I am shutting it down.” That sentence has a target asset, a symptom, a diagnosis, and an action in it, all spoken the way a person actually talks, with the hedge and the run-on intact. The image carries the evidence the words can’t: which pump, how bad the corrosion is, what the gauge actually reads. The form carries the structure the regulator wants: asset ID, severity code, the checkbox that says someone verified isolation.
None of the three is complete alone. The voice note is rich and ambiguous. The photo is unambiguous and silent. The form is precise and empty until something fills it. Fusion is where intent gets captured, and fusion is the part that earns its keep. We run the voice through transcription and pull entities and an intended action out of it. We run the image through a vision pass that reads the gauge, spots the visible fault, and confirms the asset against a tag. Then we reconcile the two against the form schema: the voice said pump two, the photo’s asset tag says pump two, the severity the inspector spoke matches the severity the gauge supports. Agreement raises confidence. Disagreement is a signal, not a failure, and it routes to a human.
Offline-first, or it does not ship
We tried cloud-first. For about a week. The failure mode was not subtle: an inspector walks into a basement, finishes the most important inspection of the day, hits save, and the work evaporates because the request timed out against a tower three floors up. You cannot ask a field worker to babysit a progress spinner. They will stop using it, and they will be right to.
So the device holds the truth. Every capture writes to a local store first, gets a client-generated ID, and is considered done the moment it lands on the phone. Sync is a background concern that happens whenever the network comes back, and the user never waits on it. This is the part of the system I would defend hardest. The phone is the source of truth until the moment a record reconciles with the server, and the sync layer is built to assume the network is the exception, not the rule.
That choice cascades into everything. It means the first pass of inference has to run on the device, because the inspector needs to know right now whether their capture was understood, not after they have walked to the next site. A small on-device transcription model and a compact vision model do the first read. Good enough to confirm intent, good enough to tell the inspector “I logged pump two, severity high, work order drafted” while they are still standing there. The heavier reconciliation, the cross-checks against the asset registry, the final structured action, those run at the edge or in the cloud when the record syncs.
The device is the source of truth. On-device inference confirms intent immediately; the heavier reconciliation and the human-review branch happen later, when the record syncs.
The confidence threshold is the product
A field system that is confidently wrong is worse than no system, because it manufactures a paper trail that says the wrong thing was verified. So the most important number in the whole design is the bar below which the machine refuses to commit and pulls in a person.
We score the fused result, not any single input. A transcription that came back clean, an image where the asset tag was readable and the fault was unambiguous, a voice intent and a visual reading that agree: that clears the bar, the structured action gets drafted, and the inspector confirms with a tap. But the moment the signals disagree, or the photo is too dark to read the tag, or the spoken severity outruns what the gauge supports, the confidence drops and the record routes to review instead of committing an action. A false “verified” in a regulated inspection log costs far more than a human glance, so the bar sits deliberately high, and disagreement between the inputs is treated as its own reason to pull a person in.
The threshold is not a model hyperparameter you tune once. It is a policy decision that belongs to whoever owns the risk. Set it too low and you drown the review queue and people start rubber-stamping, which is worse than no review at all. Set it too high and the system never commits anything on its own and you have built an expensive transcription tool. We landed it by category: a routine sign-off can clear at a lower bar than a fault that takes equipment offline. The high-severity actions, the ones with money or safety attached, always get a human, no matter how confident the machine is. (The one time we let the model auto-commit a high-severity flag to save a click, it flagged a reflection on wet metal as corrosion, and that was the end of that experiment.)
What I would tell someone building this
Design for the worst moment, not the demo. The demo has signal and good light and one clean input. The worst moment has none of those, and it is also the moment the most important capture happens. If your architecture only works in the demo, you have built a prototype that will be abandoned in the field within a month.
Treat the three inputs as one intent from the start. The instinct is to ship voice first, then bolt on image, then add the form. What you get is three disconnected pipelines and no fusion, and fusion was the entire point. The value was never in transcribing voice or reading a gauge. It was in the agreement between them, and in knowing, honestly, when they did not agree.
And put the human where the confidence runs out, not where it is convenient. The hard question in any field system is not how good the model is. It is who looks at the photo when the machine is unsure, and how fast. Get that loop right and the model can be merely decent. Get it wrong and the best model in the world just produces a faster, more confident kind of mistake.