Most conversations about adding machine learning to a product start with the model. In practice the model is the part you can change later. The decision that shapes the whole system — cost, privacy, offline behaviour, even whether the feature is viable — is where inference runs.
There are three honest answers, and one of them is "nowhere".
Option one: on the device
The model runs on the phone. Nothing leaves the handset except the result.
This is the right answer when the input is high-bandwidth and sensitive at the same time — and video is both. In the fitness platform we built, pose and movement analysis runs on-device through Google ML Kit. Streaming workout video to a server for form checking would have been expensive per minute, would have put camera footage of people exercising on someone else's infrastructure, and would have failed exactly where it needed to work: on a gym's wifi.
Running it on the phone changes all three at once. There is no round trip, so there is no latency to hide and no per-request bill to grow. The backend only ever receives results.
The cost is real, though: model size affects app size, older phones are slower, and updating the model means shipping an app release. On-device is excellent when the input is heavy and the model is stable — not when the model changes weekly.
Option two: a service beside your own
The model runs as its own worker, next to the application, consuming a stream and raising events.
In the pilgrim-safety platform we built, bracelets stream vitals into a message queue, and a Python worker subscribes to that stream, applying both explicit threshold rules and anomaly detection. When something looks wrong, it raises an alert into the same pipeline every other alert uses.
The structural point is that the model is a subscriber, not a step in a request. It can be slow, restarted, or improved without the ingestion path caring. And because it publishes the same kind of event as everything else, an anomaly alert flows through the same delivered → escalated → acknowledged lifecycle as a panic button press. The machine learning does not get its own special path with its own special bugs.
That same system illustrates the pairing we default to: learned models alongside explicit thresholds. The thresholds are readable and testable, so when a decision matters, behaviour can be explained rather than guessed at.
Option three: don't
The third answer is the one worth protecting. A rule you can read and test beats a model you cannot.
If the logic is "alert when the heart rate crosses this line", that is a threshold, and dressing it up as a model adds training data, drift, monitoring, and unexplainability in exchange for nothing. This is why our AI engagements start with a short assessment of whether a model is the right tool at all, before any build is committed — and why saying "it isn't" is a valid outcome of that assessment.
The questions that actually decide it
Before choosing an architecture, four answers determine most of it:
- Can the raw input leave the device, legally and reputationally? Video of people, medical readings, and identity documents usually should not.
- What does it cost per request at real volume? A per-inference price that is trivial in a demo is a line item at ten thousand users a day.
- What happens with no network? If the feature must work in a warehouse or on a pilgrimage route, server-side inference is not a feature — it is an outage waiting for a schedule.
- How often will the model change? Frequent changes favour a service you can redeploy; stable models favour the device.
Answer those four and the architecture is mostly decided. Whichever way it lands, the model is one component inside a system that still has to be correct, observable, and maintainable — which is the part that outlives any particular model.