AI features fail differently from ordinary product work. They can look impressive in a demo, degrade under real inputs and still generate cost while they are wrong. A staff engineer needs a framework that keeps evaluation concrete when the technology feels novel.
This is not a catalogue of model vendors. It is a set of questions I use before committing a mobile product to an AI-assisted path.
1. User value under real conditions
What job does the user finish faster, more safely or more confidently because of the model?
If the answer is only “it feels modern,” stop. Prefer a narrow job with a clear before-and-after: drafting a reply, ranking candidates, highlighting risk, extracting structure from messy input. Demand a definition of success that a non-specialist can observe.
Also ask where the feature sits in the journey. Assistive surfaces that users can ignore are cheaper to get wrong than autonomous actions that change account state, money, privacy or safety.
2. Failure behaviour
Models will be wrong. The design question is what happens next.
Useful failure modes are legible: the user can see uncertainty, recover without support and understand what was automated. Dangerous failure modes hide confidence, overwrite user intent or leave no audit trail.
On mobile, failure also includes offline, delayed responses, partial results and interrupted sessions. If the product cannot degrade to a non-AI path, the feature is not ready for production ownership.
3. Privacy and data boundaries
What leaves the device, what is retained, who can access it and under which jurisdiction?
Classify inputs before architecture: account identifiers, messages, media, location, payment context, workplace data. Prefer minimum necessary retention, clear purpose limitation and explicit user control over sensitive processing.
If the team cannot explain the data path in plain language, the feature is not ready for trust-critical surfaces.
4. Observability and quality measurement
A feature without measurement is a demo with a longer runtime.
Define quality signals early: acceptance rate, edit distance from suggestions, escalation to humans, complaint rate, latency, cost per successful outcome and offline fallback usage. Separate model quality from product quality. A correct model output can still be a poor product interaction.
Instrument ownership matters as much as dashboards. Someone must notice regressions when prompts, models or upstream data change.
5. Human control and review
Where does a person remain accountable?
For low-risk assistive work, light control can be enough: preview, edit, dismiss. For higher-risk actions, require confirmation, dual control or human review before effects land. The control surface should match the blast radius, not the excitement of the demo.
Human-in-the-loop is not a slogan. It needs capacity, tooling, auditability and a path to improve the system from reviewed outcomes.
6. Cost and operational envelope
Estimate unit economics under realistic traffic, not a weekend prototype.
Include inference cost, retries, evaluation runs, human review time, storage and the engineering cost of keeping the feature healthy. Set kill switches and budget guards. A feature that is correct but unbounded is still a production risk.
7. Maintenance and ownership
Who owns prompt changes, model upgrades, evaluation fixtures, red-team findings and user-facing copy when behaviour shifts?
Treat AI-assisted product code as a long-lived system: contracts, versioning, regression suites and a rollback plan. Shared ownership without a named steward usually means silent decay.
How I use the framework
I score each dimension as clear, partial or blocked. A single blocked dimension can stop a launch. Partial scores create an explicit backlog rather than a vague intention to “monitor in production.”
The goal is not to slow teams down. It is to keep AI work inside the same staff-level discipline we already apply to payments, auth, offline behaviour and regulated product surfaces: evidence, ownership and recoverable failure.
If a proposal cannot answer these questions, the honest next step is a smaller experiment—not a wider rollout.