How I Think About AI Products
The hardest thing about AI PM work is that the usual question does not work.
In standard product work, the question is: does this feature work? You ship, you measure, you iterate. A bug throws an error. A broken form shows a broken form.
AI products do not work that way. A wrong answer does not throw an error. It gets delivered to the customer as a confident recommendation.
That changes everything that comes before the ship.
The question that actually matters.
The question is not: does it work?
The question is: what happens when it is wrong?
Because it will be wrong. Every AI system is wrong sometimes. The difference between a good AI product and a bad one is whether you defined the acceptable failure modes before launch, or discovered them from angry customers after.
That definition is the PM's job. Engineering can build the model. Only the PM can say: at this confidence level, we flag for manual review. At this accuracy rate, we ship. Below this threshold, we do not.
Those decisions have to be made before the system goes live. Once it is live, you are making them under pressure, with customers waiting.
What I learned at MaidLink.
The AI estimator at MaidLink went through two versions before the current one.
The first was parametric: rooms, size, condition ratings. Rule-based logic, clean math. It failed. Not because the formula was wrong. Because the input was wrong.
Clients cannot accurately assess their own home's condition against professional cleaning standards. They see their space every day. Normalisation sets in. What they call “a bit messy” and what a professional cleaner calls “a bit messy” are not the same thing.
That is not a data quality problem. That is a perception problem. No amount of better instructions, clearer forms, or improved UX would fix it.
A generic PM redesigns the form.
An AI PM asks whether the form should exist at all.
We removed client self-assessment from the process entirely and replaced it with computer vision. Clients submit photos. The AI reads the actual condition. The estimate comes from what the camera sees, not what the client believes.
The evaluation framework.
Before Sindhuja wrote a single line of the computer vision estimator, I wrote the evaluation framework.
Not after. Before.
Because if you do not define what “accurate” means before the model ships, you will be arguing about it after launch, when the pressure to keep shipping is highest and the data is noisiest.
The framework covered four things:
Accuracy thresholds. What percentage of estimates had to land within acceptable range of actual job time — and what “acceptable” meant differently in beta versus general availability.
Failure mode classification. Not all wrong answers are equal. A 10% overestimate has different consequences than a 40% underestimate. We classified failure types by severity and defined what each required: correction, override, or re-prompt.
Confidence thresholds. When the model was uncertain, what did the user experience? We designed the fallback so uncertainty is handled invisibly — below a set confidence level, the estimate routes to manual review with no indication to the user that the AI flagged it.
Retraining triggers. What patterns in real job outcomes would tell us the model had drifted from reality? We defined those signals before launch so we would recognise them when they appeared.
That framework is not a document you write once and file. It is a living operational tool. Every real job outcome feeds back into it.
Why this changes how I work.
Most of what makes AI PM work different from standard PM work comes down to one thing: you are defining the product's acceptable failure modes as a primary design decision, not as an edge case.
That means working differently with data science. Not “make it more accurate” — but “here is the specific pattern we are seeing across eight jobs, here is the input signal we think we are missing, here is what we want to test.”
That means working differently with engineering. Not just what to build, but what happens when the output is below threshold. Building that path as carefully as the happy path.
That means working differently with customers. Not exposing model uncertainty as a UX problem. Designing the product so that the system's limits are handled invisibly, without eroding trust.
The framework for evaluating AI outputs is not something you learn and apply. It is something you build, decision by decision, from the data that comes back when you are wrong.
That is the part that cannot be faked. You either have the real data from a real product, or you do not.
I like talking about product problems.
If you are building AI products and want to think through what you are working on, I am happy to meet and discuss. No agenda required.