Many people got Senior AI PM roles, many have switched into AI PM entry roles, and some have up-levelled their existing job using this roadmap

Today I am going to give you a step-by-step guide to learn AI in Product Management, Program Management, Consulting or any role.

So this is that detailed guide.

Four steps.

I will tell you what I would actually do at each one, and what it would feel like, because the feeling is the part nobody warns you about.

Step 1: Learn just enough to make decisions

The first mistake I made was treating this like a degree. I thought I had to understand everything before I was allowed to build anything.

If I started over, I would give the foundations two or three focused weeks, and no more.

Here is what I would actually work through:

1. Supervised and unsupervised learning, and what each is good for

2. What training and inference really mean, in plain words

3. Transformers and attention, the intuition, not the math

4. Tokens and context windows, because they quietly drive your cost

5. Why models hallucinate, and what that forces you to design around

6. What RAG is, and when it is the right call versus overkill

7. Advanced Prompting, Data Orchestrating pipeline for building products

I would keep asking myself one question the whole time.

Can I make a decision based on this yet? Can I sit in a room and say why retrieval is right here, and a bigger model is wrong?

The day I could do that, I would close the tutorials and move on.

Step 2: Build real things

This is the step I skipped the first time, and it is the step that would have changed everything.

Here is what I would build my way through:

1. Learn about Claude Code and Vercel to build Products. Learn how to build native Android/iOS Apps using AI

2. Building RAG properly: retrieval, the knowledge base, chunking strategy, and a retrieval strategy that actually holds up. Metrics for RAG

3. Building an AI agent: autonomy levels, tools, memory, human-in-the-loop checkpoints, tested against three real scenarios

4. Building Evals: a golden test set, LLM-as-judge, and a launch threshold for good enough to ship, Eval Metrics and scorecard

5. Prototyping using Spec-driven development: the seven-phase flow from product note to functional spec to build to ship

6. UX for AI: trust signals, human-in-the-loop patterns, and the copy you write around uncertainty

The eval work is where I would grow as an AI PM.

Step 3: Learn the judgment layer

This is the part that makes you an AI PM and not just someone who can prompt. It is also what interviewers focus on.

Here is what I would work through:

1. Model selection: cost, quality, and latency, and which one to protect for a given feature

2. A model comparison matrix I can reason from

3. The metrics stack: a north star metric, supporting metrics, guardrail metrics, and an A/B plan that survives review

4. Responsible AI as real work: a bias audit, launch guardrails, a risk audit, not a disclaimer at the bottom

5. Go-to-market: where the moat is when the model is a commodity anyone can rent, and how pricing shifts when your cost scales with every user

6. B2C and B2B case studies, until the patterns start repeating

Step 4: Turn it into proof

The first time, I had knowledge and nothing to show. In an interview, that is not worth much. If I started over, I would treat proof as the goal, not something I do at the end.

Here is what I would do:

1. Pull the whole thing into one capstone: problem, spec, prototype, evals, go-to-market

2. Refine it into a tight eight-minute story anyone can follow

3. Turn the build into interview answers. The spec is my product sense answer. The evals are my metrics answer. The tradeoffs are my strategy answer.

4. Write a STAR story bank across product, metrics, strategy, behavioural, and technical questions

5. Run full mock rounds, under pressure, and fix what breaks

Now you have 3 Options

That is the route I wish someone had handed me.

Four steps, in order, one product carried through all of them until it becomes the thing you show.

1. Want to learn on your own? Just go step by step and start building.

2. Want to learn this structured approach at Self Pace? Click here (PMs from Google, Microsoft, and Coinbase learned from here, rated 4.9/5)

3. Want to learn this LIVE, with weekly feedback, Assignments, Build Hours, Capstone, Demo, and portfolio? Fill the form here

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. AI Product Manager/Builder Cohort

Yes, AI writes PRDs now.

Good ones. Yes, it turns forty customer interviews into a clean thematic synthesis in less time than it takes you to open Notion.

It writes the SQL. It builds the metric tree. It drafts the competitive teardown, generates fifteen solution options, converts a messy sixty-minute call into tidy tickets, and produces a roadmap deck that looks a lot like the one you spent Sunday night building.

Yes, a real chunk of what you spent five years getting good at now costs twenty dollars a month.

That is true. It is not hype, and pretending it is not happening protects nobody.

But here is the other true thing.

The product managers doing well right now are not the ones fighting the tools. They are the ones who worked out, quietly and early, what the tools cannot do. Then they got aggressively good at exactly that.

This was never a replacement story. It is a story about repricing.

What AI Is Actually Good At

Before we talk about your career, we should be honest about the machine: the honest answer is quite a lot.

AI is good at production. Give it a clear brief, and it returns something usable in seconds. It is good at generating options, which means you now start from twelve directions instead of a blank page.

It is good at the mechanical middle of the job: first-draft specs, acceptance criteria, edge-case enumeration, release notes, stakeholder updates, the thing you write to the thing you write.

It is very good at compression. Two hundred support tickets into eight themes. A competitor’s entire changelog into a positioning read. A quarter of Amplitude data into a paragraph you can send to your VP.

It is good at recall. It never forgets that you shipped a similar feature in 2024 and it went badly.

And it is good at speed, which is the part that quietly breaks everything. Because when the cost of producing a product artifact falls toward zero, the artifact stops being evidence of anything.

The PRD Was Never the Job. But It Was the Proof.

Here is the thing nobody in your org will say out loud.

For twenty years, product management was based on two things that were bundled together.

1. One was judgment.

2. The other was the throughput of artefacts.

Judgment is invisible, so nobody could price it directly. Artefacts are visible, so the org priced those instead and hoped they correlated.

The PRD was not the product. The PRD was the receipt. It was how you proved to people who could not see inside your head that you had actually thought.

AI just made it possible to produce the receipt without doing the thinking.

That is the whole disruption in one line. Not that your work got automated. That your proof got counterfeited, including by you, at three in the morning, when you did not have the energy to think, and the model was right there.

Which means the bundle has broken. Artefacts are now free. Judgment is now the only thing left to pay for. And a lot of PMs are about to discover that they were being paid for the wrong half.

What Just Got Expensive

1. Deciding what not to build

Engineering throughput is going up. In some teams, it has already doubled. Every instinct in a product org says: great, we can ship more.

That is the trap.

More features mean more surface area, more support load, more cognitive tax on the user, more code somebody has to maintain in eighteen months. When building was expensive, the cost of building acted as a filter. It stopped you. It killed your worst ideas for you, for free, before you ever had to argue.

That filter is gone. You are now the filter.

The PM who uses AI to ship three times more features is not three times more productive. They are quietly destroying their own product and calling it velocity. Restraint used to be a luxury. It is now the core competency, and almost nobody is training for it.

2. Context that cannot be scraped

The model has read the entire internet. It has not sat in the room where your biggest customer went quiet for four seconds before saying “yeah, no, it’s fine.”

It does not know that your CFO’s real constraint is not budget, it is a board meeting in November. It does not know that the sales team has been quietly promising a feature that does not exist. It does not know which of your three churn reasons is the real one and which two are what people say instead of the real one.

Every model in the world is working from public information. Your entire edge is private information. The PMs who are winning right now are the ones who have systematically increased their intake of the un-Googleable: sales calls, support queues, cancellation flows, the thing users do instead of the thing they said they would do.

Proprietary context is the last unfair advantage. Go get more of it.

3. Defining “good” for a wrong system four per cent of the time

This one is new, and it is worth more than everything else on this list.

Traditional software is deterministic. It works, or it does not. QA is a binary. AI features are probabilistic. They are confidently wrong sometimes, and your job is now to decide how wrong is shippable.

What accuracy do we ship at. What does a failure look like. Which failures are annoying and which ones are lawsuits? What is the eval set? Who writes the rubric? What do we do when the model is right on the metric and wrong for the user?

No model answers those questions for you, because those questions are value judgments dressed as engineering. The PM who can write a real eval, set a real threshold, and defend a real tradeoff between accuracy and latency and cost is doing something that did not exist as a job three years ago and is now the highest-leverage thing in the building.

If you learn one new skill this year, learn this one.

4. Owning the call

A model can generate a hundred roadmaps. It cannot be accountable for one.

It cannot be in the room when the number is missing. It cannot take the hit, revise the thesis, and go again with credibility intact. It cannot build the trust that makes seventeen people move in the same direction on incomplete information.

Accountability is the one input in the product process that cannot be delegated to software, because software has nothing to lose. You do. That is not a weakness in your position. That is your position.

5. Getting anyone to use the thing

When everyone can build it, building it is not the moat.

Distribution is. Adoption is. Being the reason a user changes a habit they have had for four years is. AI has made the supply of software effectively infinite and has done nothing to increase the supply of user attention. Attention is now the scarcest thing in your business, and it always was, and now everybody is about to find out.

The Part That Will Sting

The middle is collapsing.

If your day is: attend standup, translate the conversation into tickets, update the status doc, write the spec, chase the sign-off, send the summary, then you are describing a set of tasks that a competent model plus one senior PM now does in an afternoon. That is not a prediction. Look at your ratio of PMs to engineers over the last two years and extend the line.

And there is a second-order problem nobody has solved. Junior PMs used to learn judgment by writing three hundred bad specs and having them torn apart. That was the apprenticeship. If AI writes the first draft of everything, the apprenticeship disappears, and we get a generation of PMs who can review work but have never done it. Reviewing is not the same as knowing. You cannot recognise a bad tradeoff you have never personally made.

If you are early in your career, do not let the model do your reps. Write the spec badly, yourself, first. Then let the model tear it apart. The order matters more than the output.

What To Do On Monday

Five things. None of them requires permission.

1. Take your current roadmap and delete something. Not defer. Delete. Write one paragraph on why. That paragraph is worth more than the roadmap.

2. Book three customer calls this week that have no agenda and no demo. Your edge is private information. Go and get some.

3. Pick one AI feature in your product, or one you wish existed, and write the eval. What does good look like, in numbers, with examples of failure? If you cannot do this, you now know what to learn.

4. Stop using AI to produce your artefacts and start using it to attack them. Draft the thesis yourself. Then ask the model for the strongest argument that you are wrong. Then answer it.

5. Write down the last three decisions you made and what you were betting on. Not what you shipped. What you believed. That document is the only real evidence of judgment you will ever have, and increasingly, it is the only thing that separates you from a very fast intern with API access.

If you want to learn how to build a product using AI and become an AI Product Manager, you can join our 12-Week AI Product Manager/Builder Cohort (100 hours, including 20 Hours of Interview Prep)

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. AI Product Manager/Builder Cohort

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

You can spot an AI-written PRD in seconds.

• The problem statement is soft and vague.

• The solution is a feature list.

• The metric indicates a need to improve engagement and retention. Which means nobody measured anything.

People blame the model. Wrong.

The model is fine. It can reason better than most of us. The problem comes before the model runs.

We treat a PRD as writing. It is not writing. It is context.

And context is the one thing we forget to give.

A PRD is just compressed context

Think about what a PRD really is.

It is everything you know about a problem. Put in one place. So an engineer can build it without a meeting.

You know a lot.

The support ticket that started this. How three rivals handle the same moment. What your team owns, and what it does not. The PRD that got approved fast. The one that got torn apart.

None of it is written down. It sits in your head.

Then you type “write a PRD for smart substitutions.” And you send none of it.

So the model guesses. It uses the average PRD it has seen. And hands that back.

That is what generic means. The model is not lazy. It is hungry. You did not feed it.

One giant prompt will not fix it

The next idea is simple. Paste everything into one big prompt. Research, notes, rules, screenshots.

Then “now write it.”

It does not work. Here is why.

The model does not read a long prompt evenly. Stuff in the middle gets ignored. Add too much, and it starts contradicting itself.

You have felt this already. A long chat forgets what you said at the start. So you open a new chat. Now it knows nothing.

So the big prompt fails twice. It does not carry to the next PRD. And it breaks inside one chat.

You do not need a better prompt. You need a place that holds your context.

Build the engine, not the prompt

Every big tool now has this. A workspace that remembers

You load your files once. After that, it pulls what it needs into each chat. No more pasting the same files every day.

Here are the ones I use. And the one thing you learn about each is only by using it.

1. Claude Projects. Make a Project. Call it PRD Engine. Add your template, two or three old PRDs, the research, and the competitor teardown. Put your writing rules in the instructions box, not as a file. Files are what it knows. Instructions are how it behaves.

2. ChatGPT Custom GPTs. Same thing, but you can share it. Upload the files. Add your rules. Share the link. Now your whole team drafts the same way. One catch. If the rules are too long, it ignores half of them. Keep them short.

3. Gemini Gems. Best if your team lives in Google Workspace. A Gem can read straight from Drive. Point it at your folder. But it only sees files shared with the account. A locked file is invisible. Then it makes things up, and you wonder why.

4. NotebookLM. Different tool. It reads. It does not invent. Load twenty user interviews. Ask what people said about stockouts. It answers with the exact line. It never leaves your sources. That is the point. Use it to build proof. Then take that proof to your drafting tool.

Pick one and start. Do not overthink the brand.

Set it up once. Use it for every PRD. Improve it slowly.

Now, one example.

What to put in the engine

Two buckets. Rules. And reality.

Rules are your calls. Only you can make them.

The PRD template. So the model stops making up a format.

Your scope. Write it plainly. You own the substitution flow and the notification. You do not own picker staffing, stock accuracy, or the packing SLA. Skip this, and the model gives you a fix that needs better stock accuracy. Fun thing to learn live in a review.

One PRD that shipped well. One good example teaches your bar faster than the word “detailed.”

Reality is what is actually true on the ground. This is where generic dies.

Load the real research. Not a summary.

Users leave after a silent refund. It ruins the meal they planned. They also leave after a bad swap. Toned milk for full-fat. They will approve a swap. But they will not wait long.

Add the competitor teardown. In your words. What each rival does well. What annoys people?

Add the numbers. How often a stockout hits mid-order. What one bad swap costs. The refund, plus the orders they never place again.

The model can arrange all this. It cannot know it. That part is yours. Always.

Writing the draft

Here, people hand over too much. They ask the model to invent the solution.

Do not. The idea is yours.

Give the shape. Let it do the work.

For substitutions: one pushes the moment an item goes out of stock. Three ranked swaps. A ninety-second window to approve. A fallback if the user says nothing.

Then ask it to map every path. The clean one. And all the messy ones.

Point each section to its source. Problem from research. Limits from scope. Depth from the sample PRD.

Give it specifics. Get specifics back. That is the whole trick.

The loop is the real work

The first draft is a draft. That is the step everyone skips. And it is the one that matters.

Read it like a reviewer. Not like the author who just made it.

Look for one thing. Where did the model guess. And did it guess wrong.

Ask a few questions. Did it handle the messy paths, or just name them? Are the metrics real, or nice words? Did it turn a guess into a fact?

Run the substitution draft. The gap shows up fast.

The happy path is clean. User pinged. Picks a swap. Order moves on.

But the ninety-second timeout gets one weak line. The case where the user does not tap.

That line is the whole product.

Refund the item, or swap the top pick by default. One protects revenue. One protects trust. They pull opposite ways.

It is the biggest decision in the PRD. The draft treated it as small.

Why? Because you never told it the ninety seconds is fixed. It is tied to the packing SLA. Not the model’s fault. A fact you had and did not share.

So do not fix the draft by hand. Go back to the engine. Add the SLA rule. Add the “what is the default” question. Run it again.

That is the loop. Draft. Find the guess. Add the missing fact. Run again.

And every round makes the engine smarter. Not just the doc. Your next PRD starts from a workspace that already knows your SLA, your scope, your bar.

What your job becomes

You did not write the PRD by hand. You also did not accept junk.

You did the real work. You chose what context matters. You fed it in. You edited the thinking.

The model did structure. You did truth and judgment.

That is the shift.

The best PM here is not the one with the best prompt. It is the one with the best context. And the patience to keep feeding it.

The document falls out at the end. The engine is what you built.

Most people will keep typing one-line prompts into an empty box. And keep getting the same dull output.

You do not have to.

Build the engine once. Feed it well. Edit like the toughest reviewer in the room.

Those generic PRDs were never the model’s fault. They were context you had and did not share. And you are the only one who can.

If you want to learn how to build product using AI and become AI Product Manager, you can join our 12 Weeks AI Product Manager/Builder Cohort (100 Hours including 20 Hours of Interview Prep)

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. AI Product Manager/Builder Cohort

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

Imagine you are a product manager at a fintech company.

You are building an AI assistant inside the app.

The assistant needs to do four things.

1. Check account balance.

2. Pull the last 10 transactions.

3. Transfer money.

4. Update KYC details.

Four simple jobs.

You sit with the engineering lead to scope it.

Balance lives in the core banking system. Transactions live in a data warehouse. Transfers go through the payments service. KYC sits inside a compliance vendor.

Each system has its own API. Each API has its own authentication. Each API returns data in its own shape.

The engineer looks at you and says six months.

You go back to your desk and start writing the requirements. Somewhere between rate limits and error handling, you realise something uncomfortable.

You are not building an AI product. You are building glue code.

The Real Problem is Not the Model

Every AI product team is stuck in the same trap. The model is not the bottleneck. The bottleneck is everything around the model.

Here is what the trap looks like in practice.

1. Every new tool the AI needs requires a fresh integration.

2. Every API change breaks the connection.

3. Every model upgrade forces a prompt rewrite.

4. Every new use case adds another custom endpoint.

Your team spends 70 percent of its time writing plumbing. It spends 30 percent building the actual product.

This is why most AI features die in staging. Not because the model is weak. Because the integration cost is too high.

What MCP Actually Is

MCP is an open standard. It defines how AI models talk to tools, data, and context in one consistent way.

Think of it as a shared language between the model and everything outside the model.

Anthropic introduced the protocol. The industry adopted it fast, and the reason is simple.

Why Traditional APIs Do Not Fit AI Models

Traditional APIs were built for deterministic programs.

A program knows exactly what it wants. It calls the endpoint. It parses the response. It moves on.

A language model does not work like this. A model reasons probabilistically. It handles fuzzy inputs. It asks clarifying questions. It changes plans mid-task.

Instead of hard-coding every integration, the model queries what is available. It reads what each tool does. It chains tools together based on the task at hand.

The Architecture in One Picture

MCP runs on a client server model. Here is the whole system on one screen.

Two things to notice in this picture.

• The first is that the model touches only one interface. Four systems, one protocol.

• The second is that nothing below the MCP servers changes. The REST APIs, the SQL, the vendor integrations all stay exactly as they are. The MCP server wraps them.

The client is the AI agent. The server is the environment exposing resources. That is the entire mental model.

The Handshake: How Discovery Works

Here is where MCP gets interesting.

Read that response carefully. It is doing three jobs at once.

1. It names the tool.

2. It describes the tool in plain language the model can reason about.

3. It defines the exact inputs, their types, and which ones are mandatory.

This is the part that replaces prompt engineering. Nobody writes a system prompt explaining how transfers work. The schema is the documentation, and the model reads it at runtime.

No manual wiring. No static configuration. The payments team ships a new tool tomorrow, the model sees it tomorrow.

The Four Building Blocks

Every MCP server exposes some combination of four resource types.

• Tools are actions the model can invoke.
  Send an email. Search a database. Initiate a transfer. Each one carries metadata explaining what it does and what it needs.

• Resources are pieces of state or data.
  An image, a database row, a text document. Anything the model reads to reason about the task.

• Prompts are reusable templates.
  The server developer writes them once. Every model that connects gets them for free.

• Context is external information pulled into the reasoning process.
  User preferences, company data, chat history.

A useful way to remember the split. Tools are verbs. Resources are nouns. Prompts are instructions. Context is memory.

The model treats all four identically regardless of which server they come from. A calendar server, a CRM, a GitHub server. Same language everywhere. That uniformity is the entire point of the protocol.

One Request, End to End

Now let us return to our fintech PM and trace a single request through the whole system.

A customer types: I want to send 5000 rupees to my mother.

Step 1. The model reads the tools advertised by all four servers. It matches the intent to initiate_transfer on the payments server.

Step 2. The model checks the input schema. It has the amount. It needs a beneficiary_id. It queries the resources on the banking server, finds a saved beneficiary tagged as mother, and picks up the ID.

Step 3. The model calls the tool.

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "initiate_transfer",
    "arguments": {
      "amount": 5000,
      "currency": "INR",
      "beneficiary_id": "ben_88213"
    }
  }
}

Step 4. The MCP server translates this into a call to the actual payments API, waits for the result, and returns a structured response.

{
  "jsonrpc": "2.0",
  "id": 7,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Transfer of INR 5000 to ben_88213 initiated.
                 Reference: TXN-4471-2026. Status: PENDING_OTP"
      }
    ],
    "isError": false
  }
}

Step 5. The model reads the result and replies to the customer. It sees PENDING_OTP in the response and asks the customer for the OTP. It never had to be told that transfers require an OTP. The response told it.

Notice what did not happen anywhere in this trace.

Nobody wrote a custom prompt for the transfer flow. Nobody wrote glue code between the model and the payments API. Nobody hard-coded the OTP step.

The six month plan compresses into weeks. Engineering builds four MCP servers, each one a thin wrapper over an existing system, and the assistant composes them on its own.

If you want to learn how to build product using AI and become AI Product Manager, you can join our 12 Weeks AI Product Manager/Builder Cohort (100 Hours including 20 Hours of Interview Prep)

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. AI Product Manager/Builder Cohort

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

Video Sources

The interviewer is testing your ability to design a solution and a system around AI.

So in your next Product Management Interview, you might get this, so read carefully.

Let’s try to think about Deep Research Agent from first-principles thinking.

First, a deep research agent is not a search chatbot

A search chatbot takes one query, pulls a few results, and writes one answer in seconds.

A deep research agent takes one open question, builds a plan, reads dozens to hundreds of sources, works for minutes, and returns a long cited report.

The difference is not scale.

The difference is that the agent decides its own next step. That single property is what changes everything downstream.

The core design problem

Strip it down, and you are running an unbounded search over an unbounded space, on a fixed budget, against a hard reliability bar.

Four hard problems fall out of that framing.

1. Scoping. Turning a vague question into a bounded investigation.

2. Stopping. Knowing when the answer is good enough.

3. Grounding. Keeping the final report tied to real evidence.

4. Memory. Managing far more content than fits in a single context window.

Every architecture decision you make is a response to one of these four. Keep them in mind as we walk the system.

The architecture

Here is the loop at a glance.

Now the components. For each one, I have added the product decision that sits underneath it, because that is the part a PM actually owns.

1. Intake and scoping

A model pass reads the raw question and turns it into a structured brief.

• Goal

• Scope

• Success criteria

• Output format, and what is out of bounds.

If the ask is broad or ambiguous, the agent does not guess. It fires back two or three clarifying questions, then writes the brief. From that point on, the brief is the contract that every later step gets checked against.

The real choice is whether to clarify or infer. Asking first sharply improves scope and adds friction at the same time.

OpenAI’s Deep Research asks. Much of the final quality is decided right here, before a single search runs.

2. Planner

The brief goes to a model that emits a plan.

Usually a list or a shallow tree of sub-questions, each tagged with what to look for and how much effort it deserves.

In practice, this is a structured object the orchestrator can read, not prose. The plan is either fixed for the whole run or revised as evidence arrives, adding a sub-question here and dropping a dead end there.

Static or dynamic is the fork that matters. A fixed plan is cheaper and more predictable. A plan that adapts as the agent learns is where the real quality lives, and where cost and unpredictability creep in.

3. Orchestrator

This is a loop over a state object that holds the plan, the findings so far, and the remaining budget.

Each pass does four things.

1. Pick the next sub-question.

2. Choose a tool. Run it.

3. Write the result back into state.

4. Then the model looks at the updated state and decides whether to keep going.

It exits when the plan is covered or the budget hits zero.

Budget is the part you cannot skip. Steps, tokens, time, and money are all finite, and without hard ceilings, a single query can quietly cost you ten minutes and several dollars. This is the layer that stops the agent from running away.

4. Retrieval and tools

Search is only the first move. The agent issues a query, gets links, then fetches and reads the full pages, pulls the passages that matter, and follows citations down to primary sources when it needs to.

Tools are handed to the model as callable functions. Search for a query. Fetch a URL. Run code. The model decides which to call and when.

Source quality is what separates a good agent from a fluent liar.

5. Working memory

The agent reads far more than fits in one context window, so it cannot hold everything at once. It writes as it goes.

A short summary of each source, the exact quotes that matter, and a source ID pinned to each. This lives in an external store, a notes file or a vector index.

At each step, it pulls back only the notes relevant to the current sub-question, and long results are compacted into summaries before they land in the store.

Compaction is the delicate part. Summarise too hard, and you lose the detail that made the source worth reading. Keep too much, and the context bloats and the reasoning degrades. Every long run lives or dies on how well you strike that balance.

6. Reasoning and verification

Gathering and thinking alternate. After each fetch, a model pass interrogates the findings.

Is the sub-question answered?

What is still missing?

Do any two sources disagree?

When a claim is contested or load-bearing, the agent goes and finds a second independent source before it accepts it. The result of this step feeds straight back into the plan, marking a sub-question done or spawning a follow-up.

How hard to verify is the dial you turn. Cross-checking costs extra calls. Skipping it is exactly how confident, unsupported claims slip into the report. This is the trust layer, and trust is the entire product.

7. Composition

A separate writer pass builds the report. It reads the clean findings store, not the messy browsing history, and assembles structure, sections, an executive summary, and inline citations mapped from the claim-to-source pointers collected earlier. Splitting the writer from the researcher lets each do one job well. One is optimised for coverage, the other for clarity.

Resist the urge to research and write in a single call. The output gets noticeably worse when one pass tries to do both.

8. Evaluation and stopping

A final checker holds the report against three things. The plan, to confirm every sub-question was addressed. The sources, to confirm every claim carries a citation. And itself, to catch repetition. Stopping fires on whichever comes first. Full plan coverage, diminishing new information, or an exhausted budget.

Everything here rides on your definition of done. Coverage, groundedness, low redundancy. If you cannot write that definition down, the agent has no way to know when to stop either.

The one fork that defines your product

Single agent, or an orchestrator with subagents.

A single agent is simpler, cheaper, and easier to keep coherent. It struggles with breadth.

An orchestrator that spawns subagents to chase sub-questions in parallel covers far more ground and handles harder questions. Anthropic built its research system this way, with a lead agent directing workers. The cost is real. They reported it burned roughly fifteen times the tokens of a normal chat, and coordinating subagents adds a new class of failure.

There is no correct answer. There is only the answer that fits your question complexity and your unit economics. Pick deliberately.

The North Star Metric?

Number of research reports generated?? North Star??

NO

Because it is an activity metric, not a value metric.

A deep research agent can produce a thousand confident, wrong reports, and the number still goes up. Your North Star should move only when real value is delivered. So value has to be built into the definition itself.

Here is the one I would pick.

North Star. Weekly Trusted Research Tasks. A trusted research task is one the user accepts and acts on without redoing the work themselves.

Two words carry the weight.

• Trusted means the output earned action.

• Weekly means we are tracking a habit, not a one-time trial.

Volume lives in the count. Quality lives in the word trusted. A good North Star should be impossible to move by gaming a single dimension. This one clears that bar.

Question. Should time saved be the North Star instead?

Tempting, because saving time is the real benefit. But time saved is hard to measure honestly, and it is a downstream effect of trust and quality. Report it as a headline outcome. Do not steer by it.

Breaking it into components

A North Star you cannot decompose is a slogan. Here is the decomposition into three components.

Each component has a literal meaning and a physical meaning. The physical meaning is what matters, because it tells you what real-world force you are actually measuring.

• Active Researchers: Literally, people who ran at least one real research task this week. Physically, this is a demand. It is the market deciding whether your agent is worth bringing a real question to.

• Tasks per Researcher: It measures whether the agent has embedded into how someone works, or whether it was a novelty they tried once.

• Trust Rate: It is the only component that captures quality, and it is the one most teams forget to instrument.

Multiply the three, and you get value delivered. Move any one of them, and the North Star moves for an honest reason.

Failure modes to design against

• Scope drift, where it answers a slightly different question than the one asked.

• Shallow breadth or narrow depth are the two ways a plan can be wrong.

• Low-quality sources echoed as fact.

• Hallucinated citations that point to real pages that do not support the claim.

• Context rot, where the agent loses the thread across a long run.

Most of these trace back to the four core problems. Scoping, stopping, grounding, memory. If your design has a clear answer for each, you have handled most of the list.

If this is the muscle you want to build, there are two ways I can help, and they are different things.

The AI PM Builder Cohort is the deep end. A small group builds systems like this live, over twelve weeks. Real agents, real evals, real portfolio work you can show. If building beats reading for you, that is the room to be in.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. AI Product Manager/Builder Cohort

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

Your resume says you built RAG and an AI agent.

Here are the five questions that find out if you actually did.

Your resume says you built a RAG pipeline and an AI agent. So does everyone’s.

Then the interview gets specific, and surface knowledge runs out. The same questions catch everyone, because everyone wrote the same two lines.

Here are the five. Know the answers, and you have the depth. Do not, and you get found out, in the interview and again three weeks after launch.

“How do you measure if retrieval is working?”

It is two numbers, not one. People give one and get caught.

• Recall: Out of the times the right document existed, how often did it show up in the top results?
  You build a test set of questions, each paired with the document that answers it, then check.

• Faithfulness. Is the final answer actually supported by what was retrieved, or did the model make it up?

A strong model hides bad retrieval.

It guesses a fine answer from its own training, so the output looks good while retrieval is broken. Then a user asks about your newest policy, which the model never saw, and it falls apart.

Measure retrieval and the answer separately, or you are flying blind.

A better model just dropped. Do you switch?

If your answer is "let me try it and see," you are missing the point that matters.

You cannot tell if a new model is better without an eval set. The same test questions paired with right answers.

Run the old model and the new one against it, compare the scores, decide with data. No eval set means you switch on vibes.

That eval set is your real asset, not the model. Anyone can rent the model. Nobody else has your test of what good looks like for your product.

Why an agent here? Why not one model call?

Most things people call an agent are one model call and one function.

The word adds cost, latency, and new ways to break, and usually buys nothing.

Use an agent only when the task truly needs many steps and real decisions between them.

Routing a support ticket does not. A small classifier trained on your old tickets is faster and cheaper. Catching fraud does not work on fixed rules, because fraud keeps changing, so there you need a model that learns.

What happens when a tool call fails halfway through?

In a demo, nothing fails. In production, things fail constantly.

The API times out. The model returns the wrong format. Retrieval comes back empty.

A real agent expects it. The tool fails, it retries, tries another way, or stops and asks a human instead of guessing forward.

A demo agent assumes every step works.

Answers are the same. A confident wrong answer looks exactly like a right one. Most models can tell you how sure they are. When it is sure, let it act. When it is not, hand off to a person, or let it say I do not know.

The version that ships is the one that holds when something goes wrong.

What does one task cost at scale?

Answer in cost per call and you have missed it. An agent task is a loop of calls.

10Rs a call. 8 calls a task is 80Rs. A million tasks a day costs a lot. Nobody approved that, and finance will find it.

So you measure cost per finished task, and cap the steps before it stops. Speed counts too. A search that takes three seconds is broken, even with perfect answers, because the user already left.

Five questions, one job underneath. Handling uncertainty.

A normal PM removes uncertainty with a tighter spec. With AI, you cannot, because it does not behave the same way twice. So you measure it, plan around it, and budget for it.

That is the whole gap between the resume line and the real skill. Not more words. More depth.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. AI Product Manager/Builder Cohort

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

Everyone is building AI agents now.

Almost no one is building their memory.

That is the real gap. Not reasoning. Not tools. Memory.

And memory is not one thing. It is four. Most teams build one of them, call it done, and ship an agent that forgets in three other ways.

This article breaks AI agent memory down from first principles.

1. We will build one real agent, derive a simple equation for what makes it work,

2. and visualise the architecture so you can actually see it.

3. By the end you will be able to look at any forgetful agent and name exactly which memory is broken.

Let us start where the pain starts.

You Are a PM at Booking.com

Imagine you are a product manager at Booking.com.

Your job this quarter is to ship an AI travel agent. Not a chatbot that answers FAQs. A real agent.

The user types one line, “plan me a 4-day trip to Goa under 40 thousand rupees,” and the agent searches hotels, checks availability, builds an itinerary, holds a room, and completes the booking.

You demo it to leadership. It works. Everyone claps.

Then real users touch it.

A user says Plan me a Goa trip under 40 thousand.

Five messages later, the agent suggests a hotel for 55 thousand. It forgot the budget.

A returning user who has booked with you eleven times gets treated like a stranger. No memory of the beach resort she loved or the airport hotel she hated.

The agent quotes a cancellation policy that changed last month. Confidently. To a paying customer.

And once, it booked the same room twice because it lost track of which step it was on.

The model is fine. It is one of the best models in the world.

Why You, The PM, Have To Understand This

You might think this is an engineering problem. Smart agent, smart engineers, they will sort it out. Can the agent not just figure it out on its own?

No. And here is why.

Your engineers will build exactly what you spec.

If you write “the agent should remember the user,” they will ask Remember what, for how long, retrieved how, and you will not have an answer.

So they pick one kind of memory, ship it, and you get an agent that forgets in three other ways.

Memory is not one feature you can hand off. It is a set of product decisions only you can make.

• Which things the agent holds in the moment.

• Which facts it looks up.

• Which workflows it follows.

• Which past events it carries forward.

If you do not understand the four memories, you cannot spec them. And an agent you cannot spec is an agent that ships broken.

So let us break it down properly. From first principles.

The Agent Equation

Start by asking what an agent even is, mathematically.

An agent does three things. It thinks. It remembers. It acts.

Strip those down, and you get a simple equation.

—> Reasoning is the model. The planning and the thinking.

—> Action is the tools. The ability to do things in the world.

Memory is everything it can recall while it reasons and acts.

Now look closely at why these are multiplied, not added.

Multiplication means dependency. If any term drops near zero, the whole agent drops near zero. You cannot buy back a missing factor by scaling another one.

Now zoom into the Memory term, because that is where the four types live.

That is the entire thesis in one line. Now let us derive the four from first principles, using the one system you already trust. Your own brain.

Start With Your Own Brain

Before we touch the agent, look at your own head.

• You are holding a phone number right now because someone said it ten seconds ago. That is one kind of memory.

• You know Goa is in India. You did not look it up. That is a different kind.

• You can drive a car, but you cannot explain how you balance the clutch. Different again.

• You remember your last trip. The hotel. The bad breakfast. The view. Different again.

Knowing a fact is not the same as remembering an event. Knowing that something is true is not the same as knowing how to do it.

Your brain runs four memories at once. Your agent needs the same four. Working memory. Semantic memory. Procedural memory. Episodic memory.

Every failure in your Booking.com demo was one of these four breaking. Here is the full stack, then we take them one at a time.

1. Working Memory. What the agent is holding right now.

This is the phone number in your head.

For the agent, it is the context window. Every token in the current conversation. The user’s request, the last few messages, the search results it just pulled, the budget you set.

This is the memory that failed when the agent forgot the 40 thousand budget.

The budget was mentioned in message one. By message six, the conversation had filled with hotel options, dates, and itinerary details. The budget scrolled out of the agent’s working view. It cannot act on what it can no longer see.

There is a sharper failure hiding here. Even when the budget is technically still in the window, models recall the start and the end of a long context better than the middle. Bury the budget in the middle of a long chat and the agent quietly loses it. The information is there. The attention is not.

So when a vendor brags about a one-million-token context window, ask the real question. Not how much can it hold. How much can it actually use.

Use case where it is required. Any multi turn task. Trip planning, negotiation, troubleshooting. Anywhere the request evolves over several messages and the agent must hold the running state.

When working memory breaks, the agent contradicts itself and forgets what you told it two minutes ago.

The fix is not a database. It is managing what goes into the window, and pinning the constraints where the model will actually see them.

2. Semantic Memory. What the agent knows.

This is knowing Goa is in India.

Stable facts. For your agent, this is everything it needs to know about your world. Hotel details. Room types. Cancellation policies. Visa rules. The user’s loyalty tier. City guides.

None of this lives inside the model. None of it should sit in the context window all the time. It lives in a separate store, and the agent pulls in the right facts only when it needs them.

This is the memory that failed when the agent quoted the wrong cancellation policy.

In practice, this is RAG. A knowledge base, usually a vector database, that the agent searches before it answers. The policy changed last month. The knowledge base still had the old version. The agent retrieved a fact that used to be true and stated it with full confidence.

Use case where it is required. Any agent that must be correct about your specific business. Support, sales, compliance, booking. The moment the agent needs to know something the base model was never trained on, you need semantic memory.

When semantic memory breaks, the agent is fluent and wrong. The model is not the problem. Your knowledge base is.

3. Procedural Memory. How the agent acts.

This is knowing how to drive without relearning it every time.

For the agent, procedural memory is the learned workflow. The exact steps to complete a booking. Search, check live availability, place a temporary hold, confirm with the user, take payment, send confirmation, release the hold if it fails.

This is not a fact to look up. It is a sequence to follow, the same way, every time.

This is the memory that failed when the agent double-booked a room. It lost track of the step it was on. It confirmed before it held. The workflow broke.

In practice, you encode procedural memory in system prompts, rules files, and skill definitions. The booking sequence is written down so the agent does not improvise it on every run.

When procedural memory is missing, the agent is unpredictable. Same task, different path every time. In a booking flow, that is not a quirk. That is a refund and an angry customer.

4. Episodic Memory. What the agent has lived through.

A specific event, tied to a time, that happened. This is the hardest memory to build, and the one almost every product is missing.

For the agent, episodic memory is recalling specific past interactions with this user. Not a general fact. The actual event.

Last June this user booked a beach resort and complained it was 40 minutes from the station. In March, she cancelled because the hotel had no airport pickup. She always books twin beds.

This is the memory that failed when your loyal eleven-time customer got treated like a stranger.

This is the difference between an assistant that helps you and an assistant that knows you. It is also what lets the agent get better over time instead of starting from zero on every trip.

Be honest about where the industry is. The memory features you have seen, an assistant that remembers you prefer beach hotels, mostly store facts about you. That is semantic memory wearing an episodic coat. True episodic memory, recalling the specific arc of past trips and reasoning from it, is still rare and still hard.

Under the hood it usually works like this. Every interaction is written to a store as an event, with a timestamp and an embedding. When a new request comes in, the agent retrieves the most relevant and most recent past events and pulls them into the context window. Which means episodic memory only works if semantic retrieval already works, and only delivers value if working memory has room to hold what it retrieves. The four are not independent. They stack.

When episodic memory is missing, the agent never learns from the user and makes the same mistake every visit.

How The Four Stack In One Real Request

Here is the part most people get wrong. They think these four are separate features you can pick from a menu.

They are not separate. They stack.

What happens when your returning user types “book me the usual.”

Episodic memory recalls that the usual means a twin room near the beach. Semantic memory pulls the current price and the live cancellation policy. Procedural memory runs the booking steps in the right order. Working memory holds the whole thing together while it happens.

Pull out any one, and the request fails. A real agent needs all four, wired together.

Multi-Agent Memory. The Part Teams Get Wrong At Scale.

One agent is the easy case.

The moment your Booking.com system grows up, it becomes several agents. A supervisor that plans the trip. A search agent. A booking agent. An itinerary agent. This is a multi-agent architecture, and it breaks memory in a brand new way.

The question is no longer what memory does the agent need. It is which memory is private, and which memory is shared.

1. Working memory is private. Each agent has its own context window for the task at hand. The search agent should not be cluttered with the itinerary agent’s running state.

2. Procedural memory is private. Each agent has its own skills. The booking agent knows the payment sequence. The search agent does not need it.

3. Semantic memory must be shared. If the search agent and the booking agent read different price lists, they will quote different numbers and the booking will fail.

4. Episodic memory must be shared. This is the one teams miss. If the search agent remembers the user hates airport hotels but the itinerary agent does not, the system contradicts itself and the user notices instantly.

The rule is simple. Private memory makes each agent focused. Shared memory makes the system coherent. Get the split wrong, and your agents argue with each other in front of the customer.

AI PM Interview Questions On Agent Memory

If you are interviewing for an AI PM role, agent memory is now a core system design topic. Here are the questions you should be ready for, with what a strong answer covers.

1. You are building a travel booking agent. Users complain it forgets their budget mid-conversation. Which memory system is failing and how do you fix it?

2. Your support agent confidently quotes a refund policy that changed last month. Diagnose it.

3. Your team wants to fix a forgetful agent by moving to a one million token context window. What is wrong with that instinct?

4. In a multi-agent system, which memory should be shared across agents and which should stay private?

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass (Here)

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

We are launching a 12-week Cohort ( 100 Hrs of Learning) and 10+ Live Projects. Please find the details here

Become AI Product Builder

Nvidia keeps 75 cents of every dollar from its chip sales.

That margin is paid by the companies buying those chips to power AI.

Amazon spent seven years building a way to stop paying. This quarter’s AWS earnings call showed the strategy is working.

The chip business inside AWS grew 40% in a single quarter. The annual run rate is 20 billion dollars. As a standalone company, it would do 50 billion, putting it in the top three data centre chip companies in the world.

There is 225 billion dollars of committed Trainium revenue already booked from Anthropic, OpenAI, Uber, and a long tail of others.

This is the most important strategic pattern in AI right now. Almost nobody else is running it.

This is uncomfortable because almost every strategy framework taught in product circles points the opposite way.

Build a moat. Defend differentiation. Charge a premium.

That playbook works when buyers can distinguish between supplier A and supplier B.

Agents calling APIs for tool use and reasoning chains cannot. They have a latency budget and a cost budget. They do not have a brand preference.

The math that decides who survives in a commodity market is unforgiving.

Market clearing price = Cost of the worst supplier still in business
Your margin per unit   = Clearing price - Your cost per unit

The market sets the price at whatever keeps the least efficient supplier alive. Everyone with a cost below that line captures the gap as profit. Everyone above it dies.

AI demand currently exceeds supply, which means the clearing price is artificially high.

This hides the cost problem for almost every provider selling inference today. When supply catches up, the clearing price drops to whatever the worst surviving cost structure can sustain. Weak unit economics get exposed in a single quarter, not slowly.

This is what makes Trainium quietly brutal.

Trainium 2 delivers 30% better price performance than comparable Nvidia GPUs. Trainium 3 stacks another 30 to 40% on top. AWS is producing tokens at roughly half the cost of a competitor running on Nvidia silicon.

Almost all of that gap comes from one place. Decompose a dollar of inference revenue. The split below is illustrative but directionally honest.

A provider buying Nvidia chips:

Silicon cost                  : 40 cents  (Nvidia's ~75% margin baked in)
Networking, memory, storage   : 15 cents
Power and cooling             : 10 cents
Operations and overhead       : 15 cents
-------------------------------------------
Margin remaining              : 20 cents

The same dollar at AWS on Trainium:

Silicon cost                  : 15 cents  (manufacturing only, no Nvidia layer)
Networking, memory, storage   : 15 cents
Power and cooling             : 10 cents
Operations and overhead       : 15 cents
-------------------------------------------
Margin remaining              : 45 cents

The 25 cent gap is not a cost optimisation. It is Nvidia’s gross margin layer that is being eliminated by vertical integration.

The competitor’s COGS is the supplier’s profit. Apply that to 225 billion dollars of committed revenue and the compounding is obvious.

There is a second half of the story that almost nobody picked up from the earnings call.

Jassy said in passing that AI is driving growth in AWS’s core non-AI business. Post-training, reinforcement learning, agent tool use, and data movement are pulling more workloads into AWS even when the AI itself runs elsewhere. The reason is mechanical. Compute follows data. Data is heavy. Customers have been storing it in AWS for a decade. The agents that act on that data run in AWS. The inference that powers those agents runs on Trainium.

This is data gravity expressed as strategy. Every existing cloud platform is suddenly an AI platform not because it built better AI but because it sits on top of the data the AI needs. Every standalone AI product is fighting a tax that scales with how much customer data has to migrate before the product becomes useful.

Most AI product teams sit in the bottom right and act like they are in the top left. The seat-based pricing, the feature roadmap, the marketing playbook are imported from a non-commodity world. The unit economics underneath are not.

Two questions are worth sitting with if you are building anything that touches inference.

1. What is the cost per unit of value you deliver, and who controls that cost?
   Most teams know the customer-facing price. Most cannot say what one inference, one agent run, or one retrieval costs them at the silicon and power level. That number decides whether you survive when supply catches up.

2. Where does your data gravity live?
   If your product sits next to where the customer data already exists, adjacencies you did not design for will fall into your lap. If it does not, adjacencies you thought you owned will leak. AI accelerates this. It does not soften it.

The thesis has real ways it can break. If power becomes the binding constraint instead of chips, Nvidia’s tokens-per-watt advantage flips the math. If Amazon walls off Bedrock to protect Rufus from third-party shopping agents, the developer ecosystem loses trust. If the frontier labs move primary inference off Trainium, the cost lever weakens. None of these kill the thesis. They shape the timing.

The deeper takeaway is uncomfortable for most builders. The Apple model where you charge a premium for a unique product is hard to run in AI.

The categories where genuine product differentiation can be defended against a well-funded commodity provider are shrinking every quarter. The companies that look like winners today are spending huge capital to establish either a cost position or a data position before the window closes.

Amazon picked cost and data. It picked both seven years before anyone else thought to.

The question worth sitting with for the next 18 months is which of those two you can credibly build, and whether you can start before the window closes on you too.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

It searches the web. Reads a few pages. Searches again with a better query. Then it writes an answer.

Now disable web search and ask the same question. Or run it on the bare model through the API with no tools.

The same model gives you a confident answer from training data. Discontinued phones. Wrong prices. Nothing released after its cutoff.

Same model. Same prompt. Two different products.

What changed is the agent harness around the model.

In our previous pieces, we covered evals and traces. Those measure what an agent does. This piece is about what an agent actually is.

If you are building AI agents at work, or preparing for AI PM interviews, this is the model you need. We teach this in our course. (PMs at Microsoft, Coinbase, Indeed & 600+ PMs rated 4.9/ 5). See testimonials and course details — Extra 60% OFF - Use Code NYE26

The Model Is Not The Agent

The model is a function. Tokens in, tokens out. That is the entire contract.

The model can decide that a tool should be called. The model cannot run the tool. The model can decide a task is finished. The model cannot enforce that decision.

Search execution. Memory across sessions. Retries on failure. Loop termination. None of these happens inside the model. They happen in the agent harness.

What ChatGPT Actually Does When You Ask About Phones

The full agentic turn:

1. Goal received. The system records your question.

2. State check. Is there enough context to answer? No. Fresh data needed.

3. Pick a tool. The model emits a tool call: web search with a specific query.

4. Call the tool. The system runs the search and returns results.

5. Read the result. Results are fed back into the model as the next turn.

6. Decide if done. Not yet. The model emits another tool call. Browse a URL.

7. Loop. Steps three through six repeat until the model stops calling tools.

8. Stop and answer. The model emits a response with no tool call. The system exits the loop.

Steps three, six, and eight are the model. Pick the tool. Decide to continue. Decide to stop.

Every other step is code. The search has to run. Results have to be parsed and shrunk to fit context. History has to be tracked. The loop has to terminate even when the model wants to keep going.

That code is the agent harness.

Where The Loop Breaks

Five failures from products you have used:

1. The model picks the right tool but the wrong query. It searches “best phone 2026” instead of “best phone under 30000 INR 2026”.

2. The tool returns 50 results. The model loses the original constraint by the time it finishes reading them.

3. The model declares the task done. It is not. You asked for a phone under 30k. It recommended one at 45k.

4. The model decides it is not done. It is. Eight more searches. 90 seconds of waiting.

5. A tool errors out. The model has no idea why. It retries. Same error. Retries again.

Each failure maps to a specific agent harness layer.

• Failure 1: tool design. Force structured parameters, not free-text queries.

• Failure 2: context management. Return five results, not fifty.

• Failure 3: verification. Check the constraint before declaring done.

• Failure 4: stop condition. Cap turns.

• Failure 5: error handling. Classify errors, halt retry loops.

Better prompting will not solve any of these reliably. The fix sits in the harness.

The 6 Layers Of An Agent Harness

1. The Orchestration Loop

The while-loop above. Stop condition. Max turns. Tool-error behaviour. When to summarise older turns. When to spawn sub-agents.

ChatGPT caps how many searches per turn. Perplexity caps sources read. Both are product decisions, not model decisions.

A loop with no max-turn cap will burn your token budget.

2. Tool Definitions

ChatGPT’s web_search is a function. Whoever defined it decided the query format, the number of results, the fields per result, and the snippet length.

The model can pick any tool you give it. The model cannot redesign the tool.

A file-editing tool that returns just the diff scales to large codebases. One that returns the whole file runs out of context after three edits. Tool design decides what the agent can do.

3. Context Management

At some point the context window fills. Then you choose: truncate from the top, summarise older turns, keep a scratchpad, or spawn a sub-agent with a fresh context.

This decides whether your agent feels coherent at turn 30 or has forgotten the original question.

When ChatGPT says “as I mentioned earlier” about something you never discussed, the context layer failed.

4. Memory

Context lives for one task. Memory lives across tasks.

If a user comes back on Friday after talking to your agent on Monday, where is what they said stored? Vector store? Structured profile? Database keyed by user ID?

ChatGPT’s Memory feature is this layer. The base model has no memory between sessions. The harness added it.

5. Guardrails

Before any tool call executes, something has to inspect it.

Is this a write to production? Is this destructive? Did the model just try to email every customer?

When Cursor asks “Apply this change?” before editing your file, that is a guardrail. When ChatGPT Agent pauses before buying something, that is a guardrail.

Without this layer, one hallucinated tool call can do real damage.

6. Observability

Every turn, every tool call, every input, every output, every failure has to be logged. Not just for debugging. For evals.

If your agent fails in production and you cannot replay the trace, you do not have an agent. You have a black box that occasionally works.

What Product Managers Need To Decide

How agentic do you want this to be

A single tool call wrapped in a UI is not an agent. A 50-turn autonomous loop with no human in the path is the other extreme.

ChatGPT in regular mode is mildly agentic. ChatGPT in Agent mode is heavily agentic. Same model. Different harness. Different product.

How will you cap cost?

Token cost per task. Turns per task. Rupees per resolved query. Pick the unit. Track from day one.

What is your stop condition?

Is the model allowed to declare success on its own, or does another system verify the work?

In Cursor, the verifier is the test suite. In a support agent, a customer satisfaction signal. Without a verifier, the agent will tell you it is done when it is not.

Where is the human in the loop

For low-stakes tasks, the agent runs free. For high-stakes ones, it pauses for approval.

ChatGPT Agent pauses before buying. Cursor pauses before applying edits. Where those pauses sit is a product decision, not a model decision.

What do you log

If you log only the final answer, you cannot debug. Log every tool call, every model output, every reasoning step.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

Google Cloud is changing its strategy in a very big way.

They want to move beyond renting compute and storage. The goal is to become the only place enterprise AI runs.

But Why?

Because Alphabet has put more than half of its 2026 ML compute investment into Google Cloud.

Cloud is Alphabet’s smallest revenue line. It is also the only place inside the company where hundreds of billions of AI capex can be converted into uncapped enterprise revenue.

Let’s understand how using the North Star metric.

North Star Metric for Google Cloud = Annual Cloud Revenue.

Breaking Down The Metric

• Enterprise Customers: Companies running on Google Cloud.

• Workloads per Customer: Compute, storage, and databases each company runs.

• Revenue per Workload: Money Google makes per workload per month.

Two of three terms have a problem.

1. Enterprise sales is slow. AWS has the largest enterprises. Azure has Microsoft’s installed base. Sales cycles run 12 to 18 months. Hard to grow this term fast.

2. Revenue per Workload is under pressure. Compute and storage are commodities. Prices drop every year. Margins compress.

For years, this is why Google Cloud was treated as a side bet. The math did not work fast enough.

The AI Pivot

By adding Agents and Tokens, Google turns a linear business into an exponential one.

Now they have four levers to pull:

1. Getting more enterprise customers ( Lever 1 )

2. Increasing agents per customer ( Lever 2 )

3. Increasing tokens consumed per agent ( Lever 3 )

4. Driving down cost per million tokens ( Lever 4 )

Lever 1: Customers

Thomas Kurian has been rebuilding the sales motion since 2019. Vertical specialists. Financial services. Retail. Healthcare. Media.

Cloud Next 2026 was full of Fortune 500 logos. Citadel Securities. Deutsche Telekom. Home Depot. GE Appliances. Highmark Health.

Q4 2025 growth was 48% year over year. Fastest of the three big hyperscalers.

Lever 2: Agents per Customer

Old enterprise software had a ceiling. SaaS licenses tied to humans. You could not sell more seats than there are employees.

AI agents have no ceiling. GE Appliances is deploying 800 agents through Gemini Enterprise. Deutsche Telekom built MINDR, a multi-agent system that runs network operations on its own.

The Gemini Enterprise Agent Platform exists specifically to let one customer deploy thousands of agents inside Google Cloud. Each agent has a unique cryptographic ID and governance policies wired in. Once a company has 800 agents on the platform they cannot easily leave.

This is the new switching cost.

Lever 3: Tokens per Agent

Each agent burns tokens every time it does anything. Reasoning. Retrieval. Generation. Action.

The more capable the agent the more tokens it uses. A simple chatbot might use 10000 tokens per conversation. An autonomous coding agent might use 5 million for one task.

The number to watch is throughput. Google’s first-party APIs went from 10 billion tokens per minute last quarter to 16 billion this quarter. 60 percent quarterly growth on a metric that did not exist three years ago.

This is the one number that tells you if the strategy is working. Not revenue. Not market share. Token throughput growth.

Lever 4: Cost per Token

Google is driving cost per token DOWN. Compute prices got cut 8 percent across regions in Q1 2026. Google Cloud is now 5 to 10 percent cheaper than AWS or Azure for AI workloads.

Why cut prices? Because volume compounds faster than price drops. Price drops 8 percent. Volume grows 60 percent. Revenue still goes up.

Lower cost per token is what makes the TPU investment pay off. Google designs its own chips. No Nvidia margin paid on every inference call. Custom silicon brings the internal cost down. Some savings passed to customers. Customers run more workloads. Volume grows. Cycle compounds.

This is also why they spent 32 billion dollars on Wiz. Security is the trust layer that lets a CISO approve more agents. More agents means more tokens. The volume engine only runs if enterprises trust the stack.

In India, we have seen this with Jio. Jio dropped data prices to almost zero. Volume exploded. Other telecom operators died trying to match. Now Jio owns the market.

Google Cloud is playing the same game with AI. Drop the cost per token. Push volume. Lock customers in with agents that cannot be moved.

But there are two real risks.

1. Multi-cloud adoption is at 89 percent of enterprises. Companies are deliberately splitting AI workloads to avoid lock-in. If they refuse to consolidate on Google Cloud the volume game does not work.

2. The capex math is the second risk. Alphabet is putting 180 billion dollars of capex into 2026. TPU pods get obsoleted in about two years. Half the life of a traditional server. If AI adoption is real but slow Google ends up structurally over-built.

The strategy is forced not chosen. Cloud is the only place inside Alphabet where the AI capex math can possibly close. Search and YouTube have no real surface left for new monetisation.

If Google is right about exponential token growth and enterprise consolidation, Cloud quietly becomes the most important business inside Alphabet. If they are wrong on either, the integrated stack becomes the most expensive overbuild in tech history.

There is no middle outcome.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

Design a Safety Layer for an AI API?

Here is how a top 1% candidate answers this question from first principles.

Break down the question

Three loaded words.

1. Safety could mean preventing harmful outputs, preventing misuse, or preventing data leaks. Scope it.

2. Layer means a component within a larger system. Think about where it sits and what it does not own.

3. AI API means programmatic access. Not a chatbot. The users are developers. The attacks are automated. No human reviews each request.

Clarifying questions

1. What type of AI API?
   Text-only vs multimodal vs tool-use means completely different threat surfaces. A text API faces prompt injection. A multimodal API adds visual jailbreaks like harmful text hidden in images. A tool-use API adds real-world action risks. Assumption: multimodal with tool use. The hardest version.

2. Who are the consumers?
   Solo devs need defaults. Enterprises need configurable policies and compliance guarantees.
   Assumption: both.

3. Latency constraints?
   Every safety check adds latency. If the API serves real-time apps like voice assistants, 300ms of safety processing breaks the experience.
   Assumption: p50 under 50ms, p99 under 200ms.

Why does this matter?

Safety is the moat. Model capability is converging. GPT-4, Claude, and Gemini perform comparably. What differentiates an API provider is trust. Enterprises choose the API they trust not to embarrass them.

Unsafe APIs do not scale. At 100 developers, misuse is unlikely. At 100,000, misuse is daily. And the downside is asymmetric. One viral safety failure undoes years of brand equity.

User segments

1. End users are people using apps built on the API. They never see the safety layer. They bear the most harm when it fails. They have zero agency to protect themselves.

2. External developers are people calling the API. They need sensible defaults, configurable policies, and transparent error messages when requests get blocked.

3. Trust and Safety team are the internal operator. They need dashboards, investigation tools, and fast policy update workflows.

Prioritise end users. They bear the highest harm and have the least agency. They cannot adjust the safety layer. They cannot complain to you. If harmful content reaches them, they absorb the full impact with no recourse. But they never touch the API directly. Developers are the interface through which you protect them.

Design FOR end users. Design THROUGH developers.

Pain points

1. Pain point 1. Harmful content reaches end users. The model generates dangerous content even with benign inputs.
   A user asks, “How does aspirin work?” and the model includes lethal dosage info. No attack. No adversarial intent. The model just generated something it should not have.

2. Pain point 2. Adversaries bypass safety controls. Jailbreaking, prompt injection, and encoded attacks. API means programmatic means thousands of automated attacks per hour. Even excellent output classifiers get bypassed when the input is adversarial enough.

3. Pain point 3. Sensitive data leaks. PII from training data, system prompt extraction, session bleed across users.

Prioritise pain point 1. Three reasons.

1. Severity is highest. Direct psychological, physical, or legal harm to end users who have no recourse.

2. Frequency is highest. Harmful outputs occur even with non-adversarial inputs.

And solving PP1 partially solves PP2 and PP3. Even if a jailbreak succeeds and input controls fail, output controls still catch harmful content. PII in output is a subcategory of harmful output. The prioritised pain point creates a cascade.

First principle breakdown

This is where most candidates fail. They jump straight to let us add a content filter.

That is a feature answer. A systems answer starts by understanding what the system actually does.

Trace what happens when a developer calls an AI API. Step by step. From scratch.

1. Step 1. A developer sends a request. A user message. Maybe an image. Maybe a document to process.

2. Step 2. But the model does not just see that user message. The system assembles a full context window. That includes the developer’s system prompt (their proprietary instructions that shape the model’s behaviour), the conversation history from previous turns, any documents retrieved via RAG, and outputs from any tools the model previously used. All of these are stitched together into one big input. This is what the model actually sees.

3. Step 3. The model processes this assembled context and generates a response token by token.

4. Step 4. That response goes back to the developer, who passes it to their end user.

Now ask the first-principles question. At which of these steps can something go wrong?

Step 1 fails when the input itself is adversarial. A jailbreak attempt disguised as a normal query. A prompt injection hidden inside a document that the model is asked to summarise.

Step 2 fails when the assembled context contains data it should not. PII sitting inside a RAG document that nobody scanned. A previous conversation turn is slowly steering the model off course over many turns. This is the stage most candidates never think about. They conflate input with what the model sees. Those are two different things. The input is what the developer sends. The context is what the model processes.

Step 3 fails when the model generates harmful content from perfectly legit inputs. The model is a probabilistic system. It does not need to be attacked to produce something dangerous. A user asks about chemistry, and the model volunteers synthesis instructions. Nobody attacked the system.

Step 4 fails when the response contains harmful content, leaked PII, or the developer’s system prompt is reproduced verbatim.

Two more failure points sit outside this lifecycle.

Before step 1. Who is even calling this API? If you do not know the caller and what they are allowed to do, every downstream safety decision is flying blind.

After step 4. What patterns are emerging across thousands of requests?

So, from first principles, six stages where safety must operate.

1. Stage 0. Identity and access. Who is calling?

2. Stage 1. Input analysis. What are they sending? Is it adversarial?

3. Stage 2. Context assembly. What does the model actually see?

4. Stage 3. Model behaviour. What rules constrain the model?

5. Stage 4. Output evaluation. What is going on? Should it be blocked?

6. Stage 5. Post-response learning. What patterns emerged? How do we improve?

Solution

Now that the framework is derived, fill in each stage with the specific control and the reasoning behind it.

Stage 0

Tier access system.

• Tier 1 is any developer with maximum filter strictness.

• Tier 2 is verified developers with configurable filters after business verification and use-case declaration.

Stage 1

Two-pass input classification.

• Fast pass runs under 5ms on every request using pattern matching. Catches 70-80% of known attacks at near-zero latency.

• Slow pass is an ML classifier running in parallel with model inference. Not before it. So it does not add latency for genuine requests. If flagged after the model starts, the response is blocked before delivery.

Why two passes? One expensive classifier on every request is a latency added on all users.

Stage 2

Two controls.

• System prompt protection wraps every prompt in an immutable instruction plus output text-matching as a deterministic backup.

• PII scanning before inference prevents the model from ever processing data it should not see. Because once PII enters the context, even output redaction is insufficient since the model’s behaviour is already influenced.

Stage 3

Two-layer policy architecture.

• Layer 1 is immutable platform rules. No developer can disable them. Weapons, CSAM, terrorism, fraud. No use case justifies relaxation here.

• Layer 2 is developer-configurable for contextual harms. Three settings per category: strict, moderate, permissive.
  A medical app sets violence to moderate.
  A children’s app sets everything to strict.

Stage 4

Synchronous-asynchronous split. Synchronous blocks catastrophic harms, PII, and system prompt leaks in under 50ms. Uses a small, fast classifier trained specifically on catastrophic categories. Asynchronous flags contextual harms, bias, and hallucination after delivery.

Cross question: Recall or Precision?

High recall catches everything harmful but blocks legitimate requests. Developers lose trust. High precision only blocks when confident. Misses edge cases but maintains trust.

Optimise for precision in the synchronous classifier. A false positive permanently damages developer trust. Use the async pipeline to catch false negatives retroactively. High precision, real-time. High recall over time. Best of both.

Success metrics

1. False Negative Rate on Catastrophic Harms is the North Star. Harmful content reaching end users. Target under 0.01%.

2. False Positive Rate measures over-refusal. Target under 2%.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

More Resources

1. Product Management Mock Interview (Detailed)

2. Crack AI Business Roles (AI Management Consulting, AI Category Management, AI General Manager, Revenue Planning, etc.) - Course Details

3. Crack AI Program Manager Roles - Course Details

PM reads the first 50 and the last 20 and skims the rest. The middle 330 lines contain the business rules, edge cases, and constraint definitions that determine whether the feature works or does not.

Then sends it to engineering.

The engineering lead opens the file. He reads the overview section. Skims the requirements table.

Jumps to the technical constraints at the bottom. The 330 lines in the middle remain unread by a second person.

He hands it to the AI agent.

It implements exactly what the spec says, including the three contradictions between line 87 and line 312 that nobody caught because nobody read both lines.

Two weeks later, the feature ships with a critical defect. The PM blames the agent. Engineering blames the spec. The spec blames nobody because it is a markdown file, and markdown files do not defend themselves.

This is the most common failure mode in AI-native product development. And it did not start with AI.

The Problem That Existed Before Agents

Here is the uncomfortable truth. PMs have been skimming their own specs for years.

Markdown made specs easy to write. It did not make them easy to read.

A 400-line markdown file has no visual hierarchy beyond headers and bullets. No collapsible sections. No embedded mockups. No way to draw your eye to the three lines that matter most out of the 400.

Engineers asked clarifying questions. “What did you mean by this requirement?” “Does this apply to logged-out users too?” “This contradicts what you said in section 3.” Those questions caught the misalignments that skimming introduced.

Sprint reviews caught the rest. You shipped a version. It was wrong. You discussed it. You adjusted. The feedback loop was two weeks. The cost of misalignment was measured in sprints, not dollars.

AI agents removed every one of those safety nets.

The Spec-Driven Development Connection

We wrote about this problem in our article on Spec-Driven Development. The core argument was that a spec is a contract, not a document. It should be a behavioural specification that defines what the system must do, not a technical specification that prescribes how.

That argument stands. But it is incomplete.

A behavioural spec only works if someone reads it. A perfectly written contract that neither party reads is not a contract. It is paperwork.

Most PMs write behavioural specs in markdown. Those specs contain precise requirements. Constraint definitions. Edge case handling rules. Confidence thresholds. Fallback behaviours. All the things that separate a spec from a prayer.

Then nobody reads them.

The Solution: HTML as the Spec Layer?

AI agents generate HTML as easily as they generate Markdown. For the agent, the effort is identical. For the human, the difference is enormous.

An HTML spec can have collapsible sections. The PM sees high-level decisions first and drills into detail only where needed. Engineering sees the data model expanded with product context collapsed. Same document, different views, one handoff.

It can have colour-coded requirement statuses. Green for finalised. Yellow for needs-review. Red for placeholder. The PM sees at a glance which parts of the spec are done and which are still unfinished thoughts pretending to be requirements.

It can embed mockups inline. Not a link to Figma in another tab. A rendered visual sitting next to the requirement it represents. You see what you are specifying as you specify it.

It can use tabs to separate product context, behavioural requirements, technical constraints, and verification criteria. Each audience reads the tab that matters to them. Nothing gets lost crossing the handoff.

This is the Engagement-Quality Loop. Better readability leads to more engagement. More engagement leads to more edits. More edits lead to higher quality specs. Higher-quality specs lead to fewer implementation cycles. Fewer cycles mean lower compute cost.

The Costs and Limits

The token overhead is real. The ROI is better.

A markdown spec costs $0.03 to $0.10 in tokens. An HTML spec costs $0.10 to $0.40. Roughly 3 to 5x more.

But trace the cost through the chain. The markdown workflow leads to skimming, which leads to missed contradictions, which leads to rework cycles at $5 to $50 each. Two or three cycles, and you have spent $15 to $150 plus days of calendar time.

The HTML workflow costs $0.30 more upfront. You catch the contradiction before implementation. One cycle. $5 to $50. Done.

Less than 1% of tokens that most teams generate end up in production code. The rest goes into planning, iterating, and reworking.

The question is not whether the spec costs more tokens. The question is whether those tokens produce a spec someone actually reads.

The token overhead is noise. The rework is the signal.

When will markdown still win?

Short tasks where the output fits on one screen. Agent-to-agent handoffs where no human reads the document. Pure technical artefacts like type definitions and API schemas. If no human judgment is needed, use markdown. If human judgment is needed, use HTML.

A PM who does not spec is spending blindly. A PM who specs in a format nobody reads is spending with everyone’s eyes closed.

The Workflow

Five steps. Start tomorrow.

1. Spec in HTML. Prompt: “Create an HTML behavioural spec for [feature]. Collapsible sections. Colour-code requirements: green for finalised, yellow for review, red for placeholder. Embed mockups.”

2. Read your own spec. Open every section. Finalise every yellow. Remove every red. Do not send a spec with placeholder text to an agent. Every red line is a budget leak.

3. Identify decision nodes. Three to five points where human judgment matters more than agent capability. Business rules. Trade-offs. Prioritisation logic. These are the moments the PM earns their role.

4. Build micro-software for each node. Rule editors. Priority rankers. Trade-off sliders. Make decisions with full context. Cost: cents per tool. Value: precision instead of intuition.

5. Hand off clean. Fresh agent session. HTML spec as a single source of truth. One implementation cycle. Verify against the embedded checklist.

Total extra cost: $0.50 to $2.00 per feature. Total savings from eliminated rework: $10 to $200.

More Resources

1. AI PM Course - (PMs at Microsoft, Coinbase, Indeed & 600+ PMs rated 4.9/ 5)
   See testimonials and course details

2. Crack AI Business Roles (Consulting, Category Management, General Manager) - Course Details

3. Crack AI Program Manager Roles - Course Details

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

She got eliminated in Round 2.

Not by a trick question. Not by a culture fit screen.

By a single follow-up about how she would evaluate whether a RAG pipeline was retrieving the right documents.

She knew what RAG was. She could draw the architecture diagram.

But the interviewer did not want a diagram. He wanted to know what she would do when the retriever pulled the wrong documents at 2 am on a Tuesday and support tickets started spiking.

She did not have that answer.

Here is the part that should concern you. She was not underprepared. She was prepared for the wrong interview.

The AI PM interview in 2026 is a fundamentally different game. Most candidates are still running the old playbook.

CIRCLES. RICE. Feature prioritisation matrices. These are table stakes now.

Nobody gets hired for knowing them. You just get disqualified for not knowing them.

The real filter is AI depth.

I have spent two years shipping AI products in production. I have also helped 1000+ PMs prepare for AI roles. See testimonials

The pattern is impossible to miss. Five concepts keep showing up. Not ten. Not twenty. Five.

The dangerous part is not that PMs have never heard of them. Most have.

The dangerous part is that most PMs know these concepts at exactly the depth that gets them eliminated.

Let me show you what I mean.

1. RAG (Retrieval-Augmented Generation)

You probably already know what RAG is.

An LLM does not know everything. It hallucinates. It has no access to your company’s internal data. So you add a retrieval step. You search a vector database for relevant documents first. Then you feed those documents to the LLM as context. The LLM generates an answer grounded in actual information instead of its training data.

That explanation is correct. It is also the exact answer that gets you a polite nod followed by a harder follow-up.

RAG has a dirty secret. The retrieval step fails silently. The LLM does not tell you it received the wrong documents. It generates a confident, articulate, completely wrong answer. Your user has no idea. Your metrics might not catch it for weeks.

So the real question is not what RAG is. The real question is what you do when the R in RAG stops working.

• How do you measure retrieval quality separately from generation quality?

• When do you chunk documents into smaller pieces versus keeping them whole.

• What happens when the user query is ambiguous, and the retriever returns five documents that are each partially relevant, but none exactly right?

• What happens when you stuff too many documents into the context window and the LLM starts ignoring the important ones because of lost-in-the-middle effects?

These are the questions that separate PMs who have read about RAG from PMs who have debugged RAG in production.

Interviewers expect you to go deeper than this.

Sample Interview Questions

• Q1. You are building a customer support chatbot using RAG. Users report that 30% of answers are irrelevant. How would you diagnose whether the problem is in retrieval or generation?

• Q2. Your RAG system retrieves the correct document, but the LLM still produces an incorrect answer. What could be going wrong, and how would you fix it?

• Q3. A stakeholder wants to add RAG to a feature that currently uses a fine-tuned model. How would you evaluate whether this is the right architectural decision?

• Real AI PM Interview Questions (With Detailed Solution) Here

2. Evals (AI Evaluation)

Here is a question that sounds easy.

Your model accuracy improved from 84 % to 91 %. Should you ship it?

Answer it in your head right now.

If your instinct was yes, ship it; accuracy went up, you just failed the interview.

If your instinct was it depends, good. But the interview is only beginning. Because the next question depends on what. And that is where most candidates fall apart.

Most PMs treat evaluation as a checkpoint. Model hits a number. You ship. But AI evaluation is not a checkpoint. It is a continuous argument between what the model does well and what the business actually needs.

There are two worlds of evals, and most PMs only live in one.

Offline evals measure model performance before deployment. You run test datasets. You calculate precision, recall, and F1. You compare against baselines. This world feels safe. The numbers are clean. The comparisons are neat.

Then there is the second world. Online evals. What happens after deployment? User satisfaction. Task completion rates. Time to value. Edge cases your test data never imagined. The queries that real humans type at 11 pm on their phones look nothing like your curated evaluation dataset.

The gap between these two worlds is where AI products go to die.

A model can score 95% on your offline eval set and still make users miserable. Your eval set was built by engineers who write clean, well-structured queries. Your actual users write things like why is this broken and fix it and paste in screenshots of error messages.

The PM who wins the interview connects eval metrics to business outcomes. Not accuracy went up 7%. Instead. Accuracy went up 7%. Did user satisfaction improve? Did support tickets decrease? Did the revenue metric move? If you cannot draw that line from model metric to business metric, you will hear the worst four words in any interview. Let us move on.

Interviewers expect you to go deeper than this.

Sample Interview Questions

• Q1. You are the PM for an AI content moderation system. Precision is 97 percent but recall is 72%. The policy team wants a higher recall. The UX team is worried about false positives. How do you navigate this tradeoff?

• Q2. Design an evaluation framework for an AI feature that recommends products. What offline and online metrics would you track, and how would you decide when the model is ready to ship?

• Q3. Your A/B test shows the new model has 3% better accuracy but 15% higher latency. How do you make the ship or no-ship decision.

Watch Sample Video on Advanced Evals

3. Fine-Tuning vs Prompting

This is the concept where interviewers separate PMs who have shipped AI from PMs who have read blog posts about AI.

The question usually lands like this. Your AI feature is producing mediocre outputs. You have three options. Better prompts. Fine-tuning. A larger model. How do you decide?

Most candidates give a vague cost-benefit answer. Fine-tuning is more expensive but more accurate. Prompting is cheaper but limited. That answer is technically correct. It is also useless to the interviewer. Because they already know the textbook tradeoffs. What they want to hear is your decision framework. The actual sequence of steps you would follow.

Here is one that works.

Start with prompting. Always. Every single time. It is free to experiment with. You can iterate in hours, not weeks. A well-crafted prompt with good examples solves 80% of problems that PMs instinctively want to throw fine-tuning at.

But sometimes, prompting plateaus. You have tried ten prompt variations. You have added a few-shot examples. Quality is stuck at decent, and your users need excellent. Now you have a real decision to make.

Ask one question. Why is the model failing?

If the model is missing domain-specific knowledge or terminology, fine-tuning is probably the answer. A general model does not know your company’s product taxonomy. It does not understand your industry’s jargon. Fine-tuning teaches it patterns it has never seen.

If the model simply is not capable enough for the reasoning required, try a larger model first. Sometimes the problem is not knowledge. It is raw intelligence. A bigger model might handle the complexity without any fine-tuning at all.

Here is the layer that most PMs miss entirely. This is not just a technical decision. It is a product and business decision.

Fine-tuning means you now own a model. You need labelled training data to build it. You need ML engineers to maintain it. You need to retrain it when the underlying data distribution shifts. You have just taken on a recurring operational cost that compounds with every model update.

As a PM you need to justify that investment. What is the incremental quality gain? Is it large enough to warrant the ongoing maintenance burden? Could you get 70 % of the benefit with a smarter prompt and zero operational overhead?

That is the thinking interviewers want to hear. Not fine-tuning is better. But here is how I would make the decision and here is what I would measure to validate it was correct.

Interviewers expect you to go deeper than this.

Sample Interview Questions

• Q1. You are the PM for an AI writing assistant. Users complain that outputs feel generic and do not match their brand voice. Walk through how you would decide between prompt engineering, few-shot examples, and fine-tuning.

• Q2. Your team fine-tuned a model six months ago. Performance has degraded. What could be causing this, and what is your plan?

• Q3. A competitor just shipped a similar feature using GPT-4o. Your team uses a fine-tuned, smaller model that is cheaper but less capable. How do you think about this competitive dynamic?

• Real AI PM Interview Questions (With Detailed Solution) Here

4. Agents

Pay close attention to this one. If you are interviewing in the second half of 2026, agents will likely be the longest segment of your interview.

Every major tech company is building agent-based products right now. Not planning them. Building them. Shipping them. And they need PMs who understand the product challenges that agents create. Not the engineering challenges. The product challenges.

Here is the core distinction.

A regular LLM call is a one-shot interaction. You send a prompt. You get a response. Done.

An agent is a loop. It receives a goal. It breaks the goal into steps. It executes the first step. It observes the result. It decides what to do next. It keeps looping until the goal is achieved or it determines it cannot proceed.

That loop is where everything gets interesting.

Because an agent makes decisions. Autonomously. Without asking you. It might send an email on your behalf. It might delete a file it considers irrelevant. It might book a flight for the wrong date because it misinterpreted what next Friday means in context.

The technology question is how agents work. Any PM can read a LangChain tutorial and answer that. The product question is much harder and much more valuable in an interview.

• How much autonomy should the agent have?

• Where do you insert human checkpoints?

• When should it ask for permission versus making a judgment call on its own.

• How do you design an experience where the user feels in control even though the agent is doing all the work?

Here is the tension that makes this concept so rich for interviews. Users want agents to be autonomous. That is the entire value proposition. Do this for me so I do not have to think about it. But users also want to feel safe. They want confidence that the agent will not do something catastrophic or irreversible.

Speed and safety pull in opposite directions. The PM who can navigate that tension with clear product principles will stand out in every single interview.

The most common mistake candidates make is treating agents as a pure engineering conversation. Talking about function calling, tool schemas and ReAct patterns. That is the implementation layer. Interviewers want the product layer. The experience design. The trust model. The failure recovery flow.

Sample Interview Questions

• Q1. You are building an AI agent that helps users book travel. The agent can search flights, compare prices, and make reservations. Design the user experience for when the agent books the wrong dates.

• Q2. Your AI agent completes tasks autonomously, but users report feeling out of the loop. How would you redesign the experience to build trust without slowing the agent down?

• Q3. An agent-based feature has a 78 per cent task completion rate. Users love it when it works, but are frustrated when it fails. How do you decide whether to ship broadly or keep it in limited access?

5. Guardrails

Here is a question most PMs never think about until it is too late.

What is the worst thing your AI product could do?

Not the most likely failure. The worst. The one that ends up as a screenshot on social media within hours. The one that gets your CEO pulled into a meeting with Legal at 7 am. The one that makes users question whether they should trust anything your product says ever again.

If you have shipped AI in production, you already know this feeling. Because it either already happened to you or you watched it happen to a competitor and thought that could have been us.

Guardrails are how you prevent those moments. They are the systems you build to stop your AI from generating harmful content, leaking private data, producing confidently wrong information, going off topic, or being manipulated by adversarial users who know exactly how to break your system.

Here is why this topic is deceptively difficult.

The naive approach to guardrails is to block everything that looks risky. Restrict the model outputs aggressively. Add filters on every response. Flag anything that seems remotely problematic.

That approach solves the safety problem. It also kills the product. Users start hitting walls on legitimate queries. The AI becomes so cautious it is useless. You have traded one crisis for another. Instead of a harmful output going viral, you have a product that nobody wants to use because it blocks everything.

The real challenge is surgical precision. Block the dangerous outputs. Let the legitimate ones through. Do it fast enough that the user never notices a filter is running behind the scenes.

The layered approach works. Input filters catch bad requests before they reach the model. System prompts constrain the model behavior at the instruction level. Output filters catch problematic responses before they reach the user. Human review handles the edge cases that automated systems miss.

But the product question is always the same. Where do you draw the line.

A guardrail that is too strict blocks 15 percent of legitimate queries and your users leave for a competitor. A guardrail that is too loose lets one harmful response through and your product is trending for the wrong reasons.

That calibration problem is the entire interview. Not what are guardrails. But how do you set them. How do you measure whether they are working. And how do you adjust them when you get it wrong.

Interviewers expect you to go deeper than this.

Sample Interview Questions

Q1. You are launching an AI chatbot for a healthcare company. What guardrails would you implement and how would you prioritise them given a tight launch timeline.

Q2. Users have discovered they can manipulate your AI assistant through indirect prompt injection via pasted text. How do you approach this as a product problem, not just an engineering problem.

Q3. Your guardrails are blocking 12 percent of legitimate user queries. Engineering says tightening the filters further will increase false positives to 18 percent. How do you handle this.

The Pattern You Need to See

Now here is the honest question.

Go back to the 15 sample interview questions in this article. Read them carefully. For each one, ask yourself whether you could give a structured and confident answer that would satisfy a senior interviewer at a top tech company.

• If you could answer 12 or more with real depth, you are in strong shape. Keep sharpening.

• If you could answer 8 to 11, you have gaps. Targeted gaps. The kind that a focused two-week sprint could close.

• If you could answer fewer than 8, the problem is not intelligence. You are smart enough to be reading this article. The problem is exposure.

Nobody has shown you how these concepts actually play out in real product decisions. You have been learning definitions when you should have been practising trade-offs.

That gap is exactly why we built the AI PM course. 800+ PMs have taken it. 4.9 out of 5 rating

The interview has changed. Your preparation should too.

Become an AI Product Builder | 100 Hrs+ Learning

This is not a course you watch passively. It is a program you go through with a cohort of other PMs. You get office hours. You get demo sessions.

The AI Product Manager Builder 2.0 is a 12-week cohort program. 45 plus classes. Hands-on demos. Interview prep sessions every week. A capstone project. 10+ real-world projects you can add to your portfolio.

Apply for Cohort

About Author

Shailesh Sharma - I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass (Here)

I built technomanagers.in in one day.

Full product. Frontend, backend, course catalogue, question bank, deployment. One day.

Link — https://technomanagers.in/

I am not an engineer. I am a product manager.

This is what a building looks like in 2026.

The tools have changed so fundamentally that the gap between idea and shipped product is no longer months. It is days. Sometimes hours.

In 2026, the PM who gets hired is the one who builds. Not manages. Builds.

This article walks through the full pipeline.

Problem discovery to launch. Every step uses tools that did not exist 18 months ago.

If you are a PM, engineer, or builder trying to ship an AI product this year, this is the roadmap.

Step 1. Spot the problem. Validate it fast.

Most AI products fail at step zero. They start with a technology and go looking for a problem. We should build something with RAG. We should make an agent. Backwards.

Start with the problem. But how you validate has changed.

Pick a domain you understand. Not AI for healthcare. Something specific.

Radiologists spend 40 minutes per scan writing structured reports. Observable. Measurable. Worth solving.

Now, validate it in 30 minutes. Open Claude. Run a research session.

Sample prompt:

I am validating a product idea. The problem is that radiologists spend 40 minutes per scan writing structured reports. I need you to do the following. First, find 5 existing products that solve this problem today. Second, find evidence from radiology forums or medical publications that this pain point is real and widespread. Third, estimate the addressable market size using publicly available data on the number of radiologists and scans per year. Fourth, list three reasons this problem might not be worth solving.

That last line matters. You are not looking for confirmation. You are stress-testing the hypothesis.

The output of this step is a one-page problem brief in markdown.

Four sections.

→Problem statement.
→ Evidence it exists.
→ Who has it?
→ What they do today.

If you cannot fill that page with real evidence, move on.

Step 2. Competitor research with agents

This is where the 2026 flow diverges most from the old one.

Traditional competitor research meant 15 browser tabs, free trial signups, and a comparison spreadsheet built over two days.

Now you build a lightweight agent that does the data collection in hours.

Sample prompt for Claude with web search:

I am building an AI product for radiology report generation. Research the following competitors: Nuance DAX, Rad AI, DeepScribe, and Ambra Health. For each one extract the following. Target customer segment. Core AI capability. Pricing model if publicly available. Key differentiator. Weaknesses mentioned in user reviews. Present this as a structured comparison table.

The agent does not replace your analysis. It replaces the manual collection. You still look at the output and ask the hard questions.

Where are the gaps? Where are competitors overserving? Where is there a segment nobody is building for?

Save the output as competitor-matrix.md. It lives alongside your problem brief. Both are markdown. Both feed into the next step.

If you want to learn how to build these agent workflows from scratch, the Technomanagers AI PM course covers everything. 800+ students. 4.9 out of 5 rating.

Step 3. Talk to users. AI does not replace this.

You have a validated problem and a competitor landscape. Now talk to actual humans.

No agent replaces a 30-minute conversation with someone who has the problem you are solving. But how you prepare and synthesise has changed.

Before the call, generate a targeted research script.

Sample prompt:

Here is my problem brief [paste problem-brief.md]. Here is my competitor matrix [paste competitor-matrix.md]. Generate 10 user interview questions that specifically probe the gaps I identified in the competitor landscape. Focus on workflow pain points, current workarounds, and willingness to pay. Avoid generic questions.

After five conversations, paste all your notes into Claude.

Sample prompt:

Here are my notes from 5 user interviews about radiology report generation [paste notes]. Extract the following. Top 3 recurring pain points ranked by frequency. Contradictions between what users say they want and what they actually do. Unmet needs that no current competitor addresses. Willingness to pay signals.

The key discipline is triangulation. User interviews say one thing. Competitor gaps say another. Usage data says a third. The truth is in the overlap.

Step 4. Write the spec in markdown

This step separates the 2026 builder from the traditional PM.

You do not write a 20-page PRD in Google Docs. You write a product spec in markdown. In Claude.

A markdown spec is machine-readable. It can be fed directly into Claude or Cursor to generate functional code. A Google Doc cannot.

The format of your spec determines the speed of your prototype. This is an architectural decision, not a style preference.

The spec has five sections.

1. Section 1. Problem and user.
   One paragraph pulled from your problem brief.

2. Section 2. Core workflow.
   The 3 to 5 steps the user takes to get value. Not features. Steps. User uploads a scan. System extracts findings. The system generates a structured report. User reviews and edits. System learns from edits.

3. Section 3. Technical architecture.
   What model. What retrieval strategy? Input and output formats. Where the data lives. If you cannot write this section, you do not understand your own product.

4. Section 4. Eval criteria.
   How will you know if it works? Precision. Recall. Latency. Hallucination rate. Defined before you build.

5. Section 5. Out of scope.
   What you are deliberately not building in v1. This section saves more time than any other.

Save it as product-spec.md.

Step 5. Build the prototype in Claude

Take your product-spec.md. Open Claude. Paste the entire spec.

Sample prompt:

Here is my product spec [paste product-spec.md]. Build a functional prototype of this application. Use React for the frontend. Use Python with FastAPI for the backend. For the RAG component, use a vector database with cosine similarity search. Generate the full codebase, including file structure, all components, API endpoints, and database schema. Make it deployable.

Claude generates a working application from a well-written spec. Not a mockup. Not a wireframe. A working product.

The quality of the prototype is a direct function of the quality of your spec. Vague spec, vague output. Precise spec, precise output.

This is where the .md format pays off.

Then you iterate.

Sample follow-up prompts:

1. Change the retrieval to a hybrid search combining keyword and semantic matching.

2. Add error handling for cases where the API returns empty results or the context window is exceeded.

3. Refactor the output to match this JSON schema [paste schema].

Each iteration takes minutes. Within a few hours, you have something you can put in front of users. Not a deck. A working product.

Step 6. Build evals before you launch

Most AI products ship without evals. Single biggest mistake in AI product development.

Evals answer one question. Is the AI actually working?

1. For a RAG product, your metrics are Precision at K, Recall at K, and MRR.

2. For generative output, hallucination rate, relevance, and coherence.

3. For an agent, the task completion rate, step accuracy, and cost per task.

Define before launch. Automate. Set thresholds below which the product does not ship.

Sample prompt:

I have a RAG-based radiology report generator. Generate a Python evaluation script that tests the following. Precision at 5 for retrieved context chunks. Recall at 5 for relevant medical findings. Mean Reciprocal Rank for the top result. Hallucination detection by comparing generated text against source documents. Use a test set of 20 sample queries with known correct answers that I will provide.

This separates a demo from a product.

Step 7. Production hardening

Prototype works. Evals pass. Now harden it.

Latency

If your pipeline takes 8 seconds, users leave. Target under 2 seconds. Optimise chunking, cache frequent queries, and pick the right model size.

Cost

Every API call costs money. A prototype at 2 dollars per session is a burn rate, not a product. Find where smaller models work, where you can cache, and where you can batch.

Error handling

What happens when the model returns garbage? When retrieval finds nothing. When the API goes down, every failure mode needs a graceful fallback.

Monitoring. Log inputs, outputs, latencies, costs. Build dashboards that catch quality degradation before users notice.

Can you actually do all of this?

Each step is learnable. Each step uses tools available right now.

If you want to go from reading to shipping, the AI Product Builder Cohort is built for exactly this pipeline.

12 Weeks | 45+ Sessions | 100Hrs of Learning | AI Interview Prep | Portfolio Ready | Direct mentor access. Applications open.

If you prefer self-paced, the Technomanagers AI PM course covers every concept here. 800+students. 4.9 rating. It is the foundation the cohort builds on → See testimonials and course details

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

1. 90 Days Roadmap to become an AI Product Manager

2. Advanced Evals - Evals in RAG

You own roughly 80 pieces of clothing. You remember maybe 15. You rotate through 7.

The problem is not the clothes. The problem is that your closet has no search bar.

Google Photos just shipped a feature that fixes this. It scans your photo library, finds every piece of clothing you have ever worn in a picture, catalogues it, and turns your gallery into a searchable digital closet. Filter by category. Mix and match pieces into outfits. Try the whole thing on virtually before you get dressed.

That is the product. Now, let us build it.

As an AI Product Manager, you would get such problem statements.

In our previous pieces, we covered how TikTok uses session-based RNNs and why position bias destroys your ranking quality. This one is about a different kind of AI system.

Not one model does one job. Five models chained together, where every failure cascades downstream.

If you are preparing for AI PM interviews, we will prepare you with Real AI PM Interview Questions. We cover this more in our course.

The Problem Statement

Given a user’s unstructured photo library, build a system that:

1. Identifies every unique garment the user has worn in any photo

2. Creates a clean catalogue with one entry per garment

3. Let’s users combine items into outfits

4. Shows users how the outfit looks on their body

The input is chaos. Vacation photos, selfies, group shots, screenshots, food pictures, photos where you are partially visible, and photos where your jacket covers your shirt.

The output must be a clean wardrobe.

This is not one model. It is five models chained together.

Stage 1: Find the Photos That Matter

A typical library has 1000 images. Maybe 400 contain you wearing visible clothing. The system needs to find those 400 and discard the rest.

Google Photos already detects and groups faces through its People feature. That tells the system which photos contain you. But face detection is not enough. You need body pose estimation.

The system runs a pose estimator to map body joints. Shoulders, elbows, wrists, hips, knees, ankles. These key points answer two questions.

1. How much of your body is visible?

2. And where does each garment zone fall in the image?

The region between the shoulders and hips is a top. Between hips and the ankles is the bottom.

If only your face and shoulders are in frame, there is no point in trying to extract pants.

The PM decision here is the visibility threshold.

• How many keypoints must be detected for the photo to enter the pipeline?

• Too strict and you miss casual selfies that still show a good shirt.

• Too lenient and you flood every downstream stage with blurry, occluded garbage.

Stage 2: Segment Each Garment

Each photo might contain multiple garments. A shirt, a jacket, pants, and earrings. The system needs to isolate each item at the pixel level.

This is instance segmentation.

Not “there is clothing in this image” but “these exact pixels belong to this shirt, and those exact pixels belong to that jacket.”

The model takes an image and outputs bounding boxes and pixel masks, each labelled with a garment category.

Here is where the first real complexity shows up. Occlusion.

The naive answer is to only extract fully visible items. This fails.

If you always wear a jacket over a particular shirt in photos, that shirt never enters your wardrobe.

The better approach is to extract all detected garments and attach a visibility score.

Visibility Score = Visible Pixels / Estimated Total Pixels

A shirt with 80% visible scores 0.8. A shirt 30% hidden behind a jacket scores 0.3. This score determines which photo gets chosen as the representative thumbnail later.

Stage 3: The Hardest Problem in the Pipeline

You now have roughly 1,000 segmented garment patches across 400 photos. Many are the same physical item photographed in different conditions. Your favourite blue shirt appears in 40 photos. Different lighting. Different wrinkles. Different backgrounds. Different angles.

The system needs to know that these are all the same shirt.

This is a visual re-identification problem. The same class of problem that security systems use to track a person across multiple cameras.

The naive approach is pixel-level image similarity. Compare the raw pixels of two garment patches. This fails immediately.

The same white shirt photographed indoors under warm lighting looks golden. Outdoors, it looks blue-white. Against a dark background, it appears brighter. Pixel similarity would call these three different shirts. They are the same shirt.

You pass each segmented garment through a feature extraction network.

A CNN or Vision Transformer fine-tuned on fashion datasets like DeepFashion. The network outputs a compact vector, say 256 dimensions, that captures the garment’s identity. Its colour, texture, pattern, cut, and structure. Not the lighting. Not the background. Not the wrinkles.

Similarity(A, B) = cosine(Embedding(A), Embedding(B))

If cosine similarity exceeds a threshold, say 0.85, the system treats them as the same garment.

This threshold is the single most important PM decision in the entire pipeline.

Set it too high at 0.95, and the system creates duplicates. Your blue shirt appears four times because slightly different photos produced slightly different embeddings.

Set it too low at 0.70, and the system merges two genuinely different items into one. Your navy polo and your navy crew-neck collapse into a single entry.

Which mistake is more tolerable?

For a consumer product, false merges are worse. Showing two entries for one shirt is a mild annoyance. Deleting a unique garment by merging it with another item is data loss. You cannot undo it without rerunning the pipeline.

Bias the threshold higher. Accept some duplicates. Give users a manual merge option in the UI.

Stage 4: Cluster and Build the Catalogue

With 1000 embeddings, the system clusters them. Each cluster represents one unique garment. DBSCAN works well here because it does not require specifying the number of clusters in advance. It finds natural groupings based on embedding distances.

From each cluster, select a representative thumbnail. Highest visibility score. Highest resolution. Best lighting.

But a raw crop from a photo still has a messy background. Your shirt thumbnail would show a slice of a restaurant behind you.

The system takes the segmented garment mask and runs an inpainting model. It generates a clean thumbnail on a neutral background, fills in any occluded parts of the garment, and removes everything else.

The output: a clean, catalogue-style image of each garment. That is what you see in the Wardrobe UI.

Stage 5: Classification

Each garment needs a category label. Tops, bottoms, dresses, outerwear, skirts, jewellery, and footwear.

The same feature extraction network from Stage 3 can feed into a classification head. Standard multi-class classification.

The design question that matters here is whether classification should run before or after clustering.

If you classify first, the category becomes a hard constraint during clustering. A shirt and a jacket can never merge, regardless of how similar their embeddings are. This eliminates absurd false merges. But classification errors now propagate. If a blazer gets mislabelled as a shirt, it will never match with other photos of that blazer correctly labelled as outerwear.

If you run classification and clustering in parallel, you can use category agreement as a soft constraint. Same category plus high embedding similarity means high-confidence merge. Different categories, plus high embedding similarity, mean flagged at a higher threshold.

The parallel approach is more robust. It lets one model compensate for the other’s mistakes.

The Virtual Try-On

You select a top and a bottom from your wardrobe. You tap Try it on.

The system generates a photorealistic image of you wearing that combination.

Google did not build this for Photos. This technology already existed in Google Shopping, where it worked on billions of product listings. The PM reused it.

The underlying system is a diffusion-based image generation model built specifically for fashion. Here is how it works.

The model takes three inputs.

1. A photo of you.

2. A garment image.

3. A body pose estimate from Stage 1.

First, 2D warping. The garment pixels get mapped onto the region of your body where that garment would sit. The pose estimate locates your shoulders, torso, and hips. The segmentation mask provides shape and texture.

But simple warping produces artefacts. Sleeves do not match arm positions. Fabric does not fold correctly around your body.

The diffusion model takes over. It receives the warped image plus a segmentation map of the missing regions and generates the final output. Realistic fabric folds, shadows, and draping.

The critical constraint: the model preserves the exact texture, colour, and pattern of your actual garment. It only generates the physics of how that fabric interacts with your body. This is not “put a blue shirt on this person.” This is “take this specific shirt with this specific weave and show how it drapes on this specific body in this specific pose.”

This is why the output looks like your actual clothes. Not generic AI-generated clothing.

Outfit Compatibility

There is one more model that the press release does not mention, but the product requires.

When a user mixes a top with a bottom, the system should score whether the combination works visually. This is a compatibility scoring problem.

The approach: train a model on labelled fashion datasets where outfit combinations are rated as compatible or incompatible. Each item has an embedding from Stage 3. The compatibility model takes two or more garment embeddings and outputs a score between 0 and 1.

Compatibility(Top, Bottom) = sigmoid(W · concat(Embedding_top, Embedding_bottom) + b)

A navy blazer with khaki chinos might score 0.9. A navy blazer with basketball shorts might score 0.2.

The PM decision is whether to surface this score or use it passively.

Surfacing it as this outfit scores 4 out of 5 risks being wrong and annoying. Using it passively to sort moodboard suggestions by compatibility adds value without overcommitting.

The safe answer is passive integration. Let the user feel the algorithm without seeing it.

What Metrics will you track?

Most PMs would track whether users open the Wardrobe tab. That tells you almost nothing. Here is the framework that actually measures success.

Does the user come back?

Wardrobe retention at Day 7 and Day 30. Not Google Photos retention. Wardrobe-specific retention. A user who opens the Wardrobe tab once and never returns means the catalogue was not useful. A user who returns weekly is planning outfits. That is the behaviour you want.

Track the ratio of outfit creations to wardrobe visits. If users open the wardrobe but never create an outfit, the catalogue is interesting, but the mix-and-match experience is not compelling. Target at least 30 per cent of wardrobe visits resulting in an outfit action (create, save, or share).

Does Try It On convert?

Try It On usage rate among outfit creators. If users create outfits but never tap Try It On, the feature is either hidden or not trusted. Target at least 40 per cent of outfit creators using Try It On at least once.

Try It On completion rate. Does the user look at the generated image for more than 3 seconds? Do they save it or share it? If they generate a try-on and immediately dismiss it, the output quality is not meeting expectations.

Try It On repeat rate. Users who try it once and never again do not trust the result. Users who try it multiple times per session are engaged. A healthy repeat rate means the diffusion model is producing outputs that feel real.

If this breakdown changed how you think about multi-model pipelines, cascade error budgets, and AI feature design, you will find much more depth in our AI PM course. We cover RAG architectures, evaluation frameworks, and real interview questions from top companies.

Check our highest-rated AI PM course (Including AI PM Interview Preparation) · 4.9/5 · 600+ enrollments → See testimonials and course details

About Author

Shailesh Sharma - I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass (Here)

Subscribe to get the FREE Book (AI & Tech Simplified), Link in Welcome Email

Your model hears one. The user lived the other.

They are not the same story. And the gap between them is destroying your feed.

We are going to show you both versions, side by side, for every stage of the recommendation pipeline. By the time the two stories merge, you will understand position bias deeper than any offline metric can show you.

In our previous pieces, we covered how TikTok uses session-based RNNs and why recommenders suffer from catastrophic forgetting. This one is about a different failure. Your model remembers clicks. But it misremembers why they happened.

If you are preparing for AI PM interviews, position bias comes up constantly in recommendation system design rounds. We cover this and more in our course.

The Click

What your model recorded

User opened the app store. Saw App X at position 1. Clicked. Label = 1. App X is relevant.

What actually happened

User opened the app on the metro. Had 40 seconds before their stop. App X was the first thing on screen. They did not scroll. Did not compare. Saw one thing. Tapped it.

If App Y had been at position 1, they would have tapped App Y.

The click had nothing to do with App X. It had everything to do with position 1.

The Data

What your model sees:

Position 1 CTR is 0.25. Position 5 is 0.10. Position 15 is 0.03. Items at position 1 are 8x better than items at position 15.

What is actually true

95% of users see position 1. Maybe 60% reach position 5. Fewer than 20% get to position 15.

A great app at position 15 gets 0.03 CTR because nobody saw it. A mediocre app at position 1 gets 0.25 because everyone saw it.

The model has no way to tell these apart. It calls both numbers a preference.

The Loop

What your model believes

It trains weekly. Every cycle, the data confirms that items at the top are the best items. Their CTR is highest. Their scores go up. They stay at the top. The system is stable. The system is working.

What is actually happening?

Top items get more clicks because they are at the top. Those clicks inflate their CTR. Inflated CTR keeps them at the top. Next week, same thing.

A genuinely excellent app debuted at position 12 three weeks ago. CTR of 0.04. Not because users disliked it. Because most users never scrolled that far. The model scored it low. It dropped to 16. Then it disappeared.

Your feed is not ranking by quality. It is ranked by inertia.

This has two costs.

1. First, diversity dies. The same items win every cycle. New items cannot break through. Your feed feels stale. Engagement decays.

2. Second, revenue leaks. Your ranking function is f(CTR, bid). If CTR is inflated by position, you are overvaluing items that sit high and undervaluing items that bid high. That is money lost daily.

The Usual Fix (And Why It Fails)

Every team eventually notices something is off. The standard fix is simple. Add position as a feature.

The model sees [user features, item features, context, position]. It learns that clicks at position 1 should be discounted. Sounds reasonable.

What the team believes this achieves

The model now accounts for position. Bias is handled.

What actually happens at inference time?

The model needs a position value to produce a score. But the position has not been decided yet. That is what the ranking is supposed to determine.

You cannot feed a position as input when the position is the output.

So the team picks a default. Position 1 for all items. Or position 5. Or position 9.

They try position 1. They get Ranking A. They try position 5. They get Ranking B. Completely different. Different items in the top 5. Different user experience.

The ranking depends entirely on a number someone picked arbitrarily.

You are now running AB tests to find the best magic number. And the best number for one scenario does not transfer to another. The approach is a dead end.

The Question That Closes the Gap

Here is the question that resolves the two stories into one.

When a user clicks, what are they actually telling you?

Two things. Fused into a single signal.

First: I saw this item. This depends on the position. Position 1 is almost always seen. Position 20 has maybe a 10 per cent chance. This has nothing to do with the item.

Second: I wanted this item. This depends on the user, the item, and the context. This has nothing to do with position.

Every click is the product of these two.

P(click) = P(saw it) x P(wanted it, given I saw it)

How to solve this?

We can build two modules. Trains them together. Deploys them apart.

Module 1 is ProbSeen.

• One input: position.

• One output: the probability the user saw the item at that position. Think of it as a small curve. Position 1 outputs 0.95. Position 20 outputs 0.12.

Module 2 is pCTR.

• Inputs: user profile, item features, context.

• Output: the probability the user would click if they had seen it.

Position never enters pCTR.

During training, predicted click = ProbSeen x pCTR. This is compared against the actual click label. Standard cross-entropy loss.

Here is what makes it work. Both modules share the same loss. They train jointly. Gradients flow through both.

When the model sees that position 1 items get clicked more, the shared gradient forces a split. How much of that signal is visibility? ProbSeen takes it. How much is genuine preference? pCTR takes it.

Neither module can steal the other’s signal. Both are accountable for the same loss. The separation is automatic.

Why not train them separately? Because separate losses mean separate objectives. ProbSeen might absorb preference. pCTR might absorb position. The boundaries blur. Joint training forces a clean separation through coupled gradients.

At inference time, you throw away ProbSeen. You deploy only pCTR.

No position input needed. No default value. No magic number.

What your model now records: This user would click this item, regardless of where it is shown.

What actually happened: Same thing.

The two stories are finally one.

If this article changed how you think about position bias, CTR modelling, and ranking quality, you will find much more depth in our AI PM course Case Studies. (42+ Videos & 25+ Case Studies)

Check our highest-rated AI PM course (Including AI PM Interview Preparation) · 4.9/5 · 600+ enrollments → See testimonials and course details

About Author

Shailesh Sharma - I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass (Here)

The average Product Manager spends 40% of their week writing Jira tickets, updating roadmaps, and arguing over edge cases in 15-page Product Requirements Documents (PRDs).

At elite AI labs like Anthropic, Senior PMs spend exactly 0% of their week doing this.

Instead, a PM has an idea. They write a concise, 3-paragraph Product Note.

They drop it into an automated agentic workflow. 45 minutes later, there is a functional, tested Pull Request (PR) waiting in GitHub for engineering review.

No refinement meetings. No 15-page PRDs. No six-week development cycles.

This is not a sci-fi prediction for 2030. This is happening right now.

It is called the Execution Collapse — the cost and time of turning a product thought into production code has effectively dropped to zero.

To survive the next wave of tech, you have to become an “Orchestrator.”

Orchestrators don’t write PRDs. They write context.md files.

90-Day Plan: Become an AI PM

The PM Workflow of 2026

Step 1: The Product Note (The Seed)

You no longer write a PRD. You write a “Product Note.”

This is a raw, 3-to-4 paragraph summary of the user intent, the desired outcome, and the specific metrics you want to move.

It is pure strategy, stripped of any implementation details.

Step 2: The Injection (context.md)

This is the secret weapon. The PM takes the Product Note and feeds it into an orchestrating LLM, but they inject two critical system files alongside it to constrain the AI’s hallucinations:

product_area_context.md: Maintained strictly by the PM. This file defines the rigid business rules.

• Example Content: Free users can only generate 5 reports per day. Do not allow PDF exports for Free Tier. If a user hits a paywall, route them to /upgrade. Our tone is professional, never conversational.”

code_context.md: Maintained by the engineering lead. This file maps the current technical reality.

• Example Content: “We use React for the frontend and Python/FastAPI for the backend. All user data must pass through the auth_v2 middleware. Our database schema for users is located in /db/schema/users.sql.”

Step 3: The Functional Spec & The PM Review (The New Hero Skill)

The Orchestrator LLM synthesises the Product Note with the strict constraints of the Context files.

It instantly generates a highly technical Functional Spec.

This is the new job of the Product Manager. You don’t write the spec from scratch; you evaluate it.

You act as the Editor-in-Chief.

You review the AI’s logic, check for edge cases it missed, verify it adhered to the product_area_context.md rules, and adjust its assumptions. You are the taste-maker and the final human in the loop.

Once you approve it, you hit “Proceed.”

Step 4: Tech Spec to Autonomous PR

Once the PM approves the Functional Spec, the workflow becomes fully autonomous.

1. The agent converts the Functional Spec into a Tech Spec (defining architecture and data models).

2. The agent hands the Tech Spec to a coding model (like Claude 4.6).

3. The coding model writes the actual code, runs the unit tests, and automatically raises a Pull Request in GitHub.

Total time elapsed: 45 minutes. — -

AI PM 2026 — Winner’s Playbook

The Terrifying Future of Product Management

Read that workflow again. At no point did the PM schedule a backlog refinement meeting.

At no point did they write a user story in Jira.

The Execution Collapse means that engineering execution is rapidly becoming a commodity.

In this new reality, companies don’t need 50 Product Managers to coordinate sprints.

They need 5 elite PMs who understand how to structure context.md files, evaluate AI logic, and orchestrate autonomous agents.

If you don’t understand how to build these pipelines, you are fighting a losing battle against a PM who does.

FREE Book Giveaway — AI & Tech Simplified

Stop Coordinating. Start Orchestrating.

Understanding that this is the future is just a theory.

Actually building these agentic workflows for your own product is how you survive the transition.

You cannot learn this by just reading Medium articles. You have to build it.

If this article changed how you think about Product Management in the AI Era, you will find much more depth in our AI PM course.

Check our highest-rated AI PM course (Including AI PM Interview Preparation) · 4.9/5 · 600+ enrollments → See testimonials and course details

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

More Resouces

1. Crack AI Business Roles ( Consultants, Revenue Planning, Category Manager, General Manager )

2. Crack AI Program Manager Roles

AI Model Selection has become a very critical skill and getting asked in a Lot of Interviews

Imagine your CEO asks you to build an AI customer support agent for a food delivery app that handles 2 million tickets a month.

You think that Just use the best model. GPT-5. Claude Opus. Gemini 3, wire it up, and ship it.

You do the math, and you realise the ROI will not make sense.

Your agent will feel smart. Your margin will turn negative. Your CFO will ask why AI inference is eating half the savings you promised.

This is the first-principles breakdown of how AI PMs should actually make model-choice decisions. We will walk through it using one real scenario.

Building an AI customer support agent for a food delivery app at the Swiggy or DoorDash scale.

If you are preparing for AI PM interviews, this is one of the questions that now separates real AI PMs from people who have only used ChatGPT. You can check out other AI PM Interview Questions here.

The Scenario

You are the AI PM at a food delivery company.

You have 2 million support tickets a month. Human agents currently handle them at roughly 0.5$/ticket. Total support cost is 1 million dollars a month.

Your CEO gives you a mandate. Cut that by 70% with an AI agent.

The ticket distribution looks like this.

1. 50% ~ Where is my order?

2. 20% ~ missing items or cold food.

3. 15% ~ are refund disputes.

4. 10% ~ are restaurant quality complaints.

5. 5% ~ are complex multi-turn escalations involving payment failures, account issues, or angry users demanding managers.

Some tickets take three seconds to resolve. Some take a thirty-turn conversation. Some require reading six previous tickets to understand what the user is asking. Some are hungry users at 10 PM typing in all caps.

The naive answer is to wire up a frontier model and pipe every ticket through it.

Let us see what happens when you do this.

What is Model Choice, Really?

Model choice is not a vendor decision. It is a five-dimensional optimisation problem under a hard business constraint.

You are picking a point on a Pareto frontier defined by Task Fit, Latency, Cost, Context, and Controllability. No single model wins on all five. Every choice is a trade.

For this support agent, the trade is complicated.

If you use a frontier model for every ticket. Quality is high.

At two million tickets of an average 5 turn conversation, at roughly 0.27$ of inference per ticket, you spend 540,000 dollars a month. You have already burned 54% of the savings you were hired to deliver. Add infrastructure, monitoring, safety guardrails, and retries, and you are above 70%.

If you use a cheap model for every ticket. Cost is fine.

But the model hallucinates refund policies, misreads Hinglish, and escalates half your tickets to humans anyway. Your deflection rate drops from 70% to 30%. The math collapses from the other direction.

Neither extreme works. The right answer lives in between. The PM decides where.

The Strategic Bet

Here is what most PMs miss entirely.

The support agent you are about to build does not run on a single model. It runs on a routing layer that decides which model handles which ticket.

—> Where is my order needs a database lookup, a tiny model to format the response, and a sub-500ms latency target.

—> A refund dispute from an angry user who has already had three bad experiences needs a frontier model, long context, and two seconds of real reasoning.

Shoving both requests into the same model is how you lose money and lose users at the same time.

The router is where your actual product intelligence lives. It is the layer that turns commodity foundation models into a profitable support operation.

Anyone can call the OpenAI or Anthropic API. What no competitor can easily copy is your 18 months of telemetry about which ticket class wins on which model at what confidence threshold with your specific users.

Over-routing to the expensive model and inference eats the savings. Over-route to the cheap model and deflection collapses because humans have to clean up after the agent.

The Five Constraints

Before you pick a model, you score every candidate on five constraints. Most PMs only think about one or two.

Constraint 1: Task Fit

Task Fit measures how closely the model’s training matches the actual ticket.

Where is my order is not a reasoning task. It is a structured data lookup wrapped in natural language.

A small model paired with a database call beats a frontier model here. The frontier model writes more than needed, invents delivery estimates, and hedges unnecessarily.

A refund dispute is a reasoning task. The user references past tickets, implies context, negotiates, and escalates. A small model collapses. You need the frontier model.

The same agent handles both. Task Fit tells you they cannot be handled by the same model.

200 real tickets, sampled proportionally across ticket types. Every candidate model runs through it. A human or a judge model rates outputs on a rubric. This eval set is owned by the PM, not the ML team, and it is refreshed every month with real production data.

Constraint 2: Latency

Time to first token is what the user feels when they hit send.

For the 50% of tickets that are order status lookups, your target is under 500ms. The user is anxious. Every extra second is another 10% chance they open Twitter instead.

For the 5% of complex escalations, 2 seconds is acceptable if the response is visibly thoughtful. The user is already in a serious conversation and expects weight.

Your router itself has a latency budget. If your intent classifier takes 300ms to decide where to send the ticket, you have already burned most of the user’s budget before the actual model has started generating.

This is why classifier models are almost always small, often distilled or fine-tuned, tuned to run under 50ms. The router cannot be the bottleneck.

Constraint 3: Cost

At 2 million tickets a month, run the math on frontier-only inference.

Average ticket: five turns, roughly 2,000 accumulated input tokens per turn and 400 output tokens. At frontier model prices of roughly 15 dollars per million input tokens and 60 dollars per million output tokens, one ticket costs about 0.27$.

2 million tickets a month at 0.27$ each is 540,000 dollars in pure inference.

Your human baseline was 1 million dollars. Your AI inference alone is 54% of that. Your CEO is not impressed.

Now run the same math with a routed system.

1. 70% of tickets go to a cheap model at $0.01 each.

2. 25% go to a mid-tier at $0.08.

3. 5% go to the frontier model at $0.27.

4. Weighted average lands at 0.04$ per ticket. Total monthly inference cost: 81K

Same deflection rate. Nearly 7x the margin.

Constraint 4: Context and Memory

The agent needs context like past orders. previous tickets from the same user, restaurant policies, active promotions, delivery agent notes etc.

The instinct is to stuff everything into a 200K context window and let the model figure it out. This fails in two ways.

1. Models measurably degrade past a certain context length, usually between 30K and 100K tokens, depending on the model. This is the lost-in-the-middle problem, and it is real.

2. Cost scales with every token included on every turn across millions of tickets. A 50K context blindly passed every turn turns your 0.04$ ticket into a $0.25 ticket. You have rebuilt the frontier-only problem with extra steps.

The right answer is retrieval. Pull only the relevant past order, the specific restaurant’s policy, the last two tickets, and the user’s LTV tier. Keep the context under 4,000 tokens. Let the router decide when a ticket is complex enough to justify pulling the full history.

Retrieval gives you control. Massive context gives you a black box that silently gets worse and more expensive as you fill it.

Constraint 5: Controllability

A customer support model cannot invent a refund policy.

If the model says you will get a full refund plus 500 rupees credit and that is not your policy, you have two problems. You either honour the invention and bleed money. Or you refuse and face a Twitter escalation.

Controllability is how reliably the model sticks to your rules under adversarial inputs.

Frontier models are generally more capable but not always more controllable. A fine-tuned, smaller model trained on your exact refund policy will follow the rules more reliably than a frontier model with a clever prompt. For the 15% of tickets that are refund disputes, controllability beats raw capability.

Most PMs stop at the first two constraints. The best AI PMs score every candidate on all five, document the trade, and revisit the scorecard every quarter.

Why - Just Use the Best Model - Fails Here

The argument is familiar. LLM Prices are dropping or will drop drastically in future. Capability is doubling. Just pick the top model and wait.

Three reasons this is wrong for the support agent.

1. Your competitor is not waiting. If they run a routed system today, they save 450K $ a month and reinvest it into faster delivery SLAs or cheaper customer acquisition. By the time frontier prices drop, they have already eaten your growth.

2. Best is relative to the ticket class, not the benchmark. The frontier model loses to a fine-tuned, smaller model on structured refund queries.

3. Cost drops do not flow to users. Every time inference gets cheaper, users expect richer responses, longer context, and more autonomy. If your unit economics are bad today, they are still bad tomorrow on a cheaper, more capable model.

Betting on the best model is a tax you pay to avoid doing the actual PM work.

The Router Pattern, Built for the Agent

Your router has five components.

1. An intent classifier sits in front of every ticket. A small fine-tuned model, under 50ms. It reads the ticket and returns one of five labels. order_status, missing_item, refund_dispute, restaurant_complaint, complex_escalation. It also returns a confidence score.

2. A model assignment table. order_status goes to a small model plus a database call. missing_item goes to a mid-tier model with a template response. refund_dispute goes to a fine-tuned, smaller model trained on your refund policy. restaurant_complaint goes to the mid-tier. Complex escalation goes to the frontier model.

3. A confidence threshold. If the classifier returns low confidence, the ticket escalates one tier up. If the primary model returns a low-confidence answer or the user replies “this is wrong”, it escalates again. The third escalation goes to a human.

4. A cache layer. 40% of “where is my order” tickets in a one-hour window ask about the same handful of delayed orders in a single city. Cache the response per order ID with a 60-second TTL. Zero inference cost on a cache hit.

5. A telemetry layer. Every ticket logs the classifier label, model chosen, tokens consumed, latency, user reaction, and final disposition. This is where your routing intelligence compounds.

The sophistication is not in the components. It is in the ongoing tuning of the assignment table based on telemetry.

One Ticket, End to End

Follow a single ticket through the router.

A user types “bhai order kidhar hai, 45 min ho gaye” at 9:47 PM.

The ticket hits the edge. It is hashed and checked against the cache. No hit.

It goes to the intent classifier. Classifier returns order_status, confidence 0.93. 42 milliseconds elapsed.

The router looks up the assignment table. order_status with high confidence goes to the small model plus a database call.

In parallel, the system pulls the user’s active order and the delivery agent’s current GPS location. 80 milliseconds.

The small model receives the ticket plus structured context. It generates “Your order is 4 minutes away. The delivery agent is on the last stretch”. Time to first token: 210ms. Total response time: 480ms.

Telemetry logs the full trace. Ticket class, model used, tokens consumed, latency, user’s next message.

The user replies “ok thanks”. Telemetry marks this as a positive resolution.

The router chose the right model. The user got a fast answer. The ticket cost you $0.004 against a human cost of $0.5. Multiply by one million similar tickets a month, and you see where the money is actually saved.

That is the product.

Model choice is where the business is either made or broken. And it is the PM’s job to decide.

If this article changed how you think about model choice and AI product strategy, you will find much more depth in our AI PM course. We cover model selection, routing architectures, AI evals, cost modelling, and real interview questions from top companies.

Check our highest-rated AI PM course (Including AI PM Interview Preparation) · 4.9/5 · 600+ enrollments → See testimonials and course details

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )

But here’s the truth… they’re playing a completely different game.

To dominate AI, companies need four things:

1. Infrastructure,

2. Models,

3. Data,

4. and Distribution.

Apple may be weak in AI models, but they are insanely strong in distribution.

They sell 200+ million iPhones every year.

And think about how we use phones.

We ask simple things — summarise emails, find photos, send quick messages.

So Apple’s strategy is simple: use our personal data to give hyper-personalised AI, and build the best interface on devices. and let others build the giant models.

For this, Apple is simply partnering with Google.

So Apple may never build the smartest AI… but they might build the most useful AI on our phone. Subscribe for more business strategy breakdowns.

Three days later, you buy a case for it. A week after that, you buy AirPods. Normal journey.

Your recommendation feed is doing its job.

Then life happens. You buy a birthday gift for your niece. A toy. Then a book for your dad. Then a yoga mat. Then some groceries.

Now you come back looking for a screen protector for that iPhone.

Here is the problem. Your recommender has forgotten about the iPhone.

The model remembers what you did most recently.
Toy. Book. Yoga mat. Groceries.
Based on that, it is now quietly convinced you are a gifting parent with a wellness streak. It is showing you more toys, more books, more yoga equipment.

Meanwhile, the single most important signal about what you want right now, the iPhone from three weeks ago, has been washed out.

This is not a theoretical problem. This is happening on most recommendation systems you use today.

Today, we are going to see how to fix that.

In our previous piece, we explained how TikTok uses session-based RNNs to predict your next swipe. At the end, we flagged three pitfalls. One of them was Catastrophic Forgetting. This article is a deep dive into the paper that solved it.

If you are preparing for AI PM interviews, recommendation system design is the most commonly asked system design topic at senior levels. We teach this in our course.

The iPhone Problem

Let us go back to your Amazon story. Why did the model forget the iPhone?

The reason lies in how most recommenders store your history.

An RNN-based recommender works like this. Every time you buy something, the model converts that item into a short numerical fingerprint. Then it mixes that fingerprint into a single vector called the hidden state. One vector. That is the model’s entire memory of you.

Think of the hidden state like a single sticky note. Every time you buy something, the model scribbles on that same note, and whatever was written before gets slightly smudged.

After your iPhone purchase, the note says “wants tech accessories.”

After the case and AirPods, it still says roughly that.

Then you buy a toy. The note gets rewritten. Now it says “tech accessories and a gift.”

Then a book. Yoga mat. Groceries. By the time you come back for that screen protector, the sticky note no longer mentions the iPhone at all. It says something like “parent on a wellness kick with household needs.”

The iPhone signal is not lost. It is buried. Smeared under four unrelated purchases.

This is called Catastrophic Forgetting. And it is not a bug you can fix by tuning the model. It is a fundamental flaw in the architecture. The sticky note itself is too small to hold what it needs to hold.

Why This Breaks Product Experience

This has two costs that hit you directly as a PM.

1. The first cost is performance. Your model misses the highest-signal moments in a user’s journey because they get washed out by noise. A user who bought an iPhone three weeks ago is an obvious candidate for iPhone accessories. Your model does not see it. Your revenue per user suffers.

2. The second cost is explainability. You cannot tell a user why something was recommended. You cannot tell your leadership why the model did what it did. A single hidden vector is a black box even to the people who built it.

If you have ever been in a meeting where your head of product asks, “Why is the model recommending this?” and your ML lead says, “The embeddings suggest...”, you have lived this problem.

How Humans Actually Remember

Here is the interesting part. You do not have this problem.

If someone asks you what to get for a new baby, you do not scan every memory from your entire life. You pull up the specific episode of buying baby stuff for your niece last year. You focus on that. Everything else stays quiet in the background.

You have episodic memory. You can pull up specific moments on demand.

Your recommender does not have this. It only has the sticky note.

What if we gave the recommender episodic memory?

The Fix: A Memory Box, Not a Sticky Note

Instead of the single hidden vector, can we give every user a small memory box?

Think of the box as a row of 20 labelled drawers. Each drawer holds one past purchase. When you buy something new, it goes into a fresh drawer. The oldest drawer gets emptied to make space.

At any moment, your box has your last 20 purchases, sitting side by side.
The iPhone is in drawer 17.
The case in drawer 16.
The AirPods in drawer 15.
The toy in drawer 14.
The book in drawer 13.

And so on.

Nothing is smudged. Nothing is averaged. Each purchase sits cleanly in its own drawer.

Now, when you come back looking for a screen protector, the model does something clever. It does not read all 20 drawers equally. It asks a question.

“Which of these past purchases is most relevant to a screen protector?”

It scans each drawer, scores the similarity, and pays attention to the ones that match. The iPhone drawer lights up. The toy drawer stays dim. The book drawer stays dim.

The model pulls out the iPhone signal cleanly and recommends the perfect screen protector.

This is exactly how attention works in modern AI. The model decides what to focus on based on what it is trying to do right now.

Two Versions of the Same Idea

Here this we can do in two ways, both use the same core idea. They differ in what they store.

1. The first version is called item-level RUM.
   Each drawer in the box holds an actual past purchase. iPhone in one drawer. AirPods in another. This is simple. It is also explainable. You can literally tell the user that we showed you this because of that iPhone you bought three weeks ago.

2. The second version is called feature-level RUM.
   Each drawer does not hold a purchase. It holds a preference. One drawer tracks your brand preference. Another tracks your price sensitivity. Another tracks your style preference. Every time you buy something, the drawers get gently updated. Buy an Apple product, and the brand drawer leans more towards Apple. Buy something cheap; the price is more budget-friendly.

The second version tends to perform better. The first is easier to explain.

How The Memory Updates

The item version is simple. New purchase comes in, oldest one gets kicked out. First in, first out. A 20-slot box always holds the last 20 purchases.

The feature version is more interesting.

When you buy something new, the model does two things.

1. First, it decides what to forget. If you just bought an Android phone, your brand preference for Apple should fade. The model computes a forget signal and uses it to gently erase the old preference.

2. Then it decides what to reinforce. Your brand preference for Android should go up. The model computes an add signal and writes it to the drawer.

The mental model is simple. Every time you buy something, the relevant drawers in your memory box get a small dusting-off followed by a small update.

The beautiful thing is that the model learns what to forget and what to reinforce on its own. You do not write rules. You show it millions of user sequences, and it figures out the pattern.

Thing which Product Manager needs to decide

Memory size

How many drawers per user? More drawers mean richer history, but more computing. 20 might work for e-commerce. For a content platform like TikTok, where users burn through items in seconds, you might want 50 or 100.

Item level or feature level

Explainability or performance. Pick one. You cannot have both.

Memory weighting

optimal weight for recent behaviour. Start there. Then an A/B test. Stable domains like books or music can push intrinsic weight higher. Volatile domains like news or short-form video need more memory weight.

Write strategy

For item level, first-in-first-out is fine. For the feature level, you need the forget-and-reinforce approach. It is more powerful. It is also harder to debug.

If this article changed how you think about memory, recommendation architectures, and AI system design, you will find much more depth in our AI PM course.

We cover these in 40+ Videos and 25+ Case Studies, along with AI PM interview questions from top AI companies.

Check our highest-rated AI PM course (Including AI PM Interview Preparation) · 4.9/5 · 600+ enrollments.

About Author

Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. Weekly Live Webinars/MasterClass ( Here )