Imagine you are a product manager at a lending app in Bengaluru.
Five years in, good reviews, and three AI courses finished this year.
You can explain RAG in a meeting. You know what an agent is.
So you apply for an AI PM role and clear the first round.
Then the hiring manager asks one question. You launched an AI feature. How did you know it was good enough to ship?
You start talking about A/B tests. She waits. She was asking about evals, and you have never written one.
That is the gap most PMs carry into AI PM interviews. It is not a knowledge gap. It is a doing gap.
You have read about all of it. You have done very little of it. No certificate shows the difference, but an interview shows it in about two minutes.
So before your next interview, score yourself on 8 skills. It takes 10 minutes.
How to score yourself honestly
Rate each skill from 1 to 5:
1: I have never done this
2: I know the idea but have not done it
3: I have done it once, with help
4: I do it alone at work
5: I could teach it
One rule. Rate what you have done, not what you have read.
A 4 needs work you could show someone. If you only watched a video on it, that is a 2, even if you understood every word.
Below are the 8 skills. Each one has a single test question that tells you your real score, and the score you gave yourself as our lending app PM.
3 AI PM skills your current job already trained
These feel familiar, so PMs rate them too high. Answer the test question, not the topic.
Picking problems where AI fits.
Test: can you name an AI idea you dropped because the data, the cost or the error risk did not work? Knowing when not to use AI is the skill here.
You reshaped one idea once, with your tech lead’s help. Your score: 3.3.
AI metrics and experiments.
Test: for one AI feature, can you name a quality metric, an adoption metric, a business metric and a guardrail? And do you know when an offline eval is enough and when you need an A/B test?
Five years of dashboards help. Your score: 3.7.
Shipping and communicating AI work.
Test: can you explain an AI feature’s quality, risk and cost to a senior leader in 5 minutes? Do you have a plan for when the model is wrong in production, with a fallback and a person who can roll back?
This is where your PM years show. Your score: 4.3.
3 AI PM skills the AI job adds
This is where the doing gap lives, and where AI PM interviews go deepest.
Defining quality and writing evals.
Test: have you written an eval set with common cases, edge cases and never cases, then made a ship or don’t ship call from the results?
PMs who know AI often score lowest here, because evals are written, not read about. Your score: 1.7.
RAG and retrieval.
Test: when a RAG bot gives a wrong answer, can you tell if retrieval fetched the wrong chunk, or the model misread the right one?
That one call definitely decides what the team fixes next. Your score: 2.3.
Agents and tools.
Test: have you built or specified an agent that calls at least one tool, and decided where it needs a human check before it acts?
Your score: 2.0.
2 skills that decide if your AI feature survives the budget review
Many AI features work in the demo and die in a finance meeting. These two skills are why.
Cost and latency.
Test: can you work out cost per query from tokens and the price per million tokens, then name three ways to cut it, with the quality risk of each?
The maths is simple, and nobody does it. Say your loan assistant gets 2 lakh queries a day. A 50 paise difference per query is Rs 30 lakh a month. If you can’t show that number, someone in finance will decide your feature’s fate without you. Your score: 2.7.
AI strategy and pricing.
Test: Can you say where the moat of an AI product is, beyond the model? Every competitor can call the same model API.
The moat is your data, your workflow or your distribution. And can you say how cost per user changes the price? Your score: 2.7.
Why your overall AI PM score lies
Add it up, and you score 2.8 out of 5. That reads as working level. Not bad.
Now look at where the points come from.
Your three highest scores are the skills your old PM job trained: communicating, metrics, picking problems.
Your three lowest are evals, agents and RAG. The building skills.
That is exactly the profile that struggles in AI PM interviews. Strong at talking about AI work, weak at doing it. The average hides it completely.
So ignore the overall score. Sort the 8 skills from lowest to highest and look only at the bottom two. Treat them separately from the rest.
For you, that is evals at 1.7 and agents at 2.0.
What to do this week about your top 2 gaps
Don’t try to fix all 8. Fix two properly, then score yourself again.
If evals is your gap:
Pick one AI feature, yours or one you use daily, like search on a quick-commerce app.
Write 20 test cases: common ones, edge cases, and never cases where a certain answer must not happen.
For each, write what a good answer must include.
Run them, mark pass or fail, and make the call: would you ship it?
That moves you from a 1 to a 3. More useful, it gives you a real answer to the interview question this article started with.
If agents are your gap:
Build a one-tool agent with Claude. For example, one that reads a CSV of support tickets and drafts a weekly summary.
Write down every step it took.
Then decide one thing it must never do without asking you first.
That last step is the part interviewers care about. Anyone can make an agent run. The PM decides where it must stop.
Score yourself again after 4 to 6 weeks. If your bottom two moved up, you are spending your time on the right things. If they didn’t, you were reading again instead of building.
I have put this whole self-check into a free scoring sheet that works in Google Sheets or Excel. 24 statements across the 8 skills.
You rate each one. It works out your level for every skill, ranks your gaps, and gives you a next step that fits your level.
If you want to close your gaps with someone reviewing your work every week, that is what our AI PM Builder Cohort is built around.
Our next AI Product Manager & Builder Cohort is starting. For an early-bird discount, you can fill out the form here: AI PM Builder Cohort
[Alumni Shipping at Google, Microsoft, Uber]
And if this helped, subscribe so the next practical breakdown lands in your inbox.
About Author
Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking.
AI Learning Resources
12 Weeks LIVE AI PM Cohort (Alumni now ship AI at Google/Microsoft, etc)
AI PM Course [Rated 4.9/5 from 1000+ Product Managers from Google, Coinbase, Microsoft, Uber, etc]
AI Program Management Course (For Program Managers and TPMs)
AI Business Course (For General Management, Biz Ops & Strategy Roles)