Designing Safety For An AI API
AI PM Interview Questions
This question is being asked more frequently in Product Management Interviews. You can see other questions on Agents, RAG Metrics, RAG RCA, Agent RCA, Evals, AI Design, etc here. This contains almost 48 videos that focus on your PM interview preparation.
Now, coming back to this question, we first ask clarifying questions and break down the problem statement using first-principles thinking.
Clarifying questions
First, we need to understand what kind of AI this is. Is it text only, multimodal, etc. If it is text only, then we only need to think about what the model is saying, but if it is about the Images, then we need to think about safety attacks that could be hidden in the Image pixels, correct?
Let’s assume that currently we are talking about multimodal ( more challenging than text-only)Second, we need to think about who the customer segment is, like individual developers or enterprise users? Let’s assume both
First Principle breakdown of AI API Request
So imagine you call an API, so basically developer sends a request. Then the system assembles the real context ( via conversation history, RAG documents, etc etc). Then it goes to the LLM, and then we get a response.
So where can all things go wrong?
1) The input is bad
This can include jailbreaking, some prompt injection which is inside any document which the user uploaded, etc. PII data can also be there in the user’s request
2) The context assembled is bad
The majority of candidates are not able to think about this one. The model not only sees the input from the user, but it sees everything that creates the whole context, like the system prompt, RAG chunks, history, etc. So any of these sources can have the data embedded, and that can compromise safety.
3) Model processes bad output
Since AI is probabilistic, it can happen that the output is also not good/
4) The response leaks.
If the system outputs any secured data, maybe the harmful content or the PII data, that can be another failure.
Solutions
Stage 0. Identity and access
First, we need to know who is calling. Because nothing else will work down the funnel if we don’t even know who is calling.
If it is a developer, then we can restrict the RPM; we can restrict the customisation, etc
If it is Tier 2 enterprise, we can think about providing customer/configurable behaviour, dedicated safety review.
Stage 1. Input analysis
We can think about creating 2 passes which will analyse the input
Fast Pass - The fast pass will do pattern matching or heuristics. It can detect 70-80% of the safety attack. Because it is based on pattern matching, it is cheap and fast.
Slow Pass - The slow pass is a real ML classifier trained on adversarial data. 50 to 100ms, which is far too slow to sit in front of inference.
Stage 2. Context assembly
3 things can be done before the model sees the prompt.
System prompt protection first. The model should be trained and instructed never to reproduce it, and the output stage should independently check for it. Two locks, because one of them will fail.
PII scanning before inference, not after, by scanning the whole assembled context. RAG documents are the ones where this kind of surprise can occur.r.
Then re-inject safety instructions at the end of the assembled context, after the untrusted content.
Stage 3. Model behaviour
Two layers. The boundary between them is a product decision, not a technical one.
Layer one is platform level and immutable. A short list of categories that no configuration flag will ever unlock. No developer can disable it. No enterprise contract buys an exception. Being unambiguous about that in public is itself a feature.
Layer two is developer configurable. Strict, moderate, permissive, gated behind Tier 2. A medical research customer and a children’s education app genuinely need different thresholds. Pretending otherwise just means both of them route around you.
Stage 4. Output evaluation
Split by consequence, not by category.
Synchronous (< 50ms) - We can block the output delivery in case of PII in output, system prompt leakage, etc.
Asynchronous - Deliver now, FLAG later. Contextual harms, bias, hallucination, copyright. Dev notified via webhook. Repeat violations = rate limit.
Stage 5. Post response learning
Aggregate Abuse Detection - We can track the volume, topic concentration, and refusal rate. .i.e., one request is fine. 5,000 probing the same topic is an attack.
Closed-Loop Retraining: Think as if every safety decision is training data for you. If any incident happens, try to find out the root cause, retrain it, do A/B testing and deploy.
Metrics
False Positive, Precision, Recall, F1 Score,
P50, P75, p90 Latency
If this changed how you think about our Job Ready AI PM Cohort | Cohort 1 Progress so Far
Registrations open, limited seats. Fill this Form to Show Interest
About Author
Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking.
More Resources
Product Management Mock Interview (Detailed)
Crack AI Business Roles - Course Details
Crack AI Program Manager Roles - Course Details