AI Failure states & Designing for Evaluations

🧠 Understanding AI Failure States and Designing for Better Evaluations

🛠️ 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 (𝗡𝗼𝗻-𝗔𝗜) 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗮𝗿𝗲

- Predictable: Given the same input, they always produce the same output.

- Rule-based: Behavior is governed by explicitly coded logic.

- Transparent: Errors are usually due to code bugs or misconfigurations and are easier to trace and fix.

Example: "Error: Interest calculation failed due to missing rate parameter. Please contact support."

🤖 𝗔𝗜 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗮𝗿𝗲

- Probabilistic: Outputs can vary even with the same input.

- Data-driven: Behavior is shaped by training data

- Opaque: Errors can be subtle, context-dependent, and harder to debug (e.g., hallucinations, bias).

Every AI system will eventually make bad predictions, 𝗽𝗹𝗮𝗻 𝗳𝗼𝗿 𝗶𝘁

A 𝗖𝗼𝗻𝗳𝘂𝘀𝗶𝗼𝗻 𝗠𝗮𝘁𝗿𝗶𝘅 helps visualize model performance with:

✅ True Positives – Correct positive predictions

❌ False Positives – Incorrectly flagged positives

✅ True Negatives – Correctly ignored negatives

❌ False Negatives – Missed positives

⚠️ 𝗧𝗵𝗿𝗲𝗲 𝗖𝗼𝗺𝗺𝗼𝗻 𝗔𝗜 𝗘𝗿𝗿𝗼𝗿 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀

𝟭. 𝗦𝘆𝘀𝘁𝗲𝗺 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 (𝗪𝗿𝗼𝗻𝗴 𝗢𝘂𝘁𝗽𝘂𝘁)

False positives or false negatives occur due to Poor data, Biases or Model hallucinations

-> Example: "Unusual transaction. Your card is blocked. If it was you, please verify your identity."

𝟮. 𝗦𝘆𝘀𝘁𝗲𝗺 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 (𝗡𝗼 𝗢𝘂𝘁𝗽𝘂𝘁)

True negatives occur due to untrained use cases or gaps in knowledge

-> Example: "Sorry, we don't have enough information. Please try a different query!"

𝟯. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗘𝗿𝗿𝗼𝗿𝘀 (𝗠𝗶𝘀𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗼𝗼𝗱 𝗢𝘂𝘁𝗽𝘂𝘁)

True positives that confuse users due to poor explanations or conflicts

with user expectations


-> Example: User logs in from a new device, gets locked out.

AI responds: “Your login attempt was flagged for suspicious activity”

🧪 𝗧𝗵𝗲 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝗰𝗲 𝗼𝗳 𝗗𝗲𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻𝘀

Designing for evaluations is fundamental to AI system development—it guides improvement, ensures safety, and delivers systems that truly meet user needs.

𝟭. 𝗟𝗟𝗠-𝗕𝗮𝘀𝗲𝗱 𝗘𝘃𝗮𝗹𝘀 (𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲)

A separate language model serves as an automated judge to

- Grade responses

- Provide reasoning

- Assign labels (helpful/harmful, correct/incorrect)

𝟮. 𝗛𝘂𝗺𝗮𝗻 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 & 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻

Critical for real-world alignment through in-Product Feedback

- Reaction buttons 👍👎

- Comment boxes 💬

- Clarification prompts 🧭

𝟯. 𝗖𝗼𝗱𝗲-𝗕𝗮𝘀𝗲𝗱 𝗘𝘃𝗮𝗹𝘀

Automated, deterministic tests for assessing output correctness—particularly effective for structured tasks.

𝗦𝗼𝘂𝗿𝗰𝗲

- PM’s complete guide to evals

lennysnewsletter[.]com/p/beyond-vibe-checks-a-pms-complete

- Designing for Errors

pair.withgoogle[.]com/guidebook/