Any AI model you incorporate into your product is will succeed or make mistakes. Plan for the fact that your AI system will make bad predictions at some point. A Confusion matrix is a table which which helps analyse these errors and provides insight into how well the model is performing by showing the counts of true positives (correctly identifying a positive case), false positives (incorrectly identifying a positive case), true negatives (correctly identifying a negative case), and false negatives (failing to identify a negative case) .

There are broadly three scenarios where AI failure states can occur

1. System Failure or Wrong output

   Accounts for errors which are false positives and false negatives predictions. This could be because of several reasons like poor data collection and training, biases and hallucinations. For example, your bank’s AI system flags a legitimate transaction as fraudulent because it mistakenly identifies an unusually large purchase as suspicious, even though it was authorized by the user. Solution: Give a way forward for the user to report the issue or manual controls to the user to resolve issue or offer access to human support for high-stakes use cases.
   

2. System's Limitation Errors

   System Limitations occur due to true negatives i.e when there is no available output from the system for a given input. These means the underlying Model is not trained for a particular use-case. The system can’t provide the right answer, or any answer at all due to inherent limitations to the system. For example, if a user asks their bank's AI assistant about loan options for self-employed individuals, but the system doesn't have relevant data for that specific query, it returns no response because it wasn't trained to handle that scenario.

   Solution: Inform user about system’s limitations correctly during Onboarding. Check Articulate Product Capabilities
   

3. Contextual Errors 

   These errors occur when users fail to understand the system's intended output or when the system's actions are poorly explained, conflicting with the user's mental model. These are often true positives, i.e the system is working as intended but the user struggles to interpret its response. For example, your bank’s AI system flags an unusual transaction and temporarily locks the user’s credit card. While the system correctly identifies a potential fraud attempt, the user perceives it as an error because they were simply making a legitimate but rare purchase. Solution: Offer In-app settings to the user to change the way the system functions to better align with user needs and expectations.