How to define the success criteria of an AI product?

FAQs for AI Zero → One (Episode #3)

Transcript

Welcome to episode three of Unraveling AI Entrepreneurs. Essentially, this is a series of frequently asked questions that you as engineers and product managers would encounter in your journey of going from zero to one in the domain of AI.

In the previous episode, we had started off with the broader question how do you frame an AI problem? And in the last two episodes, I covered user needs and journey map. These are the two aspects that are very important parts of framing the AI problem.

In this one, let’s talk about success criteria. How do you measure the success criteria of an AI problem? This is a very important aspect. Oftentimes, we start off projects without a clear understanding of what success criteria should be.

So, the way you think about the success criteria of an AI product, you have two categories of KPIs. One is the business KPIs and the other is technical KPIs.

So let me give you a few examples. Let’s say if you are building, a fraud detection, uh, solution. In that case, you’d primarily start off with success criteria, which would be a technical KPI created with respect to precision and recall. What is the probability you’re able to detect a transaction to be fraudulent?

Let’s take another example. If you’re doing churn prediction, uh, churn prediction for your customer. If you’re building a solution for that, your success criteria would be in terms of the business KPIs. What kind of customer attrition, uh, or customer lift did your solution provide? So you would measure it before and after it, that is, without the solution and with the solution.

Stay- a few other examples, for instance, if you’re building a solution to improve productivity. Let’s say, you know, you have receipts that you want to automate the scanning analysis and filing of, filing for filing purposes, you know, that could be an example where you would, um, again, user combination here. Uh, technical KPIs would be focused on what is the accuracy with which the receipts are scanned, the OCR aspects of…