Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse

[music] >> Uh I'm leading AI engineering solutions at LangFuse uh by ClickHouse. And uh we are an a agent observability and e-boss platform that a lot of users use open source. And uh I'm here to talk to you about an experience that we had with one of a top five bank helping them getting through how they can put a safety uh guardrails whenever they were shipping AI applications into production. And uh to make sure that they are running with the e-proper e-vals on that. I I like to start with uh you know, something that we learned with the experience with this customer is that regulated industries they don't necessarily care about the public benchmarks. I know that every week uh when anytime a new model comes out um we see all the benchmarks, you know, uh Opus 5 is now leading the benchmarks in a in a multitude of dimensions and that's great. Um that tells us what the models are really good at in terms of like the technical capabilities and maybe using sometimes like open data sets that are out there. But sometimes that doesn't mean that it will translate effectively to internal data and use cases that you have in in your organization. Specifically around you know, financial services where banks are very very sensitive to the their data,

their use cases and for the most part they like to believe that that's part of their secret sauce, the way that they you know, build their own models, build their own uh data sets and also not exposing a lot of that data to uh to the public. Um so that's why we needed to figure out a way that the you know, financial services and this bank in particular that they can then say, "Look, we don't necessarily care what the model can do. What we care about is really whether this application that we're trying to build can ship into production. A good example would be like a 10K analyst that they were trying to build for their investment bankers. You know, every time an earnings call is is out there, they want to make sure that their teams can accelerate and get leverage from all the information that is out there and as soon as an earnings call is done, they can get an analysis on all the transcripts and all that. However, that is not just a simple task of just taking that the call transcript and just like summarizing it and what not because there's a lot of knowledge that the analyst has and a lot of context about how the bank and the firm operates that is not necessarily ingrained or trained in the models out of the box. So, how do then you bring that knowledge and that context into the model?

And then furthermore, once you've done that, how do then you reliably are able to evaluate that the output that is giving you are the the the correct ones and that you can kind of clear those gates that you have put in place. So, that was that was again a kind of a a big learning for us that they and they told us directly, "Look, we don't certify models here. We certify applications, right? So, it it could be a 10K analyst, it could be an investment analysis and what not." The other learning from this experience is that in regulated industries and in finance in particular, there are three main requirements when it comes to building the solutions. The first one that more more often than not, they wanted to stay on their own premises. Um and this could be either their own data centers because it's it's funny how things are kind of coming back to to to center because we have done we have been doing all this movement into the cloud, but now because of AI and how sensitive the information that we're feeding these models is, then especially banks are rethinking and repurposing a lot of their data centers that they had and actually standing it up again and and kind of standing up GPU infrastructure to run models locally because of the sensitivity of the data that they are dealing with on these models. The other thing is the other requirements that it has to be gated. It cannot be just going out to the internet obvious for obvious

reasons. They don't want to have even the the appearance that some model provider will train on their data even if there is like zero data retention and all and all these, you know, guarantees. But banks are very sensitive to that and they want to make sure that that stays within their control and sometimes you have even banks that actually have kind of research divisions that actually spend time and have PhDs training their own models and fine-tuning them and all that. And the third aspect is with all of this is that the deployment needs to be air-gapped, right? So that only the people that needs to see that data and it's within their infrastructure. That is very very critical. The The other thing is when when we were talking to them it's like, "Okay, now you have all these platform, we have all these models, but then we need to prove that the application is actually working to the way that we want. One, second, that is actually meeting the compliance requirements that we have as a regulated industry. And third, that our users are not going to come to us complaining that is not usable because then, you know, it becomes a situation where you don't get user adoption and then there's a dissatisfaction and a bad sentiment about using AI-enabled solutions. So we needed to find a way to help them come up with a framework and a methodology for them to have like a stamp of approval pass fail if an application or an AI agent had any change on their configuration. That could be the system

prompt, that could be the model that they're using, a new tool, but then how do I give that, "Okay, this is ready to go into production or not based on on these changes that we have given them?" And then if we go like very high-level, how does that work mechanically? Right? The first thing that we need is to build a golden data set. That's the first thing. That's kind of the anchor. Um a lot of this good good news is a lot of these banks they typically already have a golden data set that they want to test against. But sometimes what is missing is how they can operationalize the testing of the test cases against that data set. But that's to me typically is the anchor. The way that I like to think about this if you're coming from a software engineering background is is these are your test cases, right? You will never push anything into production without passing it through QA, right? So this is the same principle. Then once you run this um and and you operationalize it, you're going to generate a bunch of traces. And the traces are the evidence of what the model was trained on. Or sorry, what the model was evaluated on, right? Against this data set. So you have that evidence on the trace. And then at the end with the trace as the as the as the kind of the uh as that, you know, evidence that you have, you can then generate the scores that are going to give you the pass fail across different dimensions that you uh

select. When we started out this journey with this customer, it was we first started with a couple of with a bunch of models, um a small large models with different levels of intelligence. And we noticed something interesting, right? So this one is is a particular example of two numbers that stood out for us. The The first one, 97% in groundness, is that this model was really good at extracting information from, you know, internal sources and grounding the responses on actual data that the bank had. Perfect. 97% That is a passing bar. Amazing. But then when it came down to give, you know, a accurate numbers, then the number was trip the model was tripping up. Because again, LLMs inherently are not great at math. They're just next token predictors. And then what happened was that even though it was grounded the responses on actual information from the bank, whenever it needed to give financial numbers or just do simple math, the model was failing. And then, I'll show you in a minute how we were able to go about that.

And this leads us to then, again, what I said before, that they don't approve models. They approve applications. So, how they can put a harness or controls around this model, so irrespective of if tomorrow I swap a model, then the quality of my application will still remain the same. Now, we're expanding into kind of the process that we help them establish. You're starting from the left with the golden data set, then you go into the inside of the agent. The agent is going to do a bunch of tasks. It's going to have a planning phase. It's going to retrieve data from your official official sources. And then, you see the little box that is is circled in blue, he's going to calculate. In this case, what we did, we gave the model a tool for those areas where the model was com- coming up short, right? The 60% where it was not scoring well, then we said, "You know what? This is a task that the model doesn't need to do on its own. We can just give it a tool because math, you know, is deterministic. It's it's going to be the same all the time. And then, we can improve the the accuracy of the model."

But at the same time, if you notice all the dimensions that we have, we are evaluating the application against a multitude of dimensions, not just one. We're not going to say, "Well, because everything in the average is giving me above 90%, then it's going to pass." That might not be the case. You might have different degrees or thresholds for the different categories that you have here in terms of the accuracy, regulatory compliance, and whatnot. And then, at the end is where you define your gate. You see, not anything that doesn't clear the gate based on your thresholds, doesn't get promoted into production. This is just a a snippet of the code and and at the end I'll link to the repo and to the blog post so you can read more in depth about the work that we did here, but it's effectively you want to build your own benchmark instead of relying on the public ones that are available there. And the the first thing you need to do that is to build the data set like I said before. This is an example of what a typical data set would look like.

It's They are key pairs. You have inputs, expected outputs, and some metadata that you are then using to evaluate how the agent is performing both in terms of quality, behavior, and the responses that it's giving you as well. This is how it looks after we have a scored or evaluated one agent run. And you notice that I circled the calculate tool call because in this case now, this is how we were able to get the model from 60% in numerical accuracy all the way down to like 95% because we gave it a tool. Anytime there's a number that needs to be calculated, just use this little Python code and off you go. In In that way, you can improve the the results that you're giving back to your to your customers. The other step on on this is about scoring. How do you score each of the dimensions that I was telling you before? We have a couple of ways that you can do this. You can have deterministic scores that again, like the name implies, is, you know, very very simple. This is This is not LLM based. You just have a code that is running and it's for the same inputs, it's going to generate the same outputs all the time. And what we've noticed is with these solutions as they get more complicated, you need a blend of scores and evaluators that are deterministic

and non-deterministic because otherwise relying 100% just on LLM judges is not going to cut it. Um and then this the second one, the path, is you also want to figure out if the agent is taking the right, you know, route or path to get to the answer. Because it might be that the answer might look okay or complete uh or semi-complete, but then maybe the route it took to get there, it might not be the appropriate one. Might be missing steps. So, you also can evaluate that. And the third one how you use an LLM as a judge to judge some of the more qualitative aspects of that answer. And by doing all of this, then we got to taking this little model that was failing with 60%, we got it to over 90% and then we are clearing the 85% gate that the bank had for this particular use case by adding that tool, but also by adding this evaluation framework on top that then we can measure the accuracy and the quality of the agent across many different dimensions. And this is something very important because now you can go back and show the evidence of how you actually measure the quality of the agent, but also more importantly, you can link it to the changes that were made into the agent, into the configuration, system prompt, model, tools that were used, and then

you say, "Oh, but look at how by adding this tool, changing this model, we went from 60% accuracy over to 95, 90 plus percent accuracy." And now this is where you start defining your gates, right? And this is where working with your domain experts becomes very, very critical. You don't want to just rely on technical kind of ability for this. You want to make sure that if you're delivering uh an a risk use case that is going to be in a in a enabled, that you have the risk people involved to tell you what is the bar that it needs to meet for them to feel comfortable putting this either internally or in front of customers? And that bar might be different depending on the use case. So, that is why you need to have first the dimensions or the scores uh on which you're going to evaluate this application. And then for each of those and for each use case or application, you end up with different thresholds that the agent needs to meet in order for it to have {quote} that stamp of approval uh for it to go into production.

And this is what it looks like when you we have like a full end-to-end run across, you know, we had one data set that it was like 150 questions, another one was like 1,500. So, but now you can measure the configuration of the agent across these different dimensions and vertices. And you see how even though it is passing on groundness, completeness to use, you might say, "Well, if I take the average of all of these, then it should be okay." But in regulatory compliance is getting a zero. So, that is one that is just binary. It's just either zero or 100. You don't want to put anything if you're a bank out there that is going to violate any regulatory compliance requirement that you have because you might get penalties, you might incur into reputational damage, and all of that. So, it might be that one dimension actually completely shuts down the gate for you to move this into production. So, that's why it's critical to have a multi-pronged view on how you evaluate your agents.

And this is where when you start to kind of picking up all these failures or what we call them failure modes, this is where bringing back to the main experts is really critical. You can curate for them what we would call an annotation queue. So, then say, "Anytime that there is a failure on regulatory compliance, I want to curate those specific traces or evidence, right, for my evaluation and put it in a place that my domain expert can come in and look at the trace, look at what happened, and then give me feedback. And with that feedback, I can then go back and, you know, feedback into the agent to improve it. And this is what it would look like connecting it to the process before, but including that human expert into the middle. If you notice down below, that anytime a couple of things are happening here, anytime a prompt gets promoted or a change in the configuration, then it triggers a rerun of the gate clearance process or the evaluation process. But then, anytime a rewriter or a human expert comes in and gives me feedback, that is a potential candidate to go back into my golden data set. Because now we have evidence from runs of the agent that have been reviewed, and now we can say, "Oh, actually this one piece is a better, you know, bar to evaluate this particular dimension on the agent." And this is kind of the whole process end to

end to make sure that we are including the the domain experts, but also making sure that we're clearing our our quality gates. Then this got us to a place where we were able to help them go from weeks to minutes when it comes to, you know, evaluating a applications, especially in like risk and compliance, which typically take even months sometimes. So, this process it becomes very iterative. We can be very specific and focus on the areas that we want on the agent. Then there's no template for this. Um, you know, this industry's been been developed right now. So, typically you need to define what is it that you want to evaluate in in in kind of the metrics that matter to you. And the three things that I like to leave you with is you need to write your own benchmark because like I said, the public ones don't really matter for your data because they were not done on your data per se. The second one, you got you have to gate the application, not the model. That gives you more flexibility if tomorrow you want to swap a model, you have already all the controls in the application and in the configuration of the agent, and then it becomes easier for you to adjust. And then the third, let me make sure that the domain experts come come in and set the bar or the thresholds that you are deem reasonable in order for you to promote this into production.

And finally, you know, use your data, use your bar, use your gates, especially in regulated industries where you want to keep things in your premises and you're probably not relying on publicly available information. So, thank you for the time. I know I'm a little bit over, but I wanted to give you with the detailed blog post of this work that we did. It also links to the repo that you can reproduce this yourself, and I'm around if you have any questions. So, thank you.