Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Good day everyone. Good day everyone. My name is Ankur. I'm My name is Ankur. I'm My name is Ankur. I'm going to teach you going to teach you going to teach you how to get your agent to how to get your agent to how to get your agent to "cook." Again "cook." Again "cook." Again , this isn't about the food. I don't , this isn't about the food. I don't , this isn't about the food. I don't know. But yeah, let's know. But yeah, let's know. But yeah, let's get to the point. get to the point. get to the point. So, before we So, before we So, before we get to that, get to that, get to that, let's start with a quick let's start with a quick let's start with a quick question and question and question and survey for everyone. survey for everyone. survey for everyone. I'm sure I'm sure I'm sure many of you many of you many of you are creating agents or have are creating agents or have are creating agents or have even already launched even already launched even already launched them. I them. I them. I wonder how many of you are wonder how many of you are wonder how many of you are currently guided only by currently guided only by currently guided only by intuition? And when I intuition? And when I intuition? And when I say "intuition," it's say "intuition," it's say "intuition," it's not a bad thing. I not a bad thing. I not a bad thing. I mean, you mean, you mean, you check check check the answers, look the answers, look the answers, look at them, and make sure at them, and make sure at them, and make sure everything is working everything is working everything is working correctly, based correctly, based correctly, based only on the input and only on the input and only on the input and output data. You output data. You output data. You check whether it is check whether it is check whether it is correct or not. correct or not. correct or not. How many of you How many of you How many of you do this? Good. Quite a do this? Good. Quite a do this? Good. Quite a few of you. few of you. few of you. How many of you have a How many of you have a How many of you have a set up set up set up evaluation system? Well, evaluation system? Well, evaluation system? Well, you know, like, an LLM you know, like, an LLM you know, like, an LLM as a judge, or as a judge, or as a judge, or software testing, or software testing, or software testing, or something like that. Okay, something like that. Okay, something like that. Okay, a little less, but not a little less, but not a little less, but not much. Actually, much. Actually, much. Actually, that's pretty good. And that's pretty good. And that's pretty good. And how many of those who have a how many of those who have a how many of those who have a set-up set-up set-up assessment system trust assessment system trust assessment system trust it so much that they are it so much that they are it so much that they are confident in full confident in full confident in full test coverage and are test coverage and are test coverage and are ready to constantly ready to constantly ready to constantly move forward? Wow, that's move forward? Wow, that's move forward? Wow, that's really impressive. It really impressive. It really impressive. It was a confidently was a confidently was a confidently raised hand. Which, raised hand. Which, raised hand. Which, actually, I expected.

actually, I expected. actually, I expected. I mean, it's very I mean, it's very I mean, it's very hard to trust hard to trust hard to trust estimates because they're estimates because they're estimates because they're hard to create. You hard to create. You hard to create. You don't quite know what you're don't quite know what you're don't quite know what you're looking for. This is a looking for. This is a looking for. This is a non-deterministic non-deterministic non-deterministic case, and it is very difficult case, and it is very difficult case, and it is very difficult to determine exactly what to determine exactly what to determine exactly what needs to be evaluated needs to be evaluated needs to be evaluated to make sure to make sure to make sure our agents are working our agents are working our agents are working correctly. This is one of the correctly. This is one of the correctly. This is one of the biggest problems. biggest problems. biggest problems. Many people start with a Many people start with a Many people start with a good starting point good starting point good starting point based on based on based on intuition. This intuition. This intuition. This helps to get started and helps to get started and helps to get started and make sure your make sure your make sure your agent is doing what you agent is doing what you agent is doing what you expect. But expect. But expect. But realistically, you can't realistically, you can't realistically, you can't rely rely rely on a "yes, this is on a "yes, this is on a "yes, this is right" feeling because in a right" feeling because in a right" feeling because in a production production production environment you won't environment you won't environment you won't know where know where know where errors are occurring and errors are occurring and errors are occurring and what's going wrong unless what's going wrong unless what's going wrong unless you have real you have real you have real tests to find the tests to find the tests to find the source of the problems and source of the problems and source of the problems and fix them. So fix them. So fix them. So before we get before we get before we get straight to the straight to the straight to the grading, I want to grading, I want to grading, I want to take a step back and take a step back and take a step back and talk about what talk about what your your grades should be based on in the first place. And this will be in grades should be based on in the first place. And this will be in grades should be based on in the first place. And this will be in addition to the tracings. addition to the tracings. addition to the tracings. And why do we need And why do we need And why do we need tracing? If you've tracing? If you've tracing? If you've ever ever ever done done done traditional traditional traditional software development, you've experienced software development, you've experienced software development, you've experienced tracing challenges. tracing challenges. tracing challenges. This is for APM monitoring, This is for APM monitoring, This is for APM monitoring, it's a guarantee that it's a guarantee that it's a guarantee that your systems are reliable your systems are reliable your systems are reliable and up and running. But and up and running. But and up and running. But realistically, what realistically, what realistically, what you need, you you need, you you need, you can safely can safely can safely debug, and it's debug, and it's debug, and it's very easy to do very easy to do very easy to do since you have a since you have a since you have a clear clear clear deterministic flow deterministic flow . Isn't that right? You . Isn't that right? You . Isn't that right? You can even see it can even see it can even see it right here. It's very right here. It's very right here. It's very easy to track down, and this is what the easy to track down, and this is what the code looks like. You have a code looks like. You have a method for handling the method for handling the method for handling the send. You send. You send. You check if it is check if it is check if it is valid. If not, it valid. If not, it valid. If not, it throws an error. If throws an error. If throws an error. If so, it so, it so, it moves on to the moves on to the moves on to the next list and next list and next list and continues the process.

continues the process. continues the process. So, it's very easy So, it's very easy So, it's very easy to figure out exactly where the to figure out exactly where the to figure out exactly where the failures are occurring, failures are occurring, failures are occurring, based simply on based simply on based simply on this path history. this path history. But with the agent, unfortunately , you , you , you only have one line of code in your code, only have one line of code in your code, only have one line of code in your code, which simply calls which simply calls which simply calls agent.run and then the request. And agent.run and then the request. And agent.run and then the request. And ultimately, what he ultimately, what he ultimately, what he is going to do is up is going to do is up is going to do is up to him to him to him . Will . Will . Will he summon the instrument? Will he summon the instrument? Will he summon the instrument? Will he perform any he perform any he perform any analysis? Will he be analysis? Will he be analysis? Will he be pondering pondering pondering something internally? something internally? something internally? What is his next What is his next What is his next best move and what is best move and what is best move and what is actually the best actually the best actually the best plan of action? We don't know plan of action? We don't know until we see until we see until we see the trace. You won't the trace. You won't the trace. You won't know what know what know what your agent is doing. So this your agent is doing. So this your agent is doing. So this trace helps trace helps trace helps you identify that. And you identify that. And you identify that. And here's a good here's a good here's a good example of what a example of what a example of what a trace might look like trace might look like . You have an . You have an . You have an agent call. You can agent call. You can agent call. You can see the calls for see the calls for see the calls for LLM. You have documents LLM. You have documents LLM. You have documents that you receive, that you receive, that you receive, tools that you tools that you tools that you engage. You engage. You engage. You actually see actually see actually see every step of the way every step of the way every step of the way your agent is performing. And your agent is performing. And your agent is performing. And here you can even here you can even here you can even see, hey, we have, see, hey, we have, see, hey, we have, for example, this for example, this for example, this getorderstatus tool.

getorderstatus tool. getorderstatus tool. He called the wrong He called the wrong He called the wrong tool. tool. tool. A user asked A user asked A user asked us: hey, I should have us: hey, I should have us: hey, I should have gotten a gotten a gotten a refund. Where is my refund. Where is my refund. Where is my refund? If refund? If refund? If we respond: hey, we respond: hey, we respond: hey, your refund your refund your refund has been processed. And has been processed. And has been processed. And they know that was they know that was they know that was their status based on the their status based on the order status, it's not what order status, it's not what they were looking for. They they were looking for. They they were looking for. They wanted to understand wanted to understand wanted to understand why their why their why their refund had not yet refund had not yet refund had not yet arrived. So, we arrived. So, we arrived. So, we need to potentially need to potentially need to potentially call other call other call other tools, tools, tools, perform other actions. perform other actions. perform other actions. There are so many There are so many There are so many different options, different options, different options, even just knowing even just knowing even just knowing that there is a problem here. But that there is a problem here. But that there is a problem here. But we'll dive into that a we'll dive into that a we'll dive into that a little later. So little later. So little later. So when you finally when you finally when you finally understand: okay, we understand: okay, we understand: okay, we have a trace. Now I have a trace. Now I have a trace. Now I want to move on to want to move on to want to move on to fixing what's fixing what's fixing what's going wrong with my going wrong with my going wrong with my agent. We'll start agent. We'll start agent. We'll start where...First, you where...First, you where...First, you have to identify these have to identify these have to identify these problems, right? These are problems, right? These are problems, right? These are product failures. Like product failures. Like product failures. Like we just did on the we just did on the we just did on the last slide. We last slide. We last slide. We found out that the found out that the found out that the tool was tool was tool was wrong. But wrong. But wrong. But what is your next what is your next what is your next step? You want step? You want step? You want to find out more than just to find out more than just to find out more than just where the where the where the error occurred. Can I error occurred. Can I error occurred. Can I get a good get a good get a good sample of such sample of such sample of such information that can then be information that can then be used for used for experiments and experiments and experiments and iterations to iterations to iterations to improve the agent improve the agent improve the agent when I want to when I want to when I want to test these test these test these cases. So what cases. So what cases. So what was the real was the real was the real problem, right? Sure, the problem, right? Sure, the problem, right? Sure, the tooltip tooltip tooltip was wrong, but was wrong, but was wrong, but should the should the should the LLM prompt be changed? Should LLM prompt be changed? Should LLM prompt be changed? Should we add we add we add fuses? Should fuses? Should fuses? Should we change the we change the we change the tool? Maybe tool? Maybe tool? Maybe the tool just was the tool just was the tool just was n't doing what it was supposed to, n't doing what it was supposed to, n't doing what it was supposed to, right? There are so many right? There are so many right? There are so many different areas that we different areas that we different areas that we need to check need to check need to check and test. And and test. And and test. And so we can so we can so we can test them, test them, test them, create assessments to create assessments to create assessments to continuously continuously continuously monitor these monitor these usage scenarios. So, usage scenarios. So, for example, if you for example, if you for example, if you find, "Hey, we find, "Hey, we find, "Hey, we need to need to need to make sure we make sure we make sure we set up a set up a set up a fuse here." "How are we fuse here." "How are we fuse here." "How are we going to going to going to do this?" We want to...

do this?" We want to... And then we want to And then we want to And then we want to make sure that make sure that make sure that the fuse does the fuse does the fuse does its job. So we its job. So we its job. So we install install install an evaluator on top of an evaluator on top of an evaluator on top of this fuse, this fuse, this fuse, checking that it is checking that it is checking that it is doing its doing its doing its job by prohibiting the job by prohibiting the job by prohibiting the leakage of certain leakage of certain leakage of certain information. Finally, information. Finally, information. Finally, you make these changes, you make these changes, you make these changes, run that run that run that experiment experiment experiment I talked about at the beginning, I talked about at the beginning, I talked about at the beginning, and eventually release and eventually release and eventually release it, it, it, repeating this whole repeating this whole repeating this whole cycle over and over again. This is cycle over and over again. This is cycle over and over again. This is your standard your standard your standard process for process for process for improving improving improving your agent's performance. It's your agent's performance. It's your agent's performance. It's quite long, quite quite long, quite quite long, quite tame, and quite tame, and quite tame, and quite tiring. It tiring. It tiring. It works, but we want to works, but we want to iterate and iterate and improve it to some extent. improve it to some extent. improve it to some extent. How can we How can we How can we do this? We want to do this? We want to do this? We want to make sure we don't make sure we don't make sure we don't spend hours or spend hours or spend hours or days searching for the problem days searching for the problem , tracing it, , tracing it, , tracing it, identifying all those identifying all those identifying all those failure points, developing a failure points, developing a failure points, developing a potential solution, potential solution, potential solution, and going through and going through and going through this entire this entire this entire eight-step eight-step eight-step process manually. We process manually. We process manually. We want to bring it all want to bring it all want to bring it all together and reduce the time together and reduce the time together and reduce the time to minutes. The best to minutes. The best to minutes. The best way to do it is way to do it is way to do it is right here. We right here. We right here. We collect all the data from collect all the data from collect all the data from all our different all our different all our different sources into one sources into one sources into one specific place. We'll specific place. We'll specific place. We'll start by, start by, start by, you know, pulling up you know, pulling up you know, pulling up our traces so our traces so our traces so you can get all you can get all you can get all the information about what the information about what the information about what your agent was doing. You will your agent was doing. You will your agent was doing. You will pull up your pull up your pull up your grades to see grades to see grades to see exactly where things went exactly where things went exactly where things went wrong. You will add wrong. You will add wrong. You will add annotations from people annotations from people annotations from people who who who can theoretically provide a little can theoretically provide a little can theoretically provide a little more information more information more information about what can be about what can be about what can be fixed and how to fixed and how to fixed and how to improve the work. You improve the work. You improve the work. You load your load your load your actual APM actual APM traces, traces, traces, like, "Hey, is like, "Hey, is like, "Hey, is the system up or down?"

the system up or down?" . You load . You load . You load your codebase to your codebase to your codebase to figure out, "Okay, figure out, "Okay, figure out, "Okay, based on this based on this based on this specific area specific area specific area where the where the where the tool call is failing, tool call is failing, tool call is failing, what changes can I what changes can I what changes can I make to the code so that we're make to the code so that we're make to the code so that we're taking the right taking the right taking the right paths, calling the paths, calling the paths, calling the right tools, right tools, right tools, and fixing the broken ones?" and fixing the broken ones?" and fixing the broken ones?" And you bring it all together in And you bring it all together in And you bring it all together in one final one final one final space, which we space, which we space, which we call our call our call our coding agent, coding agent, coding agent, whether whether whether you use you use you use Cloud Code, Cursor, or Codex—it Cloud Code, Cursor, or Codex—it Cloud Code, Cursor, or Codex—it doesn't matter. You doesn't matter. You doesn't matter. You can plug can plug can plug all this data into it and it all this data into it and it all this data into it and it will automatically do will automatically do will automatically do all this work for you. all this work for you. I know this may I know this may sound really complicated, I sound really complicated, I sound really complicated, I promise. This probably doesn't promise. This probably doesn't promise. This probably doesn't make things any make things any make things any easier on the surface, easier on the surface, easier on the surface, but it's but it's but it's actually very easy to set up. actually very easy to set up. actually very easy to set up. It only takes a few It only takes a few It only takes a few steps—just steps—just steps—just set up your set up your set up your C Alliance skills and you're C Alliance skills and you're C Alliance skills and you're ready to go. You ready to go. You ready to go. You get access to a get access to a get access to a bunch of information and bunch of information and bunch of information and different tools different tools different tools that your agent can that your agent can that your agent can use, use, use, like, "Hey, you like, "Hey, you like, "Hey, you can can can instrument your instrument your instrument your code." This means you code." This means you code." This means you can stream can stream can stream all traces from all traces from all traces from your agent to your agent to any platform you any platform you any platform you choose, such as choose, such as choose, such as Arise. You can Arise. You can Arise. You can download these download these download these traces, traces, traces, create create create datasets, run datasets, run datasets, run experiments, and experiments, and experiments, and evaluate. You evaluate. You evaluate. You get the full get the full get the full range of capabilities range of capabilities range of capabilities directly in directly in directly in your coding agent.

your coding agent. So what I'm going to do now is we're going to look at a we're going to look at a we're going to look at a specific problem, and specific problem, and specific problem, and I want to hear some thoughts from I want to hear some thoughts from I want to hear some thoughts from this side of the room to this side of the room to this side of the room to get some ideas: " get some ideas: " Hey, what do you think Hey, what do you think Hey, what do you think might be the real might be the real might be the real cause?" So, I'll give cause?" So, I'll give cause?" So, I'll give you a little context. you a little context. you a little context. This agent here is a This agent here is a simple financial simple financial simple financial agent. I asked him, agent. I asked him, agent. I asked him, "Should I "Should I "Should I invest in this invest in this invest in this company?" I think company?" I think company?" I think I chose Tata Motors, well, I chose Tata Motors, well, I chose Tata Motors, well, anyway. anyway. anyway. In the end, he gave me In the end, he gave me In the end, he gave me the answer: "Yes, it's worth it the answer: "Yes, it's worth it ." I don't think it's ." I don't think it's ." I don't think it's directly reflected in the directly reflected in the directly reflected in the screenshot, but I promise screenshot, but I promise , he told me so. , he told me so. , he told me so. Um, and ultimately, this is Um, and ultimately, this is Um, and ultimately, this is not what we not what we not what we want to do at all, want to do at all, want to do at all, since I am not since I am not since I am not licensed to provide licensed to provide licensed to provide financial financial financial advice, nor is advice, nor is advice, nor is this agent, and I don't want to this agent, and I don't want to this agent, and I don't want to get into legal get into legal get into legal trouble. If you trouble. If you trouble. If you can help can help can help me figure it out, me figure it out, me figure it out, maybe, "Hey, what do maybe, "Hey, what do maybe, "Hey, what do you think might be the you think might be the you think might be the problem with this problem with this problem with this agent?" So she agent?" So she agent?" So she said, "Hey, maybe said, "Hey, maybe the agent needs to the agent needs to the agent needs to ask a lot ask a lot ask a lot more counter more counter more counter questions to questions to questions to get a little get a little get a little more information more information more information before he can before he can before he can give a concrete give a concrete give a concrete answer." This is a very answer." This is a very answer." This is a very good assumption. Does good assumption. Does good assumption. Does anyone have any other anyone have any other anyone have any other ideas? OK. And how do ideas? OK. And how do ideas? OK. And how do you think we could you think we could you think we could improve it? By the way, improve it? By the way, improve it? By the way, he said the same thing, he said the same thing, he said the same thing, quite similarly: he should quite similarly: he should quite similarly: he should ask more ask more ask more questions before questions before questions before making such a making such a making such a recommendation. But how do recommendation. But how do recommendation. But how do you think we you think we you think we could do that?

could do that? could do that? More context, More context, More context, more questions, more questions, more questions, more details, etc. more details, etc. more details, etc. Okay, so maybe Okay, so maybe Okay, so maybe we should change we should change we should change the prompts to add the prompts to add the prompts to add context and get context and get context and get more information. more information. more information. Anyone else? Anyone else? Anyone else? Where is the validation? Where is the validation? Where is the validation? Sorry? Sorry? Sorry? Where is the validation? Where is the validation? Where is the validation? Where is the validation? Yes, that's Where is the validation? Yes, that's Where is the validation? Yes, that's true. We don't have true. We don't have true. We don't have any objective any objective any objective data for comparison. data for comparison. data for comparison. We don't know if We don't know if We don't know if any of this is any of this is any of this is correct. So, correct. So, correct. So, there is no check to see there is no check to see there is no check to see if any of if any of if any of the answers are the answers are the answers are correct. You are correct. You are correct. You are right. There could be right. There could be right. There could be a lot of different a lot of different a lot of different issues here, and issues here, and issues here, and honestly...Oh yeah, honestly...Oh yeah, honestly...Oh yeah, go ahead. go ahead. go ahead. One question: does One question: does One question: does the agent know about the the agent know about the user's financial situation? user's financial situation? That's a great question. That's a great question. That's a great question. In this case—no, he In this case—no, he In this case—no, he doesn't know. And yes, he doesn't know. And yes, he doesn't know. And yes, he definitely needs to definitely needs to definitely needs to get a lot get a lot get a lot more information more information more information about this. Uh, he about this. Uh, he about this. Uh, he asked, by the way: does asked, by the way: does asked, by the way: does the agent know any the agent know any the agent know any information about the information about the user's financial situation? Um, and yes , he doesn't know that for sure , he doesn't know that for sure , he doesn't know that for sure , there are many , there are many , there are many aspects of this aspects of this aspects of this agent that can be agent that can be agent that can be fixed, and fixed, and fixed, and really, there really really, there really really, there really isn't one isn't one isn't one right answer right answer , is there? But I honestly , is there? But I honestly , is there? But I honestly don't even know what the don't even know what the don't even know what the right answer is right answer is right answer is to this, because this to this, because this to this, because this agent is still not agent is still not agent is still not working correctly right now, and working correctly right now, and working correctly right now, and we're going to we're going to we're going to try to fix try to fix try to fix that today in this that today in this that today in this demo. So demo. So demo. So what we're going to what we're going to what we're going to do is do is do is set up a set up a set up a command command command line interface (CLI) and skills line interface (CLI) and skills line interface (CLI) and skills so that our so that our so that our coding agent can coding agent can coding agent can understand what's going understand what's going understand what's going wrong, download wrong, download wrong, download traces, traces, traces, analyze the analyze the analyze the evaluations performed and evaluations performed and evaluations performed and existing annotations, existing annotations, existing annotations, perhaps do our perhaps do our perhaps do our own analysis, and own analysis, and own analysis, and suggest ideas suggest ideas suggest ideas for improving the for improving the for improving the agent. For example, that

agent. For example, that agent. For example, that he ask more he ask more he ask more questions or have questions or have questions or have restrictions so that he does not restrictions so that he does not restrictions so that he does not give financial give financial give financial advice where it is advice where it is advice where it is inappropriate. So there are inappropriate. So there are inappropriate. So there are many different many different many different ways he ways he ways he can make these can make these can make these corrections. To corrections. To corrections. To start, I'll start, I'll start, I'll open it now...let's see. open it now...let's see. Do I have it here? Do I have it here? Yes. Let's Yes. Let's Yes. Let's open a new open a new open a new session. Ahem. Okay, session. Ahem. Okay, session. Ahem. Okay, great. I'm in the great. I'm in the great. I'm in the right place. right place. right place. OK. So, OK. So, OK. So, usually when you usually when you usually when you want to set up CLI want to set up CLI want to set up CLI and skills, it's very and skills, it's very and skills, it's very easy, easy, easy, just a few just a few just a few steps. Sorry. Oh, steps. Sorry. Oh, steps. Sorry. Oh, there it is. I know what I'm there it is. I know what I'm there it is. I know what I'm doing, guys. Ah, here doing, guys. Ah, here doing, guys. Ah, here it is. So let's start by it is. So let's start by it is. So let's start by configuring the CLI. You configuring the CLI. You configuring the CLI. You just need to just need to just need to copy this line of copy this line of copy this line of code to get everything code to get everything code to get everything set up. So, I set up. So, I set up. So, I go back to my go back to my go back to my terminal and terminal and terminal and install this. Of course install this. Of course , I installed it just , I installed it just , I installed it just before this, but before this, but before this, but the setup is very the setup is very the setup is very simple and quick. simple and quick. simple and quick. Then you create Then you create Then you create your profile, which your profile, which your profile, which will help connect will help connect will help connect it to where it to where it to where your data is stored your data is stored your data is stored . In this case . In this case . In this case it's "araise" and you it's "araise" and you it's "araise" and you create a profile create a profile create a profile so it can so it can so it can download these download these download these traces. And traces. And traces. And visualize and visualize and visualize and get that get that get that information from those information from those information from those assessments and annotations.

assessments and annotations. assessments and annotations. Once the profile Once the profile Once the profile is created, you is created, you is created, you proceed to proceed to proceed to set up your skills set up your skills . You can do this in a . You can do this in a . You can do this in a variety of ways, variety of ways, variety of ways, depending on your depending on your depending on your encoding agent. encoding agent. encoding agent. Today we're Today we're Today we're using Claude using Claude using Claude code, so I code, so I code, so I used used used the marketplace to the marketplace to the marketplace to set it up, but it's set it up, but it's set it up, but it's pretty easy to do pretty easy to do pretty easy to do either way. either way. Now let's Now let's ask him...And ask him...And ask him...And again, again, again, I warn you, I'm I warn you, I'm I warn you, I'm using the using the using the dangerous dangerous dangerous permission skip option. permission skip option. permission skip option. Please don't do Please don't do Please don't do this. This is actually this. This is actually this. This is actually not a good practice not a good practice . This is just for . This is just for . This is just for demonstration so that demonstration so that demonstration so that I don't have to I don't have to I don't have to keep clicking " keep clicking " yes". So we'll yes". So we'll yes". So we'll turn it over to Claude now to turn it over to Claude now to turn it over to Claude now to review these review these review these traces. I'll try traces. I'll try traces. I'll try to find the query I to find the query I to find the query I made earlier: made earlier: made earlier: use this use this use this skill and download skill and download skill and download the trace from that the trace from that the trace from that specific agent. specific agent. Find out what Find out what problems there are that problems there are that problems there are that we can identify from we can identify from we can identify from any...as I any...as I any...as I said before, right? What said before, right? What said before, right? What problems exist problems exist problems exist based on any based on any based on any assessments or annotations assessments or annotations assessments or annotations we have made. we have made. we have made. Conduct your own Conduct your own Conduct your own analysis and tell me what analysis and tell me what analysis and tell me what assessments I could assessments I could assessments I could create in the create in the create in the future to future to future to avoid encountering avoid encountering avoid encountering these problems again. This these problems again. This these problems again. This will take a few minutes.

will take a few minutes. In the meantime, I'll show In the meantime, I'll show you what these you what these you what these traces look like and how you can traces look like and how you can traces look like and how you can start sorting them start sorting them start sorting them if you want to if you want to if you want to do it manually. do it manually. But while that's running in the But while that's running in the background, background, background, let's get back let's get back let's get back to the actual agent to the actual agent to the actual agent that has the requests that has the requests that has the requests we were making. So, we'll we were making. So, we'll we were making. So, we'll ask him a ask him a ask him a pretty similar pretty similar pretty similar question. Let's say, question. Let's say, question. Let's say, should I should I should I invest invest invest in, for example, Nvidia in, for example, Nvidia in, for example, Nvidia this time? This is a slightly this time? This is a slightly this time? This is a slightly better known promotion. better known promotion. better known promotion. We will get a We will get a We will get a response and he response and he response and he will try to give us will try to give us will try to give us some information. some information. some information. As this data As this data As this data comes in, it seems like comes in, it seems like comes in, it seems like I've run this before. I've run this before. I've run this before. You see them You see them You see them loading. From the loading. From the loading. From the previous launch, previous launch, previous launch, we can already see we can already see we can already see what it has done. You what it has done. You what it has done. You see our query, see our query, see our query, you see the result you see the result you see the result it generates, and it generates, and it generates, and in a minute we'll in a minute we'll in a minute we'll see how see how see how similar it is to the similar it is to the similar it is to the result of the result of the result of the previous run previous run . You can also . You can also . You can also see all the steps see all the steps see all the steps that that that your agent actually took, your agent actually took, your agent actually took, right? You can right? You can right? You can understand: we had a understand: we had a understand: we had a supervisory agent and a supervisory agent and a supervisory agent and a financial financial financial data agent. You can data agent. You can data agent. You can see LLM challenges.

see LLM challenges. see LLM challenges. You can see all You can see all You can see all the calls of different the calls of different the calls of different tools. You tools. You tools. You get get get a lot of information right away a lot of information right away a lot of information right away that you can that you can that you can use, as use, as use, as we did on the we did on the we did on the opening slide, opening slide, opening slide, to see where the to see where the to see where the problems might be. problems might be. problems might be. Well, let's see if Well, let's see if Well, let's see if Claude has finished the job, shall we? It Claude has finished the job, shall we? It Claude has finished the job, shall we? It looks like it's still looks like it's still looks like it's still working. He is working. He is working. He is minding his own minding his own minding his own business. Let's give him business. Let's give him business. Let's give him time, you know. This, in time, you know. This, in time, you know. This, in fact, is the fact, is the fact, is the point. Um, while it's being point. Um, while it's being point. Um, while it's being processed, does processed, does processed, does anyone have any questions anyone have any questions anyone have any questions about how to about how to about how to set up set up set up scoring, scoring, scoring, tracing, or anything tracing, or anything tracing, or anything like that that you'd like that that you'd like that that you'd like to know like to know like to know more about? Yes. more about? Yes. more about? Yes. Can you Can you Can you tell us a little tell us a little tell us a little more about how the more about how the more about how the tracks work? In tracks work? In tracks work? In the assistant, or do we the assistant, or do we the assistant, or do we place it place it place it directly in the directly in the directly in the logs? logs? Excellent question. Yes , look what it , look what it , look what it does—it does—it does—it downloads a lot of downloads a lot of downloads a lot of these traces to your these traces to your these traces to your local system. local system. So, let me So, let me show you how it show you how it show you how it actually accesses the actually accesses the system itself. When you system itself. When you give it access give it access give it access to these tools, to these tools, to these tools, you are effectively giving you are effectively giving you are effectively giving it access to the CLI under it access to the CLI under it access to the CLI under the hood. The the hood. The the hood. The implication is that you'll have implication is that you'll have implication is that you'll have all these different CLIs all these different CLIs all these different CLIs that it can that it can that it can integrate with. So, integrate with. So, integrate with. So, you can you can you can customize and customize and customize and find find find datasets. You can datasets. You can datasets. You can download download download datasets. You can datasets. You can datasets. You can have it have it have it interact with the interact with the user interface where it user interface where it can configure the can configure the platform's assessments on your platform's assessments on your behalf. You can behalf. You can behalf. You can continuously continuously continuously monitor when monitor when monitor when your agents your agents your agents fail, create fail, create fail, create datasets, and datasets, and datasets, and run run run experiments. It

experiments. It experiments. It can do can do can do almost anything you almost anything you almost anything you could do on the could do on the could do on the platform. Sometimes it platform. Sometimes it platform. Sometimes it downloads files, downloads files, downloads files, sometimes it makes API sometimes it makes API requests, but that's the requests, but that's the requests, but that's the general idea. Excellent general idea. Excellent general idea. Excellent question. Do you have any more question. Do you have any more question. Do you have any more questions about questions about questions about CLI tracing? Okay, CLI tracing? Okay, CLI tracing? Okay, let's see let's see let's see what Claude did. Oh, he's what Claude did. Oh, he's what Claude did. Oh, he's still working. So, as still working. So, as still working. So, as we can see, let's we can see, let's we can see, let's go through what go through what go through what Claude managed to do. Claude managed to do. We see that we have We see that we have received the spans. He received the spans. He received the spans. He digs deep into digs deep into digs deep into the assessment to the assessment to the assessment to find out exactly where find out exactly where find out exactly where the problems are occurring. the problems are occurring. the problems are occurring. It seems he found It seems he found It seems he found one very serious one very serious one very serious problem with numerical problem with numerical problem with numerical validity. I validity. I validity. I remember someone remember someone remember someone mentioning this before as a mentioning this before as a mentioning this before as a potential problem potential problem potential problem or something related to or something related to or something related to it. We also see it. We also see it. We also see that some tool calls that some tool calls fail directly. Ahem. And look, we And look, we also see that also see that also see that we get a lot of we get a lot of we get a lot of different errors. Oh, different errors. Oh, different errors. Oh, it looks like he just it looks like he just it looks like he just finished. As you can see, finished. As you can see, finished. As you can see, we got a complete we got a complete we got a complete list of what's going list of what's going list of what's going wrong with our agent and wrong with our agent and wrong with our agent and what we can do what we can do what we can do to fix it.

to fix it. to fix it. So while it's So while it's So while it's wrapping this up, you wrapping this up, you wrapping this up, you can see can see can see one of the most one of the most one of the most disturbing disturbing disturbing things here: about 44% of things here: about 44% of our interactions with an our interactions with an our interactions with an agent result agent result agent result in a bad outcome. in a bad outcome. This is really bad. I did This is really bad. I did n't know that. So, n't know that. So, n't know that. So, we definitely we definitely we definitely need to need to need to fix this. Another fix this. Another fix this. Another big problem—oh, I did big problem—oh, I did big problem—oh, I did n't want to n't want to n't want to scroll the scroll the scroll the page. Another page. Another page. Another big problem big problem big problem is that we is that we is that we get a lot of get a lot of get a lot of made-up answers made-up answers , which is just as bad , which is just as bad , which is just as bad as incorrect as incorrect as incorrect generalization. It seems like generalization. It seems like generalization. It seems like it might be it might be it might be related, or it might related, or it might related, or it might not. We can not. We can not. We can try to try to try to fix this. We also fix this. We also fix this. We also see a relatively see a relatively see a relatively high level of high level of high level of over-calling of over-calling of over-calling of tools. This is a waste of tools. This is a waste of tools. This is a waste of our our our time, resources, and time, resources, and time, resources, and money. Obviously, we money. Obviously, we money. Obviously, we make many make many make many mistakes that need to be mistakes that need to be mistakes that need to be corrected. And that's just based corrected. And that's just based corrected. And that's just based on the estimates on the estimates on the estimates we've already set up. From we've already set up. From we've already set up. From Claude's own analysis, Claude's own analysis, Claude's own analysis, he can even he can even he can even figure out, "Hey, there are figure out, "Hey, there are figure out, "Hey, there are other other other problems." "We problems." "We problems." "We get hidden get hidden get hidden instrument failures."

instrument failures." instrument failures." He tells us that everything He tells us that everything He tells us that everything worked out, but ultimately worked out, but ultimately worked out, but ultimately doesn't provide any doesn't provide any doesn't provide any real answers. real answers. real answers. We get a We get a We get a situation where the agent situation where the agent situation where the agent hallucinates when the hallucinates when the hallucinates when the tool tool tool fails, instead of fails, instead of fails, instead of reporting it reporting it . Again, this is . Again, this is . Again, this is related, but it related, but it related, but it means that we means that we means that we may need to not may need to not may need to not just make assessments just make assessments just make assessments to understand when the to understand when the to understand when the tool is tool is tool is failing, but also make failing, but also make failing, but also make changes to the prompts so that the changes to the prompts so that the changes to the prompts so that the agent is not hallucinating. agent is not hallucinating. And there are a lot of different And there are a lot of different things that it things that it things that it will tell you about, and also assessments will tell you about, and also assessments will tell you about, and also assessments that I can create. that I can create. So what we can So what we can do is do is do is tell it, "Let's tell it, "Let's tell it, "Let's create the first create the first create the first point, which is the estimate of the point, which is the estimate of the point, which is the estimate of the hidden failure when hidden failure when the tool is successfully executed, and the tool is successfully executed, and fix the numerical fix the numerical fix the numerical validity so that validity so that validity so that it always relies it always relies it always relies on real data and doesn't on real data and doesn't on real data and doesn't make up these numbers." make up these numbers." "But we're also "But we're also going to going to going to test this on a test this on a test this on a dataset, on the dataset, on the dataset, on the same segments that same segments that same segments that initially caused the initially caused the initially caused the crash, using crash, using crash, using another Arise skill that's going to be another Arise skill that's going to be another Arise skill that's going to be here: the Arise skill here: the Arise skill here: the Arise skill for working with for working with for working with datasets, Arise, oh, here it is." The datasets, Arise, oh, here it is." The Arise Evaluator skill, as well Arise Evaluator skill, as well as the as the as the Arise Experimenter skill.

Arise Experimenter skill. So, here he can... let me let me let me just run this. just run this. just run this. Let it work. And Let it work. And Let it work. And create estimates. create estimates. create estimates. OK. So, again OK. So, again OK. So, again , I was pretty , I was pretty , I was pretty vague, but vague, but vague, but basically I'm just telling basically I'm just telling basically I'm just telling him: make the changes him: make the changes him: make the changes you recommended, and you recommended, and you recommended, and also run these also run these also run these experiments for us. experiments for us. experiments for us. So now we can So now we can So now we can compare to where compare to where compare to where it failed to it failed to it failed to make sure make sure make sure it now passes it now passes it now passes that test. Yes? So in that test. Yes? So in that test. Yes? So in this scenario we this scenario we this scenario we see: okay, it will see: okay, it will see: okay, it will go through the whole process and go through the whole process and go through the whole process and spend time on it. spend time on it. spend time on it. Um, I'll show you a Um, I'll show you a Um, I'll show you a pre-launch, because pre-launch, because again, this is again, this is again, this is going to take time, and while we going to take time, and while we going to take time, and while we want to let it want to let it want to let it run, I don't run, I don't run, I don't want you want you want you staring at a screen staring at a screen staring at a screen where Claude is just "stuttering where Claude is just "stuttering " or whatever it's " or whatever it's " or whatever it's doing in its free time. doing in its free time. doing in its free time. Um, so let's Um, so let's Um, so let's see where I see where I see where I actually asked actually asked actually asked Alex, or, excuse me. Alex, or, excuse me. Claude, to code Claude, to code the answers to some of the answers to some of the answers to some of our previous our previous our previous problems that I problems that I problems that I encountered. One of them encountered. One of them encountered. One of them was, was, was, "Hey, I've had "Hey, I've had "Hey, I've had times where I've been times where I've been times where I've been trying to get trying to get trying to get answers about stocks answers about stocks answers about stocks that aren't necessarily from the that aren't necessarily from the that aren't necessarily from the United States." And United States." And United States." And so we have so we have so we have errors when we errors when we errors when we talk about talk about talk about South Korea, or South Korea, or South Korea, or they all seem to be they all seem to be they all seem to be based on based on based on South Korea. But South Korea. But South Korea. But let's assume there let's assume there let's assume there are other answers, are other answers, are other answers, maybe Japan or maybe Japan or maybe Japan or China or whatever, China or whatever, China or whatever, right? And you want right? And you want right? And you want to see, you want to see, you want to see, you want to understand, "Hey, I want to be to understand, "Hey, I want to be to understand, "Hey, I want to be able to able to able to compare stocks across compare stocks across compare stocks across different markets." And different markets." And different markets." And people are clearly asking about it people are clearly asking about it people are clearly asking about it , but we do , but we do , but we do n't support it.

n't support it. n't support it. And that's what Claude did: And that's what Claude did: And that's what Claude did: he found the he found the he found the areas where areas where areas where this problem existed. As I this problem existed. As I this problem existed. As I said, it seems like said, it seems like said, it seems like everything here was just about everything here was just about everything here was just about South Korea. But South Korea. But South Korea. But he found these he found these he found these areas. He created a areas. He created a areas. He created a dataset for me that dataset for me that dataset for me that I I I can test with can test with can test with to make sure to make sure to make sure everything is working. And then everything is working. And then everything is working. And then he conducted he conducted he conducted experiments. He experiments. He experiments. He made several changes, made several changes, made several changes, starting with the starting with the starting with the initial version initial version —the baseline, which was —the baseline, which was —the baseline, which was buggy, which, as buggy, which, as buggy, which, as we expected, we expected, we expected, resulted in 0% resulted in 0% success rates success rates , as was the case when , as was the case when , as was the case when we received the we received the we received the initial working initial working initial working traces. He also traces. He also traces. He also made a quick made a quick made a quick fix by fix by fix by normalizing the data normalizing the data normalizing the data by adding the by adding the by adding the country name. It didn't country name. It didn't country name. It didn't work. Why? work. Why? work. Why? Because the tool Because the tool Because the tool I called I called I called was also outdated was also outdated was also outdated and I needed to and I needed to and I needed to update to a update to a update to a stable version. stable version. stable version. So he did a third So he did a third So he did a third iteration, and you can iteration, and you can iteration, and you can see that in see that in see that in this chart as well. Where we this chart as well. Where we this chart as well. Where we were able to see, "Hey were able to see, "Hey , the first iteration didn't , the first iteration didn't , the first iteration didn't work." We work." We work." We went back, made went back, made went back, made corrections, and eventually corrections, and eventually corrections, and eventually reached the final reached the final reached the final iteration, where we have iteration, where we have iteration, where we have about 90% success.

about 90% success. about 90% success. Of course, this is not 100%. Of course, this is not 100%. Of course, this is not 100%. We still need to We still need to We still need to make some changes, make some changes, make some changes, but we are doing but we are doing but we are doing much better than much better than much better than before. So, this is a before. So, this is a before. So, this is a great way to great way to great way to start so you can start so you can start so you can go from "Hey, I go from "Hey, I go from "Hey, I know there's a problem" know there's a problem" know there's a problem" to "here's the solution." to "here's the solution." Now I know that we Now I know that we still have a lot of time. I still have a lot of time. I still have a lot of time. I want to know how you are. want to know how you are. want to know how you are. How is everyone doing? How are How is everyone doing? How are How is everyone doing? How are you feeling? Do you feeling? Do you feeling? Do you have any questions about you have any questions about you have any questions about how you can get how you can get how you can get started on this? Yes. started on this? Yes. Excellent question. Excellent question. So his question So his question So his question was about where was about where the experiments are actually being run? Where are the experiments are actually being run? Where are the assessments performed? the assessments performed? the assessments performed? Where does all this work? Um, Where does all this work? Um, Where does all this work? Um, you have a bit of you have a bit of you have a bit of room for room for room for customization. In this customization. In this customization. In this scenario, I ran scenario, I ran scenario, I ran the experiment on my the experiment on my the experiment on my local machine, which I local machine, which I local machine, which I think will be the think will be the think will be the most common case most common case . And then what I . And then what I . And then what I did, or you did, or you did, or you can put it can put it can put it in the cloud or wherever you in the cloud or wherever you in the cloud or wherever you want to do want to do want to do experiments. And then experiments. And then experiments. And then you can safely you can safely you can safely run assessments run assessments run assessments directly in Arize.

directly in Arize. directly in Arize. So in this case So in this case So in this case I created LLM as a judge, I created LLM as a judge, I created LLM as a judge, which will then allow, which will then allow, which will then allow, let's see let's see let's see if I see the evaluator himself here if I see the evaluator himself here if I see the evaluator himself here . As you can see, . As you can see, . As you can see, we created two different we created two different we created two different estimators: estimators: estimators: code-based and LLM-based. I code-based and LLM-based. I code-based and LLM-based. I asked Claude to do asked Claude to do asked Claude to do it. He created this for it. He created this for it. He created this for me on the platform. And me on the platform. And me on the platform. And this was the final this was the final this was the final option for LLM as a judge, option for LLM as a judge, option for LLM as a judge, because it was not possible because it was not possible because it was not possible to configure a to configure a to configure a code-based evaluator. So code-based evaluator. So code-based evaluator. So we have two of them. So you we have two of them. So you we have two of them. So you see LLM as a judge, see LLM as a judge, see LLM as a judge, he created this for he created this for he created this for us with us with us with this query and this query and this query and ran it so we ran it so we ran it so we could compare, "Hey, did the could compare, "Hey, did the could compare, "Hey, did the agent work agent work agent work correctly?" Are there any more correctly?" Are there any more correctly?" Are there any more questions? By the way, that questions? By the way, that questions? By the way, that was a very good was a very good was a very good question. Yes. Very, question. Yes. Very, question. Yes. Very, very good question. very good question. very good question. Um, the ratings can Um, the ratings can Um, the ratings can be customized however you be customized however you be customized however you like. Of course, like. Of course, like. Of course, people can people can people can use use whatever models they whatever models they whatever models they want. Nobody want. Nobody want. Nobody wants to be limited to wants to be limited to wants to be limited to just Claude or just OpenAI, just Claude or just OpenAI, just Claude or just OpenAI, especially in this day and especially in this day and especially in this day and age where AI models are age where AI models are age where AI models are coming out literally coming out literally coming out literally every day and you want to every day and you want to every day and you want to say, "Okay, now say, "Okay, now say, "Okay, now I want I want I want to use that one." Is to use that one." Is to use that one." Is n't that right? Additionally, n't that right? Additionally, n't that right? Additionally, some people some people some people prefer slightly prefer slightly prefer slightly cheaper models.

cheaper models. cheaper models. Some prefer to Some prefer to Some prefer to make sure their make sure their make sure their ratings are the best ratings are the best ratings are the best because they know that because they know that because they know that the quality of their agents the quality of their agents the quality of their agents depends on the quality of depends on the quality of depends on the quality of the ratings. Therefore, they the ratings. Therefore, they the ratings. Therefore, they use use use the most powerful models the most powerful models the most powerful models for evaluations, even for evaluations, even for evaluations, even if they do not if they do not if they do not use them in a use them in a use them in a working environment. working environment. working environment. So, it depends on the So, it depends on the So, it depends on the situation. You have situation. You have situation. You have full control over full control over full control over the settings. Here the settings. Here the settings. Here you can see a you can see a you can see a complete list of any complete list of any model providers you want model providers you want to connect to, even to connect to, even to connect to, even if they are local if they are local if they are local open open open source models. Yes. Good source models. Yes. Good source models. Yes. Good question. Her question question. Her question question. Her question was, "If I was, "If I was, "If I run a smaller run a smaller run a smaller agent to agent to agent to fix some fix some fix some bugs, how does that bugs, how does that bugs, how does that compare to this?" compare to this?" You don't actually You don't actually have to have to have to use Claude use Claude use Claude code or Codex. You can code or Codex. You can code or Codex. You can use use any agent you any agent you any agent you want. The advantage of want. The advantage of want. The advantage of our solution is that our solution is that you get a you get a you get a skill that allows you skill that allows you skill that allows you to extract to extract to extract information from information from information from traces for traces for traces for making corrections. making corrections. making corrections. You do You do You do n't make corrections just n't make corrections just n't make corrections just because, " because, " Hey, I know this is Hey, I know this is Hey, I know this is wrong." " wrong." " The result The result The result is incorrect." You is incorrect." You is incorrect." You make corrections make corrections make corrections based on, " based on, " Hey, I know there's a Hey, I know there's a Hey, I know there's a mistake here." This is where the mistake here." This is where the mistake here." This is where the failure occurred. That's why it failure occurred. That's why it failure occurred. That's why it happened. Here's how we happened. Here's how we happened. Here's how we can fix it.

can fix it. Because you have not Because you have not only codebase data only codebase data only codebase data , but also information from , but also information from , but also information from APM logs, Jira tickets, APM logs, Jira tickets, APM logs, Jira tickets, etc., as well as your etc., as well as your etc., as well as your Arise traces that Arise traces that Arise traces that show: "Here's the path show: "Here's the path show: "Here's the path that the agent took." " that the agent took." " That's where he That's where he That's where he made a mistake." made a mistake." made a mistake." Your ratings, annotations, Your ratings, annotations, Your ratings, annotations, etc. Any more questions? etc. Any more questions? etc. Any more questions? Yes. Yes. Excellent question. Yes . He asked, "Are . He asked, "Are . He asked, "Are these features available in these features available in these features available in Phoenix?" Phoenix is ​​the Phoenix?" Phoenix is ​​the Phoenix?" Phoenix is ​​the open source version of Arise. open source version of Arise. open source version of Arise. I believe the skills I believe the skills I believe the skills and MCP are also in Phoenix, you and MCP are also in Phoenix, you and MCP are also in Phoenix, you can connect them can connect them can connect them to your agent and he to your agent and he to your agent and he will be able to get the same will be able to get the same will be able to get the same information. information. information. Of course, there are some Of course, there are some Of course, there are some advantages in the advantages in the advantages in the corporate version corporate version , where access to online , where access to online assessments is assessments is assessments is more conveniently organized, they are easier more conveniently organized, they are easier more conveniently organized, they are easier to format and to format and to format and customize. You customize. You customize. You get more get more monitoring capabilities and monitoring capabilities and a few extra a few extra a few extra features, but yes, features, but yes, features, but yes, skills and CLIs can be skills and CLIs can be skills and CLIs can be connected to your connected to your connected to your agent for agent for agent for programming. Yes.

programming. Yes. programming. Yes. In the presentation you In the presentation you In the presentation you showed, the agent showed, the agent showed, the agent said that this was supposedly a said that this was supposedly a said that this was supposedly a bad conclusion. So bad conclusion. So bad conclusion. So how do you classify how do you classify how do you classify something as a bad something as a bad something as a bad conclusion? How does conclusion? How does conclusion? How does the agent know if this is a the agent know if this is a the agent know if this is a good or bad good or bad good or bad outcome? outcome? outcome? A very good A very good A very good question. So her question. So her question. So her question was, question was, question was, "In what I "In what I "In what I showed earlier, showed earlier, showed earlier, there was a score or there was a score or there was a score or a response from Claude a response from Claude a response from Claude that said, 'Hey, that that said, 'Hey, that that said, 'Hey, that was a bad score.' was a bad score.' " How do we know what " How do we know what " How do we know what is a bad and what is a bad and what is a bad and what is a good is a good is a good outcome? In that outcome? In that outcome? In that case, what I case, what I case, what I actually set up actually set up was based on one was based on one was based on one of the estimates I of the estimates I of the estimates I created earlier. And created earlier. And created earlier. And so we can now so we can now so we can now look at this look at this look at this estimator here. We estimator here. We estimator here. We can look at, can look at, can look at, let's see, let's see, let's see, some I don't actually some I don't actually some I don't actually remember which one, remember which one, remember which one, but I'll open this one and but I'll open this one and but I'll open this one and hope it's the hope it's the hope it's the right one. Um, so as right one. Um, so as right one. Um, so as you can see here, this is you can see here, this is you can see here, this is essentially a essentially a essentially a summary score that I summary score that I summary score that I created. It created. It created. It actually analyzes actually analyzes actually analyzes the information that we've the information that we've the information that we've gathered through our gathered through our gathered through our agent, all the agent, all the agent, all the context that we've context that we've context that we've given that given that given that summary agent, and summary agent, and summary agent, and makes sure that it's makes sure that it's makes sure that it's linking to the linking to the linking to the right points, right points, right points, answering answering answering questions questions questions appropriately, and so on. Um, and here's appropriately, and so on. Um, and here's appropriately, and so on. Um, and here's how we determined whether it how we determined whether it how we determined whether it was a good conclusion was a good conclusion was a good conclusion or not.

or not. or not. What if you just What if you just What if you just set the scale set the scale set the scale from the beginning from the beginning from the beginning to get a good to get a good to get a good summary? summary? summary? Uh, what if we Uh, what if we Uh, what if we set the scale set the scale set the scale from the beginning from the beginning from the beginning to see if to see if to see if he could make a good he could make a good he could make a good summary? summary? summary? For example, does it give For example, does it give For example, does it give us a good summary us a good summary us a good summary or not? For example, what or not? For example, what or not? For example, what if you just if you just if you just set up a scale set up a scale set up a scale instead of instead of instead of running raters? running raters? running raters? Yes, great question Yes, great question . So, she asks, . So, she asks, . So, she asks, why can't we why can't we why can't we just set up just set up just set up scales that, you know, scales that, you know, scales that, you know, guarantee that he'll guarantee that he'll guarantee that he'll give a good give a good give a good summary? And yes, summary? And yes, summary? And yes, of course, that's exactly what of course, that's exactly what of course, that's exactly what you want to do. you want to do. you want to do. You want to You want to You want to make sure make sure make sure you update your you update your you update your prompts, your prompts, your prompts, your skills, your skills, your skills, your coding workflows coding workflows coding workflows so that your agents so that your agents so that your agents are delivering the best are delivering the best are delivering the best results. The assessments are results. The assessments are results. The assessments are only needed to only needed to only needed to help help help you identify where you identify where you identify where problems are occurring, problems are occurring, problems are occurring, track those track those track those problems, and problems, and problems, and make sure they make sure they make sure they don't recur. Is don't recur. Is don't recur. Is n't that right? So you've n't that right? So you've n't that right? So you've set up this scale, set up this scale, set up this scale, great. She works for the great. She works for the great. She works for the next 2 months. next 2 months. next 2 months. After 2 months, either because of After 2 months, either because of After 2 months, either because of model regression or model regression or model regression or because people because people because people start asking start asking start asking questions differently, you questions differently, you questions differently, you get different get different get different results, and you don't results, and you don't results, and you don't want your want your want your performance performance performance to suffer because of to suffer because of to suffer because of that. So you need to that. So you need to that. So you need to be able to be able to be able to control and control and control and monitor this. This is the point of monitor this. This is the point of evaluations. Did that evaluations. Did that help? Perfectly. Um, I help? Perfectly. Um, I help? Perfectly. Um, I don't know how much don't know how much don't know how much time we have left, but time we have left, but time we have left, but keep going.

keep going. I'll have a question I'll have a question later, but later, but later, but most of the assessments most of the assessments most of the assessments I've worked with are I've worked with are I've worked with are somewhat focused on somewhat focused on somewhat focused on linting. And as linting. And as linting. And as a result I get a a result I get a a result I get a bunch of linting on top of the bunch of linting on top of the bunch of linting on top of the processes. It annoys me processes. It annoys me processes. It annoys me to work in this to work in this to work in this mode. mode. So, your question So, your question was about whether was about whether was about whether I encountered I encountered I encountered linting issues linting issues linting issues in agents? in agents? in agents? With agents With agents With agents piling up piling up piling up linting as a way... It linting as a way... It seems to me that seems to me that they spend they spend they spend more time more time more time linting than... linting than... Good question. So, Good question. So, it's essentially saying, "Hey, have it's essentially saying, "Hey, have it's essentially saying, "Hey, have you ever had this you ever had this you ever had this problem: you gave problem: you gave problem: you gave Claude access to all Claude access to all Claude access to all these skills, these skills, these skills, traces, and traces, and traces, and information, but information, but information, but in the end, with Claude Code or in the end, with Claude Code or in the end, with Claude Code or other similar other similar other similar coding agents, they coding agents, they coding agents, they tend to tend to tend to focus on focus on focus on linting more than linting more than linting more than other problems." Did I other problems." Did I other problems." Did I understand correctly? understand correctly? understand correctly? Yes, that is a very valid Yes, that is a very valid Yes, that is a very valid observation. That's why observation. That's why observation. That's why we developed these we developed these we developed these skills to skills to skills to help them help them help them focus on the focus on the focus on the real tasks at hand real tasks at hand , right?

, right? They get all the They get all the trace information and trace information and realize, "Hey, realize, "Hey, realize, "Hey, the problem isn't the problem isn't the problem isn't linting," because they can see linting," because they can see linting," because they can see that the that the that the wrong wrong wrong tool is being called. And this tool is being called. And this tool is being called. And this will not happen through will not happen through will not happen through linting. They have some linting. They have some logical thinking skills logical thinking skills to figure this out. So, to figure this out. So, to figure this out. So, frankly, I've frankly, I've frankly, I've hardly seen that hardly seen that hardly seen that when using these when using these when using these skills. Perfectly. So, skills. Perfectly. So, skills. Perfectly. So, how...Oh, yeah. Does how...Oh, yeah. Does how...Oh, yeah. Does the system make it the system make it the system make it easy to understand, easy to understand, easy to understand, excuse me? excuse me? excuse me? How much does it cost to How much does it cost to How much does it cost to run these assessments? run these assessments? run these assessments? Great question. Yes, Great question. Yes, Great question. Yes, his question his question his question was whether it was was whether it was was whether it was easy to understand the easy to understand the easy to understand the cost of performing cost of performing cost of performing these assessments. these assessments. these assessments. Certainly. One of the Certainly. One of the Certainly. One of the best benefits of best benefits of best benefits of running assessments on the running assessments on the running assessments on the platform is platform is platform is the ability to get a the ability to get a the ability to get a full history of what full history of what full history of what they did, they did, they did, plus a nice bonus: plus a nice bonus: plus a nice bonus: you see a you see a you see a small cost here.

small cost here. So you can So you can understand how much understand how much these these assessments are actually costing you, because of assessments are actually costing you, because of assessments are actually costing you, because of course, every time course, every time course, every time you start another you start another you start another LLM, it incurs LLM, it incurs LLM, it incurs additional costs. additional costs. additional costs. So you want So you want So you want to make sure that's to make sure that's to make sure that's not the case. And of course, what's not the case. And of course, what's not the case. And of course, what's really nice about really nice about really nice about creating assessments, whether creating assessments, whether creating assessments, whether it's in the front end or it's in the front end or it's in the front end or through the through the through the coding agent, is you can coding agent, is you can coding agent, is you can set up baseline set up baseline set up baseline requirements, like, " requirements, like, " Hey, we want to Hey, we want to Hey, we want to selectively test selectively test selectively test certain scopes so we don't certain scopes so we don't certain scopes so we don't run this on 100% of run this on 100% of our traces and 100% of our traces and 100% of our calls." You our calls." You our calls." You can customize can customize can customize the focus on specific the focus on specific the focus on specific areas, and of course, areas, and of course, areas, and of course, you can always you can always you can always set this in the set this in the set this in the encoding agent itself or encoding agent itself or encoding agent itself or in the platform itself. in the platform itself. So, just as a little So, just as a little recap of everything recap of everything recap of everything we've we've we've covered today, again covered today, again covered today, again , this is... Oh, that's not it. , this is... Oh, that's not it. , this is... Oh, that's not it. Oh, I think I have it Oh, I think I have it Oh, I think I have it here. Ahem. So, here. Ahem. So, here. Ahem. So, basically, what I wanted basically, what I wanted basically, what I wanted you to take away from you to take away from you to take away from this is—these AI agents are this is—these AI agents are this is—these AI agents are amazing. They will do amazing. They will do amazing. They will do practically anything practically anything practically anything you ask them to do, but you ask them to do, but you ask them to do, but again, it's not again, it's not again, it's not magic. As he said, magic. As he said, magic. As he said, they could very well they could very well they could very well focus on focus on focus on linting. They can get linting. They can get linting. They can get bogged down in the bogged down in the bogged down in the minutiae, but the minutiae, but the minutiae, but the best thing about them is that best thing about them is that best thing about them is that they automate they automate they automate your entire your entire your entire workflow from start to workflow from start to workflow from start to finish. So, does it finish. So, does it finish. So, does it start with start with start with observing where the observing where the observing where the problems lie, problems lie, finding those finding those problems, being able to problems, being able to problems, being able to download them, download them, download them, understand where the understand where the understand where the errors lie, create errors lie, create errors lie, create datasets, datasets, datasets, develop precise develop precise develop precise fixes, fixes, fixes, test them to

test them to test them to make sure they make sure they make sure they work. And finally, work. And finally, work. And finally, you can act you can act you can act as a human checker as a human checker as a human checker to make sure to make sure to make sure he didn't just he didn't just he didn't just go back and go back and go back and fix the fix the fix the linting changes, right? And we linting changes, right? And we linting changes, right? And we can make sure can make sure can make sure that he really made that he really made that he really made changes that helped changes that helped changes that helped improve the agent, improve the agent, improve the agent, not made it not made it not made it worse. This is worse. This is worse. This is the general gist of what the general gist of what the general gist of what I wanted to I wanted to I wanted to show today. Are there any more show today. Are there any more show today. Are there any more questions on this questions on this questions on this before I before I before I maybe show maybe show maybe show the last thing...Oh, he's the last thing...Oh, he's the last thing...Oh, he's still finishing what he still finishing what he still finishing what he was trying to do was trying to do was trying to do earlier. I was hoping earlier. I was hoping earlier. I was hoping he would be he would be he would be done by then. This done by then. This done by then. This took about 14 took about 14 took about 14 minutes. But while he's minutes. But while he's minutes. But while he's wrapping up, are there any more wrapping up, are there any more wrapping up, are there any more questions about how questions about how questions about how we can we can we can use use use our agents to our agents to our agents to automate our automate our automate our workflows? And workflows? And workflows? And for the improvement of for the improvement of for the improvement of your agents. Yes. your agents. Yes.

Excellent question. Excellent question. So his question So his question So his question was, do we was, do we was, do we mostly mostly mostly rely on Claude rely on Claude rely on Claude or any of these or any of these or any of these coding agents coding agents coding agents to to to evaluate for themselves whether evaluate for themselves whether evaluate for themselves whether everything is working well or not? Yes, everything is working well or not? Yes, everything is working well or not? Yes, you can definitely you can definitely you can definitely rely on Claude to rely on Claude to rely on Claude to build all of build all of build all of this. You this. You this. You can also do this can also do this can also do this manually. You manually. You manually. You can also ask Claude can also ask Claude can also ask Claude to perform specific to perform specific to perform specific assessments that you assessments that you assessments that you need. So in this need. So in this need. So in this case, when you case, when you case, when you saw that nice clean saw that nice clean saw that nice clean table, it was table, it was table, it was actually determined by actually determined by actually determined by the estimates that my team and I the estimates that my team and I the estimates that my team and I had set up had set up had set up beforehand. We beforehand. We beforehand. We have analyzed certain have analyzed certain have analyzed certain areas where we areas where we areas where we know there are know there are know there are potential limitations potential limitations potential limitations for most for most for most agents working agents working agents working in this in this in this workflow. And we workflow. And we workflow. And we set them up as set them up as set them up as general, which is why we general, which is why we general, which is why we got a bad got a bad got a bad summary. We had some summary. We had some summary. We had some questions: "Is the questions: "Is the questions: "Is the agent working?" Does it provide agent working?" Does it provide agent working?" Does it provide the right answers, the right answers, the right answers, or are we or are we or are we calling calling calling the tools correctly? We the tools correctly? We the tools correctly? We know that these are know that these are know that these are general directions. general directions. general directions. But what Claude was able to But what Claude was able to But what Claude was able to do was do was do was look at our look at our look at our actual traces and use actual traces and use actual traces and use them them them to understand what to understand what to understand what problems we are problems we are problems we are facing today. facing today. facing today. True? Our agent True? Our agent True? Our agent works in real works in real works in real time. I won't be able to time. I won't be able to time. I won't be able to go through all of this go through all of this go through all of this manually and find manually and find manually and find errors. But Claude can.

errors. But Claude can. errors. But Claude can. Or Claude, Cursor, Codex, or Or Claude, Cursor, Codex, or Or Claude, Cursor, Codex, or whatever, right? whatever, right? whatever, right? And that's one of the And that's one of the And that's one of the biggest superpowers biggest superpowers biggest superpowers you can give to you can give to you can give to these coding agents to do these coding agents to do it for you. I it for you. I see another see another see another question here. Good question here. Good question here. Good question. So his question. So his question. So his question was, is question was, is question was, is there some kind of there some kind of there some kind of testing dataset testing dataset testing dataset that we can that we can that we can look at, or some kind of look at, or some kind of look at, or some kind of visualization that visualization that visualization that would say, "Hey, is this a would say, "Hey, is this a would say, "Hey, is this a better quality LLM for better quality LLM for better quality LLM for evaluation, more expensive evaluation, more expensive evaluation, more expensive or cheaper?" or cheaper?" or cheaper?" Realistically, no. This is something Realistically, no. This is something Realistically, no. This is something you will have to you will have to you will have to do for your own do for your own do for your own analysis. You'll analysis. You'll analysis. You'll have to have to have to figure this out on figure this out on figure this out on your own because, your own because, your own because, ultimately, as I often say, ultimately, as I often say, ultimately, as I often say, if you if you if you use use use standard standard standard evaluation methods for your evaluation methods for your evaluation methods for your agent, you'll get the same agent, you'll get the same agent, you'll get the same standard standard standard results. And that's not what results. And that's not what you need you need you need when you're creating when you're creating when you're creating specialized AI specialized AI agents tailored agents tailored agents tailored specifically to your needs specifically to your needs . And the same goes for . And the same goes for . And the same goes for tests, right? Tests tests, right? Tests are great. They'll are great. They'll are great. They'll show you, "Hey, this is a show you, "Hey, this is a show you, "Hey, this is a good agent." This is a good agent." This is a good agent." This is a good, sorry, this is a good, sorry, this is a good, sorry, this is a good LLM. But what good LLM. But what good LLM. But what you really you really you really need to know is this a need to know is this a need to know is this a good LLM for good LLM for good LLM for me? And it will me? And it will me? And it will depend on your depend on your depend on your specific specific specific use case. So, use case. So, use case. So, if you work in if you work in if you work in finance, finance, finance, you might choose a you might choose a you might choose a specific model.

specific model. specific model. If you're creating If you're creating If you're creating something for law in something for law in something for law in finance, maybe it'll finance, maybe it'll finance, maybe it'll even be a slightly even be a slightly even be a slightly different model, right? different model, right? different model, right? So it really So it really So it really depends on what depends on what depends on what you're you're you're building it for. Steeply. I think that's all I have building it for. Steeply. I think that's all I have building it for. Steeply. I think that's all I have . . . If anyone has any more If anyone has any more If anyone has any more questions, I'll be happy questions, I'll be happy questions, I'll be happy to answer, but yeah, to answer, but yeah, to answer, but yeah, thank you, friends.