Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

My name is Fuad Ali. I My name is Fuad Ali. I work in the work in the Arize AI product team, working on agent experiments and agent tools in general in the agent tools in general in the Arize stack. You just Arize stack. You just Arize stack. You just heard from Lori, who heard from Lori, who heard from Lori, who is also part of our is also part of our is also part of our team here at Arize. He team here at Arize. He team here at Arize. He will walk around the will walk around the will walk around the room. Aparna, our room. Aparna, our room. Aparna, our co-founder and CPO, will co-founder and CPO, will co-founder and CPO, will also be here to also be here to also be here to help you if help you if help you if needed needed needed during this workshop. during this workshop. during this workshop. Today we'll Today we'll Today we'll talk about talk about talk about creating creating creating self-learning cycles for self-learning cycles for self-learning cycles for your agents. your agents. Let me tell you a little Let me tell you a little about myself and about myself and about myself and how I got how I got how I got into this field. I believe into this field. I believe into this field. I believe the concept of the concept of the concept of self-learning loops is self-learning loops is self-learning loops is fairly new to AI fairly new to AI engineering, and Arize has engineering, and Arize has engineering, and Arize has many features that many features that many features that help help help support this. To support this. To support this. To give you some context, I give you some context, I give you some context, I worked in the distributed server space worked in the distributed server space at SpaceX, and it's interesting how we at SpaceX, and it's interesting how we come full circle in come full circle in come full circle in the world of telemetry, the world of telemetry, the world of telemetry, because a rocket system because a rocket system because a rocket system that's constantly that's constantly that's constantly correcting its course every correcting its course every correcting its course every millisecond is millisecond is millisecond is actually consuming actually consuming actually consuming its own telemetry and its own telemetry and its own telemetry and acting on it to acting on it to acting on it to correct itself. So correct itself. So correct itself. So I'll explain how we at Arize I'll explain how we at Arize I'll explain how we at Arize built the idea of built the idea of built the idea of self-learning loops self-learning loops self-learning loops for your agents and for your agents and for your agents and how you can how you can how you can use them use them use them to self-correct.

to self-correct. Perfectly. To tell a little bit Perfectly. To tell a little bit about about about the history of Arize, we the history of Arize, we the history of Arize, we made some made some made some big bets. In big bets. In big bets. In 2023, we were the first to 2023, we were the first to 2023, we were the first to enter the market with the enter the market with the enter the market with the LLM model as a judge. LLM model as a judge. LLM model as a judge. We created the first We created the first We created the first set of semantic set of semantic set of semantic inventions for LLM inventions for LLM applications with open applications with open applications with open source inference. source inference. source inference. We built the first We built the first real-time AI data warehouse, ADB. In real-time AI data warehouse, ADB. In 2025, we released 2025, we released 2025, we released agent ratings and Alex agent ratings and Alex —our own —our own —our own internal agent internal agent internal agent that helps you with that helps you with that helps you with AI engineering. Imagine a AI engineering. Imagine a AI engineering. Imagine a Cursor for AI Cursor for AI engineering. And in 2026, the engineering. And in 2026, the engineering. And in 2026, the next big next big next big bet we bet we bet we want to make is want to make is want to make is to create agents that to create agents that to create agents that self-correct self-correct self-correct through through through feedback loops. feedback loops. I think everyone understands I think everyone understands how how how powerful this will be for the development of the powerful this will be for the development of the powerful this will be for the development of the agent concept agent concept agent concept in general. Ahem. So I in general. Ahem. So I in general. Ahem. So I really think the really think the really think the software engineering industry is software engineering industry is evolving quite a bit. There used to be a classic used to be a classic "Software 1.0": an app, "Software 1.0": an app, "Software 1.0": an app, tracking in Datadog, and tracking in Datadog, and tracking in Datadog, and some kind of some kind of some kind of observation platform that observation platform that observation platform that people read. Then a people read. Then a people read. Then a human developer human developer human developer writes code to writes code to writes code to fix the errors. I fix the errors. I fix the errors. I think 2.0 is when think 2.0 is when think 2.0 is when people interact with people interact with people interact with it through it through it through prompts. So, it's the prompts. So, it's the prompts. So, it's the same thing, but instead of same thing, but instead of same thing, but instead of writing code, you writing code, you writing code, you write a prompt that write a prompt that write a prompt that corrects the behavior corrects the behavior corrects the behavior that you're observing in that you're observing in that you're observing in your application. We your application. We your application. We move on to the move on to the move on to the third phase, which is a third phase, which is a third phase, which is a completely autonomous completely autonomous completely autonomous cycle. In this loop, cycle. In this loop, cycle. In this loop, your app creates your app creates your app creates telemetry and

telemetry and telemetry and observability events, and observability events, and observability events, and your code writing agent reads them your code writing agent reads them your code writing agent reads them . It's . It's . It's no longer just a person no longer just a person no longer just a person reading data and reading data and reading data and entering a prompt. If entering a prompt. If entering a prompt. If any of you any of you any of you use use use tools like Cursor tools like Cursor , Cloud Code , or Datadog CLI , you , Cloud Code , or Datadog CLI , you , Cloud Code , or Datadog CLI , you already see part of already see part of already see part of this process. this process. this process. The gap is that The gap is that The gap is that you still probably have to you still probably have to you still probably have to prompt prompt prompt it to read it to read it to read information from a backend, information from a backend, information from a backend, such as Datadog. And I such as Datadog. And I such as Datadog. And I think when you think when you think when you move to the needs of, move to the needs of, move to the needs of, say, thousands of say, thousands of say, thousands of agents in a stack, it agents in a stack, it agents in a stack, it becomes virtually impossible for humans to becomes virtually impossible for humans to process it. So I'll process it. So I'll talk a little bit about the talk a little bit about the talk a little bit about the new features in Arise new features in Arise new features in Arise that we've released to that we've released to that we've released to support this support this support this workflow. I workflow. I workflow. I want you to want you to want you to focus on the fact focus on the fact focus on the fact that observability that observability that observability needs to evolve from needs to evolve from human-readable dashboards to software human-readable dashboards to software interfaces that interfaces that interfaces that agents consume, agents consume, agents consume, work on, and work on, and work on, and can can can self-reflect on. self-reflect on.

So, some evidence of So, some evidence of market success: Arise market success: Arise market success: Arise has processed over 1 has processed over 1 has processed over 1 trillion spans. We trillion spans. We trillion spans. We conduct over 60 conduct over 60 conduct over 60 million assessments per million assessments per million assessments per month. We have month. We have month. We have over 3 million over 3 million over 3 million downloads per downloads per downloads per month. We are trusted by month. We are trusted by month. We are trusted by many cool teams, many cool teams, many cool teams, like Spotify, Reddit, Uber, Trip like Spotify, Reddit, Uber, Trip like Spotify, Reddit, Uber, Trip Advisor. This is to make sure Advisor. This is to make sure Advisor. This is to make sure you know: I'm you know: I'm you know: I'm not just "vibe- not just "vibe- coding" here and telling you coding" here and telling you coding" here and telling you how it's supposed to work. how it's supposed to work. how it's supposed to work. Ahem. I think if Ahem. I think if Ahem. I think if you look at you look at you look at the industry, it is the industry, it is the industry, it is also moving in this also moving in this also moving in this direction. This direction. This direction. This applies not only to applies not only to applies not only to these teams, but also to those these teams, but also to those these teams, but also to those who are not even who are not even who are not even Arise customers, like Arise customers, like Arise customers, like Ramp Labs. They Ramp Labs. They Ramp Labs. They move on to the idea of move on to the idea of move on to the idea of creating creating self-improving self-improving and self-healing software and self-healing software . I emphasize this . I emphasize this to show that the industry is to show that the industry is to show that the industry is indeed moving in indeed moving in indeed moving in this direction. It's not this direction. It's not this direction. It's not just us. You just us. You just us. You need agents who need agents who need agents who can detect can detect can detect a problem, investigate a problem, investigate a problem, investigate it, and implement it, and implement it, and implement a fix in one a fix in one a fix in one cycle. This is exactly the cycle. This is exactly the cycle. This is exactly the cycle I'm going to cycle I'm going to cycle I'm going to show you show you show you today. Perfectly.

today. Perfectly. today. Perfectly. So, the main So, the main So, the main problem I want to problem I want to problem I want to outline: the mistakes outline: the mistakes outline: the mistakes that cost you that cost you that cost you money are invisible at money are invisible at money are invisible at launch. This is by launch. This is by launch. This is by definition true, because definition true, because definition true, because if you noticed them, if you noticed them, if you noticed them, they wouldn't they wouldn't they wouldn't cost you anything. You cost you anything. You cost you anything. You would have already implemented would have already implemented would have already implemented the fix. And I the fix. And I the fix. And I think, really, what think, really, what think, really, what I want to I want to I want to focus on is that focus on is that focus on is that your agent your agent your agent is working in the is working in the is working in the experimental phase. It experimental phase. It experimental phase. It works with the grades works with the grades works with the grades you set up. you set up. you set up. You don't actually You don't actually You don't actually know all the estimates know all the estimates know all the estimates that need to be set that need to be set that need to be set before launch, before launch, before launch, otherwise you would have already otherwise you would have already otherwise you would have already written them down. And something written them down. And something written them down. And something breaks, some breaks, some breaks, some edge case that edge case that edge case that your team your team your team just didn't know about. just didn't know about. just didn't know about. For example, in this For example, in this For example, in this example, the shopper example, the shopper example, the shopper is looking for a is looking for a is looking for a birthday gift under birthday gift under birthday gift under $30. They $30. They $30. They get a list of get a list of get a list of products that are more expensive than products that are more expensive than $30, $50. And then $30, $50. And then $30, $50. And then in a productive in a productive in a productive environment, you see an environment, you see an environment, you see an agent confidently agent confidently agent confidently buying the wrong product, buying the wrong product, buying the wrong product, always doing so with always doing so with always doing so with extreme confidence extreme confidence , because they , because they , because they never test the product. never test the product. never test the product. And they, again, And they, again, And they, again, don't appear during the don't appear during the don't appear during the testing phase, testing phase, testing phase, they appear they appear they appear when users when users when users complain. They complain. They complain. They appear when you appear when you appear when you receive a receive a receive a support ticket, when you are support ticket, when you are support ticket, when you are called to called to called to duty at 3 am duty at 3 am duty at 3 am . And this . And this . And this gap—it works in gap—it works in gap—it works in testing, it doesn't work testing, it doesn't work testing, it doesn't work in production, something goes in production, something goes in production, something goes wrong—is what I'm trying to wrong—is what I'm trying to wrong—is what I'm trying to close here. Class.

close here. Class. close here. Class. So, really quickly, to So, really quickly, to So, really quickly, to back up our back up our back up our conversation with an example, conversation with an example, conversation with an example, we have a thing we have a thing we have a thing called the Wonder called the Wonder called the Wonder Toys Assistant. This is a shopping Toys Assistant. This is a shopping Toys Assistant. This is a shopping assistant, a very assistant, a very assistant, a very simple application, simple application, simple application, has several has several has several tools for tools for tools for searching for products, searching for products, searching for products, getting getting getting product details, getting product details, getting product details, getting information information information needed for a needed for a needed for a specific product specific product , purchasing a product, , purchasing a product, , purchasing a product, checking checking checking order status, and order status, and order status, and canceling canceling canceling an order. We have the an order. We have the an order. We have the OpenAI Agents SDK running OpenAI Agents SDK running OpenAI Agents SDK running on top of this, a backend on top of this, a backend on top of this, a backend on FastAPI, search with on FastAPI, search with on FastAPI, search with vector search and vector search and vector search and text text text fallback, and it's all fallback, and it's all fallback, and it's all tracked in Arize, tracked in Arize, tracked in Arize, so you can see so you can see so you can see the truth about what exactly the truth about what exactly the truth about what exactly this agent is doing in this agent is doing in this agent is doing in production. And just production. And just production. And just to give you to give you to give you a little a little a little preview of what this preview of what this preview of what this agent is, it's very agent is, it's very agent is, it's very simple, you see the web simple, you see the web interface for Wonder Toys, interface for Wonder Toys, interface for Wonder Toys, and I'll zoom in a little bit so and I'll zoom in a little bit so and I'll zoom in a little bit so you can see. Ahem. you can see. Ahem. I can click " ask about product". ask about product".

Ask the agent to buy the product, and don't to buy the product, and don't worry, worry, worry, no no no real credit real credit real credit cards are cards are cards are used in this demonstration. used in this demonstration. Class. So, that Class. So, that gives us a little bit of context for the gives us a little bit of context for the gives us a little bit of context for the example we're going to example we're going to talk about today. And I talk about today. And I think the main thing here is that think the main thing here is that think the main thing here is that we're going to go through we're going to go through we're going to go through a few paths to a few paths to a few paths to explain some of the new explain some of the new explain some of the new features in Arize. But features in Arize. But features in Arize. But it's actually going to be it's actually going to be it's actually going to be four commands and four commands and four commands and then a bit of clicking then a bit of clicking then a bit of clicking around the around the around the user interface. You user interface. You user interface. You clone the repository clone the repository , and don't worry, I'll , and don't worry, I'll , and don't worry, I'll come back to this come back to this come back to this slide so you have slide so you have slide so you have plenty of time plenty of time plenty of time to work through everything. You to work through everything. You to work through everything. You just need to just need to just need to clone clone clone the repository, navigate the repository, navigate the repository, navigate to it, add your to it, add your to it, add your API keys, run npm API keys, run npm API keys, run npm run dev, and tracing run dev, and tracing run dev, and tracing will be set up after will be set up after will be set up after you get your you get your you get your API key from Arise and API key from Arise and API key from Arise and expose your OpenAI API key expose your OpenAI API key expose your OpenAI API key to run the to run the to run the OpenAI Agent SDK. Well, that's OpenAI Agent SDK. Well, that's OpenAI Agent SDK. Well, that's pretty much it, and once pretty much it, and once pretty much it, and once you get going, you get going, you get going, you'll see that you'll see that you'll see that traces will traces will traces will automatically start automatically start automatically start coming into Arise. Cool coming into Arise. Cool . So, we'll move on . So, we'll move on . So, we'll move on to how we can to how we can to how we can start breaking this start breaking this start breaking this thing, but the specific thing, but the specific thing, but the specific bug I added to bug I added to bug I added to this repository is this repository is this repository is around the idea of around the idea of around the idea of price caps.

price caps. Most requests Most requests will work and you will work and you will work and you can experiment on your own can experiment on your own can experiment on your own , , , but if you request a but if you request a but if you request a birthday gift birthday gift birthday gift under under under $30, you will receive $30, you will receive $30, you will receive items more expensive than items more expensive than items more expensive than $30. It is very $30. It is very $30. It is very common to have an edge common to have an edge common to have an edge case where a certain case where a certain case where a certain barrier has failed, and barrier has failed, and barrier has failed, and without the context of the without the context of the without the context of the repository code and the repository code and the repository code and the context of the traces, it is context of the traces, it is context of the traces, it is very difficult to find very difficult to find very difficult to find such cases in an such cases in an such cases in an opaque system opaque system opaque system that simply outputs that simply outputs that simply outputs telemetry data. telemetry data. The agent will not mark this The agent will not mark this as an error, it will as an error, it will as an error, it will confidently return a confidently return a confidently return a list of completely list of completely list of completely inappropriate inappropriate inappropriate products. How do you write products. How do you write products. How do you write an estimate for a an estimate for a an estimate for a limiting case limiting case limiting case you haven't seen before? you haven't seen before? The first of the new features we have in Arise we have in Arise we have in Arise to address this to address this to address this issue is called issue is called issue is called Signal. Signal is a ready- Signal. Signal is a ready- Signal. Signal is a ready- to-use agent that to-use agent that to-use agent that runs on the runs on the runs on the Arise platform in a sandbox and Arise platform in a sandbox and Arise platform in a sandbox and can be enabled with a can be enabled with a can be enabled with a single toggle in the single toggle in the project settings. Signal is constantly project settings. Signal is constantly working with your working with your working with your work work work traces in the traces in the traces in the background. It background. It background. It will read each will read each will read each incoming trace, incoming trace, incoming trace, group failures by group failures by group failures by root cause, and root cause, and root cause, and remove duplicates remove duplicates remove duplicates for recurring for recurring for recurring errors. Since it is an errors. Since it is an errors. Since it is an agent with a full agent with a full agent with a full set of tools set of tools , it can access , it can access , it can access external external external services. You can services. You can services. You can connect Signal to connect Signal to connect Signal to your your your GitHub repository. You can GitHub repository. You can GitHub repository. You can connect Signal to connect Signal to connect Signal to data sources like data sources like data sources like DataDog. Signal

DataDog. Signal DataDog. Signal will work with your will work with your will work with your trace data, trace data, trace data, which is the source of truth which is the source of truth which is the source of truth about your about your about your agents' performance, and agents' performance, and agents' performance, and analyze the analyze the analyze the agent code you agent code you agent code you run. And run. And run. And it will access it will access it will access external external external data sources, such as DataDog. And data sources, such as DataDog. And data sources, such as DataDog. And he can actually he can actually he can actually conduct conduct conduct an investigation, an investigation, an investigation, determine the type of determine the type of determine the type of seriousness seriousness seriousness of the incident, gather of the incident, gather of the incident, gather evidence, and then evidence, and then evidence, and then even write and even write and even write and open a PR for you. open a PR for you. So, going back to the So, going back to the beginning of my beginning of my beginning of my presentation, where I presentation, where I presentation, where I talked about the evolution of talked about the evolution of talked about the evolution of software engineering software engineering . We are actually . We are actually . We are actually moving into the moving into the moving into the third phase, where third phase, where third phase, where the agent consumes the agent consumes the agent consumes its own telemetry its own telemetry its own telemetry data from the Observe platform. data from the Observe platform. data from the Observe platform. It then analyzes It then analyzes It then analyzes its code and its code and its code and tries to tries to tries to suggest suggest suggest fixes based on the fixes based on the evidence it receives. Class. So, evidence it receives. Class. So, this is the first part—Signal this is the first part—Signal . The second part . The second part . The second part concerns the concept of concerns the concept of concerns the concept of skills. skills.

We will have a workshop later today where We will have a workshop later today where we will we will we will go into detail on all go into detail on all go into detail on all the skills we have the skills we have the skills we have released. But released. But released. But maybe you don't want maybe you don't want maybe you don't want this agent to be this agent to be this agent to be managed through Rise. managed through Rise. managed through Rise. Maybe you just Maybe you just Maybe you just want to be able to want to be able to want to be able to retrieve data from retrieve data from retrieve data from your storage. So we your storage. So we your storage. So we also introduced also introduced also introduced the concept of the concept of the concept of Rise skills. Rise skills Rise skills. Rise skills Rise skills. Rise skills enable your enable your enable your coding agent, coding agent, coding agent, whether whether whether you use you use you use Cursor, Cloud Code, CodeX, or Cursor, Cloud Code, CodeX, or any of the 40+ any of the 40+ any of the 40+ agents like Open Coder agents like Open Coder agents like Open Coder or Meese, to work more or Meese, to work more or Meese, to work more efficiently. This efficiently. This efficiently. This allows you to start allows you to start allows you to start requesting requesting requesting traces, and a good traces, and a good traces, and a good example of this, I example of this, I example of this, I think, is...Give me a think, is...Give me a think, is...Give me a second. I can second. I can second. I can ask my ask my ask my coding agents. This is Cloud coding agents. This is Cloud coding agents. This is Cloud Code, but I can Code, but I can Code, but I can task my task my task my agents to read agents to read agents to read my traces when my traces when my traces when I'm in the repository so I'm in the repository so I'm in the repository so they can see what the they can see what the they can see what the agent is doing based on the agent is doing based on the trace data, trace data, while also seeing the while also seeing the while also seeing the code itself and making really code itself and making really code itself and making really actionable fixes.

actionable fixes. Class. So, there are Class. So, there are a lot of skills. We a lot of skills. We a lot of skills. We have the skills to have the skills to have the skills to create create create fault-based datasets fault-based datasets . We have the skills to . We have the skills to . We have the skills to read traces read traces read traces and analyze errors in and analyze errors in and analyze errors in them, as well as to them, as well as to them, as well as to run evaluators run evaluators run evaluators and perform tasks and perform tasks and perform tasks directly on the directly on the directly on the platform. So the platform. So the platform. So the first part of the first part of the skills workflow is skills workflow is that Cloud Code can that Cloud Code can that Cloud Code can write the patch write the patch write the patch for you because it's for you because it's for you because it's in your in your in your repository, similar repository, similar repository, similar to Signal. Signal works to Signal. Signal works to Signal. Signal works on top of the system. It on top of the system. It on top of the system. It has access to your has access to your has access to your GitHub repository. He GitHub repository. He GitHub repository. He can independently can independently can independently submit the PR for you. submit the PR for you. These two paths are kind of These two paths are kind of equivalent. equivalent. equivalent. You have Cloud Code with You have Cloud Code with You have Cloud Code with Arise skills for Arise skills for Arise skills for tracing. This is something like an tracing. This is something like an tracing. This is something like an "on- "on- "on- demand" method. Let's say demand" method. Let's say demand" method. Let's say you're an engineer who's you're an engineer who's you're an engineer who's constantly working on constantly working on constantly working on your product, your product, your product, sitting down and trying to sitting down and trying to sitting down and trying to improve it, improve it, improve it, unlike Signal, which unlike Signal, which unlike Signal, which is an always- is an always- is an always- on, on, on, continuous method continuous method . This constantly provides . This constantly provides . This constantly provides you with new insights, new you with new insights, new you with new insights, new analysis based on analysis based on analysis based on your your your trace data. Uh, and Signal, trace data. Uh, and Signal, trace data. Uh, and Signal, as I said, can as I said, can as I said, can actually determine actually determine actually determine the root cause the root cause the root cause because it has access because it has access because it has access to your to your to your code repository. It code repository. It code repository. It can compile an can compile an can compile an investigation report and investigation report and investigation report and even try to even try to even try to create a create a create a merge request (PR), and we merge request (PR), and we merge request (PR), and we look not just at look not just at look not just at individual traces, but individual traces, but individual traces, but at the whole context of what at the whole context of what your agent is doing your agent is doing your agent is doing in production. So, in production. So, in production. So, quickly, because I want to quickly, because I want to quickly, because I want to get to the heart of this

get to the heart of this get to the heart of this presentation. I'm going to presentation. I'm going to presentation. I'm going to pause and leave this pause and leave this pause and leave this slide. As I said, slide. As I said, slide. As I said, we have Lori and Aparna in the hall we have Lori and Aparna in the hall we have Lori and Aparna in the hall , they , they , they will help you all will help you all will help you all get set up. get set up. Aparna is here. Aparna is here. It seems like Lori is somewhere at the It seems like Lori is somewhere at the It seems like Lori is somewhere at the other end of the hall, other end of the hall, other end of the hall, but he will come to but he will come to but he will come to you. And Dat from you. And Dat from you. And Dat from our team is also with us. So, our team is also with us. So, our team is also with us. So, really quickly: these are really quickly: these are really quickly: these are literally four literally four literally four commands you commands you commands you need to execute. I need to execute. I need to execute. I would recommend would recommend would recommend signing up for a signing up for a signing up for a free account free account free account at arise.com to get a at arise.com to get a at arise.com to get a Rise API key and start Rise API key and start Rise API key and start seeing seeing seeing trace streams. For this trace streams. For this trace streams. For this presentation, you will presentation, you will presentation, you will also need an also need an OpenAI API key. So I'll OpenAI API key. So I'll OpenAI API key. So I'll pause, leave the slide, and pause, leave the slide, and pause, leave the slide, and we'll start to we'll start to we'll start to figure out how to figure out how to figure out how to get a trace get a trace get a trace for the Wonder Toy Signal. And then for the Wonder Toy Signal. And then for the Wonder Toy Signal. And then we'll walk through we'll walk through how this cycle how this cycle how this cycle gets into production. gets into production. So, yes. And if So, yes. And if anyone needs a key, anyone needs a key, anyone needs a key, Dat will gladly give it to you Dat will gladly give it to you Dat will gladly give it to you . So, how are . So, how are . So, how are things progressing?

things progressing? And if you don't have And if you don't have a laptop, don't a laptop, don't a laptop, don't worry. I'll show you worry. I'll show you worry. I'll show you what this what this what this process looks like if you were process looks like if you were process looks like if you were doing it yourself. I'll give doing it yourself. I'll give you another 2-3 you another 2-3 minutes to minutes to minutes to set up API keys, set up API keys, set up API keys, run these commands, run these commands, run these commands, and clone the and clone the and clone the repository. I know the repository. I know the repository. I know the Wi-Fi is Wi-Fi is Wi-Fi is a little spotty today, a little spotty today, a little spotty today, so I won't blame you for that so I won't blame you for that so I won't blame you for that . . Oh no, you will. You Oh no, you will. You still get still get still get access to the evaluators, access to the evaluators, access to the evaluators, there are just a few there are just a few there are just a few features like Cody features like Cody features like Cody Valls and so on. Yes. And in Valls and so on. Yes. And in Valls and so on. Yes. And in this process, we will this process, we will this process, we will mainly show mainly show mainly show how Cloud Code can how Cloud Code can how Cloud Code can perform perform perform trace analysis through trace analysis through trace analysis through skills and Signal. So skills and Signal. So skills and Signal. So it's not necessary for this it's not necessary for this it's not necessary for this . Is anyone . Is anyone . Is anyone having problems? Does having problems? Does having problems? Does anyone need anyone need anyone need help? Raise your help? Raise your help? Raise your hand and we will come hand and we will come hand and we will come to you. Sorry. Here, I'll to you. Sorry. Here, I'll to you. Sorry. Here, I'll go down now.

go down now. Sorry. Oh, okay. Sorry. Oh, okay. Cool, then it's probably a Cool, then it's probably a Wi-Fi problem. So, Wi-Fi problem. So, Wi-Fi problem. So, if you have problems with if you have problems with if you have problems with Wi-Fi, let me know when you Wi-Fi, let me know when you Wi-Fi, let me know when you can see can see can see the repository. This may the repository. This may the repository. This may take a little time, take a little time, take a little time, but in general, this is the but in general, this is the but in general, this is the structure of structure of structure of the repository. We have an the repository. We have an the repository. We have an agent, all the agent, all the agent, all the scripts he needs, and a scripts he needs, and a scripts he needs, and a bunch of other things. I'll give bunch of other things. I'll give bunch of other things. I'll give you a few more you a few more you a few more seconds because I know the Wi-Fi seconds because I know the Wi-Fi seconds because I know the Wi-Fi is a little is a little is a little slow. Once you slow. Once you slow. Once you get to get to get to the repository, the repository, the repository, please don't forget please don't forget please don't forget to get an API key. to get an API key. Also get your Also get your OpenAI API key so we OpenAI API key so we OpenAI API key so we can get started. Should can get started. Should I continue? And

I continue? And I remind you, all you I remind you, all you I remind you, all you need to do is need to do is need to do is run these few run these few run these few commands. So, you commands. So, you commands. So, you git clone the repository, and git clone the repository, and git clone the repository, and then then then CD into it. So, I'm CD into it. So, I'm CD into it. So, I'm already inside. Then already inside. Then already inside. Then run NPM install, and run NPM install, and run NPM install, and make sure you make sure you make sure you have uploaded your have uploaded your have uploaded your .env file. This is where you .env file. This is where you .env file. This is where you will enter your will enter your will enter your three keys. Of course, here three keys. Of course, here three keys. Of course, here at the end. Wait, at the end. Wait, at the end. Wait, sorry. Which one exactly? Oh, sorry. Which one exactly? Oh, sorry. Which one exactly? Oh, the one at the end. the one at the end. For example, I understood. I can't change it now , but we'll , but we'll , but we'll figure it out. Steeply.

figure it out. Steeply. Okay, I'll leave this on Okay, I'll leave this on the screen for another 10 seconds. the screen for another 10 seconds. the screen for another 10 seconds. It's very simple, It's very simple, It's very simple, make sure you make sure you make sure you enter the values ​​in your enter the values ​​in your enter the values ​​in your .env file. There might be .env file. There might be .env file. There might be an error there, but just an error there, but just an error there, but just check the .env file and check the .env file and check the .env file and insert your own keys. insert your own keys. To show you an To show you an example. I'll take it off example. I'll take it off example. I'll take it off so you can't see my so you can't see my so you can't see my keys, but there are your keys, but there are your keys, but there are your keys there. And then keys there. And then keys there. And then run npm install and npm run npm install and npm run npm install and npm run dev. Then you will see run dev. Then you will see that your interactions with that your interactions with that your interactions with Wonder Toys, that is, with this Wonder Toys, that is, with this Wonder Toys, that is, with this chatbot, chatbot, chatbot, are tracked in Arize. In are tracked in Arize. In are tracked in Arize. In Arize traces, you will Arize traces, you will Arize traces, you will be able to see everything be able to see everything be able to see everything that comes through that comes through that comes through this chatbot, right? Here I this chatbot, right? Here I this chatbot, right? Here I see that the agent workflow has started see that the agent workflow has started . Then the . Then the Wonder Toys agent responded, Wonder Toys agent responded, Wonder Toys agent responded, searched for the products, searched for the products, searched for the products, took another step, took another step, took another step, found them and returned them to found them and returned them to found them and returned them to me, I can see the me, I can see the me, I can see the conclusion of what conclusion of what conclusion of what this agent is doing. It's a kind of this agent is doing. It's a kind of this agent is doing. It's a kind of basic value: basic value: basic value: can I track can I track can I track and understand what's and understand what's and understand what's happening? And now the happening? And now the happening? And now the second part I second part I second part I wanted to show is wanted to show is wanted to show is what Signal can do what Signal can do what Signal can do on top of all of this.

on top of all of this. on top of all of this. So you have all this So you have all this So you have all this trace data trace data trace data coming through Arize. And what coming through Arize. And what coming through Arize. And what you you you should really start doing should really start doing should really start doing is enabling Signal to is enabling Signal to is enabling Signal to run run run on top of your agents. on top of your agents. And I'll show you an And I'll show you an example of Signal working in a example of Signal working in a example of Signal working in a repository. Signal repository. Signal repository. Signal runs constantly in the runs constantly in the runs constantly in the background. background. background. Once Once Once trace data starts trace data starts trace data starts arriving, it takes arriving, it takes arriving, it takes a few minutes to a few minutes to a few minutes to start up. So, I'll show start up. So, I'll show start up. So, I'll show you—we have a little bit of you—we have a little bit of you—we have a little bit of slow Wi-Fi here, but I'll slow Wi-Fi here, but I'll slow Wi-Fi here, but I'll show you an example of Signal already show you an example of Signal already show you an example of Signal already set up and set up and set up and ready to go ready to go ready to go , and how it , and how it , and how it does a does a does a self-improvement cycle. self-improvement cycle. As I mentioned As I mentioned earlier, the earlier, the earlier, the trace data comes trace data comes trace data comes through Arize. Your every through Arize. Your every through Arize. Your every interaction with the app interaction with the app interaction with the app is recorded. This is what the is recorded. This is what the Signal interface looks like. Signal performs Signal interface looks like. Signal performs cluster analysis of the cluster analysis of the cluster analysis of the various issues various issues various issues it detects in your it detects in your it detects in your app. You will see a app. You will see a app. You will see a table table table categorizing these categorizing these categorizing these problems. You can problems. You can problems. You can observe the observe the observe the dynamics of these dynamics of these dynamics of these problems over time. And problems over time. And problems over time. And then you can then you can then you can look at examples of what's look at examples of what's look at examples of what's actually actually actually happening, right?

happening, right? happening, right? In this example, In this example, In this example, we see problems with we see problems with we see problems with serialization in Python. serialization in Python. And this leads to And this leads to corruption of the corruption of the corruption of the agent's input data. You agent's input data. You agent's input data. You will see that he has already will see that he has already will see that he has already created a created a created a merge request (PR) for me. merge request (PR) for me. merge request (PR) for me. So I can So I can So I can go to go to go to this PR with one click and start this PR with one click and start this PR with one click and start viewing it. There is also an viewing it. There is also an viewing it. There is also an idea to create a idea to create a idea to create a ticket based on ticket based on ticket based on this research. You'll this research. You'll this research. You'll see an overview of see an overview of see an overview of what this agent does, what this agent does, what this agent does, such as inferring such as inferring such as inferring results, a hypothesis results, a hypothesis results, a hypothesis about the causes, evidence in a about the causes, evidence in a about the causes, evidence in a separate trace, separate trace, separate trace, and a and a and a suggested suggested suggested fix that you fix that you fix that you can apply can apply can apply yourself. yourself. Returning to Returning to the concept of agents the concept of agents the concept of agents using using using telemetry data: telemetry data: telemetry data: this agent runs in this agent runs in this agent runs in the background, processes data the background, processes data , prepares evidence, and , prepares evidence, and , prepares evidence, and offers you solutions. offers you solutions. In this example, we In this example, we still left the still left the still left the code review to a code review to a code review to a human. You human. You human. You decide how far you decide how far you decide how far you want to go in these want to go in these want to go in these matters. We matters. We matters. We recommend that you recommend that you recommend that you do the same. But you can do the same. But you can do the same. But you can see the idea of see the idea of self-healing software in self-healing software in action. How are you doing with the action. How are you doing with the action. How are you doing with the setup? Did setup? Did setup? Did everyone manage everyone manage everyone manage to connect to Wi-Fi? Did to connect to Wi-Fi? Did to connect to Wi-Fi? Did you succeed? No? Okay you succeed? No? Okay , great. I'll continue , great. I'll continue , great. I'll continue my presentation, and my presentation, and my presentation, and at the end, we'll at the end, we'll at the end, we'll maybe take some time maybe take some time so I can walk so I can walk so I can walk around the room and around the room and around the room and help you. So, I help you. So, I help you. So, I wanted to go over wanted to go over wanted to go over four steps with you, and the four steps with you, and the four steps with you, and the first one is discovery.

first one is discovery. first one is discovery. That is, detecting a That is, detecting a That is, detecting a backup backup backup mechanism error before the mechanism error before the user reports it. Second: user reports it. Second: Based on this error Based on this error Based on this error and the estimates and the estimates and the estimates running on your running on your running on your platform, you need to platform, you need to platform, you need to investigate what is investigate what is investigate what is happening happening happening using real using real using real traces. In our traces. In our traces. In our example, there was an example, there was an example, there was an error while error while error while checking the price. The checking the price. The Signal system and your Signal system and your skills through Cloud Code skills through Cloud Code skills through Cloud Code were able to review were able to review were able to review the trace, the trace, the trace, understand the understand the understand the user's request within user's request within user's request within the trace, the trace, the trace, for example, about a price for example, about a price for example, about a price up to $30, up to $30, up to $30, combine this with combine this with combine this with an analysis of the code that an analysis of the code that an analysis of the code that checks the price checks the price checks the price filter, and filter, and filter, and offer you offer you offer you a fix. There a fix. There a fix. There was a bug in my code in was a bug in my code in was a bug in my code in how the how the how the price price price filter worked. filter worked. Using Signal, Using Signal, tracing data in Arize, tracing data in Arize, tracing data in Arize, and skills through Cloud and skills through Cloud and skills through Cloud Code, I was able to see this Code, I was able to see this Code, I was able to see this interaction with a real interaction with a real interaction with a real user and then user and then user and then fix this fix this fix this incorrect filter incorrect filter incorrect filter in the agent code. And in the agent code. And in the agent code. And finally, we have the finally, we have the finally, we have the verification stage. I wasn't verification stage. I wasn't verification stage. I wasn't planning on covering planning on covering planning on covering this today because it this today because it this today because it requires a bit of requires a bit of requires a bit of setup, but I'm setup, but I'm setup, but I'm happy to talk about happy to talk about happy to talk about it, and we'll definitely it, and we'll definitely it, and we'll definitely come back to it come back to it come back to it when everything is fine with Wi-Fi when everything is fine with Wi-Fi when everything is fine with Wi-Fi . So, once again, a . So, once again, a . So, once again, a quick summary of the quick summary of the quick summary of the main points.

main points. main points. We have an idea for a main We have an idea for a main We have an idea for a main loop running on a loop running on a loop running on a real agent: it real agent: it real agent: it detects an age detects an age detects an age filtering bug filtering bug filtering bug , investigates it using , investigates it using Arize tracing or via Signal, and Arize tracing or via Signal, and fixes the problem, fixes the problem, fixes the problem, with Claude actually with Claude actually with Claude actually patching the fallback patching the fallback patching the fallback mechanism. And then mechanism. And then mechanism. And then verification, and I'll verification, and I'll verification, and I'll explain how it explain how it explain how it works in Arize using works in Arize using assessment tools and agent experiments. Signal agent experiments. Signal can also do this can also do this can also do this in parallel when you in parallel when you in parallel when you don't want to work don't want to work don't want to work at the keyboard and at the keyboard and at the keyboard and want to run the want to run the want to run the process in the background process in the background process in the background . Signal will be . Signal will be . Signal will be customized customized customized to your to your to your own taxonomy own taxonomy own taxonomy and your tracking. and your tracking. and your tracking. You can also You can also You can also set it to set it to set it to send notifications send notifications . So, for example, . So, for example, . So, for example, if you have if you have if you have an alert for a certain an alert for a certain an alert for a certain score that needs to score that needs to score that needs to stay below, stay below, stay below, say, a 1% say, a 1% say, a 1% trigger level, you can trigger level, you can trigger level, you can set the alarm to set the alarm to trigger on the trigger on the alert, or, alert, or, alert, or, for example, on a for example, on a for example, on a spike in latency or spike in latency or spike in latency or costs, or something costs, or something costs, or something like that. You can like that. You can like that. You can configure your own configure your own Harness referee tools to Harness referee tools to run in CI run in CI run in CI for these PRs. You can for these PRs. You can for these PRs. You can experiment with experiment with experiment with them and then them and then them and then optimize your workflows in some way optimize your workflows in some way based on the results. And based on the results. And another quick another quick another quick announcement: at 2:00 PM, Lori announcement: at 2:00 PM, Lori announcement: at 2:00 PM, Lori will be hosting a workshop, will be hosting a workshop, will be hosting a workshop, hopefully the Wi-Fi hopefully the Wi-Fi hopefully the Wi-Fi will be working a little will be working a little will be working a little better, where we'll better, where we'll better, where we'll delve into some of delve into some of delve into some of these workflows in detail.

these workflows in detail. So, back to the So, back to the main point, I mentioned main point, I mentioned main point, I mentioned this cycle. I think this cycle. I think this cycle. I think we all understand the we all understand the we all understand the big picture: big picture: big picture: I have an application, it's I have an application, it's I have an application, it's being monitored, this being monitored, this being monitored, this trace data trace data trace data comes in, and on top comes in, and on top comes in, and on top of that, whether it's of that, whether it's of that, whether it's a signal or your a signal or your a signal or your own own own coding agents, having coding agents, having coding agents, having trace data, which trace data, which trace data, which is the source of truth about is the source of truth about is the source of truth about what your what your what your agent is doing, and the agent is doing, and the agent is doing, and the actual actual actual repository data, which is the repository data, which is the repository data, which is the code that defines what the code that defines what the code that defines what the agent should do, agent should do, agent should do, allows for allows for allows for really interesting really interesting really interesting analysis and the ability analysis and the ability analysis and the ability to make to make to make improvements at improvements at improvements at scale. I think there's scale. I think there's scale. I think there's one more thing we have one more thing we have one more thing we have n't talked about yet, and I n't talked about yet, and I n't talked about yet, and I want to touch on want to touch on want to touch on how we evaluate how we evaluate how we evaluate agents with the help of agents with the help of agents with the help of other agents. This is other agents. This is other agents. This is kind of an example: kind of an example: kind of an example: get a trace, get a trace, get a trace, look at my code, look at my code, look at my code, and figure out what's wrong. and figure out what's wrong. and figure out what's wrong. When you want to When you want to When you want to get more get more get more adaptive analysis, we adaptive analysis, we adaptive analysis, we tend to think of tend to think of tend to think of it as Harness as a judge. it as Harness as a judge. it as Harness as a judge. So, raise your hands, So, raise your hands, So, raise your hands, who among you is familiar with the who among you is familiar with the who among you is familiar with the concept of an LLM as a judge, concept of an LLM as a judge, concept of an LLM as a judge, in the general in the general in the general sense. Steeply. sense. Steeply. sense. Steeply. Perfectly. I think almost Perfectly. I think almost Perfectly. I think almost everyone in the room should already everyone in the room should already everyone in the room should already be familiar with this. This is a be familiar with this. This is a be familiar with this. This is a very common very common very common pattern: you provide pattern: you provide pattern: you provide input, your LLM input, your LLM input, your LLM reads it as a judge, whether reads it as a judge, whether reads it as a judge, whether it's a it's a it's a generalization task, or a generalization task, or a generalization task, or a hallucination assessment, hallucination assessment, hallucination assessment, or something else, and then or something else, and then or something else, and then provides a score on provides a score on provides a score on how well the how well the how well the agent handled agent handled agent handled the response. We are the response. We are the response. We are moving towards a moving towards a moving towards a "Harness as a Judge" approach, and there "Harness as a Judge" approach, and there "Harness as a Judge" approach, and there are several reasons for this.

are several reasons for this. First, LLM as a judge is First, LLM as a judge is First, LLM as a judge is really a fixed really a fixed really a fixed rubric. So when rubric. So when rubric. So when you have a specific case you have a specific case you have a specific case and an assessment that you and an assessment that you and an assessment that you want to get a want to get a want to get a result from, this result from, this result from, this works well for identifying works well for identifying works well for identifying known issues. known issues. known issues. However, as I just However, as I just However, as I just explained with the explained with the explained with the price filter example, you price filter example, you price filter example, you don't always know don't always know don't always know what these what these what these issues will be before going into issues will be before going into issues will be before going into production. Therefore, Harness as a production. Therefore, Harness as a production. Therefore, Harness as a judge is a way to judge is a way to judge is a way to conduct such conduct such conduct such adaptive analysis. adaptive analysis. adaptive analysis. So, observe what the So, observe what the agent is actually doing and agent is actually doing and adjust what you are adjust what you are adjust what you are testing as testing as testing as it runs. it runs. it runs. Thus, the criteria seem to Thus, the criteria seem to Thus, the criteria seem to emerge from the emerge from the emerge from the trajectory of actions itself. trajectory of actions itself. trajectory of actions itself. Was it the right Was it the right Was it the right choice of tool? choice of tool? choice of tool? Were the arguments Were the arguments Were the arguments valid? Was it valid? Was it valid? Was it justified by the justified by the justified by the correct correct correct context context context conveyed by another agent, conveyed by another agent, conveyed by another agent, for example? This idea for example? This idea for example? This idea of ​​"unknown of ​​"unknown of ​​"unknown unknowns" and that unknowns" and that unknowns" and that price filter was an price filter was an price filter was an example of something example of something example of something I might not have I might not have I might not have written a test case for, written a test case for, written a test case for, but Harness as a judge but Harness as a judge but Harness as a judge is able to perform is able to perform is able to perform such adaptive such adaptive such adaptive analysis for me through analysis for me through trace analysis. So, it's trace analysis. So, it's like a like a like a simple language test. You don't simple language test. You don't simple language test. You don't need to create need to create need to create column mappings, column mappings, column mappings, write template code, write template code, write template code, or evaluate or evaluate or evaluate templates and texts.

templates and texts. Criteria—it's just Criteria—it's just one sentence, it one sentence, it one sentence, it reads the trace tree reads the trace tree directly, and directly, and then runs in a then runs in a then runs in a sandbox on sandbox on sandbox on real data. And I real data. And I real data. And I think it gives you a think it gives you a think it gives you a bunch of extra bunch of extra bunch of extra tools tools tools because it also because it also because it also works in a specific works in a specific works in a specific environment. One environment. One environment. One example I often example I often example I often see with many see with many see with many clients involves clients involves clients involves access verification. LLM access verification. LLM access verification. LLM as a judge can read as a judge can read as a judge can read input and input and input and output, but since output, but since output, but since it has no other it has no other it has no other data, unless you data, unless you data, unless you insert each insert each insert each user of the system user of the system user of the system into the prompt, it into the prompt, it into the prompt, it doesn't really know doesn't really know doesn't really know the context of what the context of what the context of what the user is supposed to the user is supposed to the user is supposed to do or do or do or have access to. Whereas have access to. Whereas have access to. Whereas Harness as a referee, Harness as a referee, Harness as a referee, because it can be because it can be because it can be connected to connected to connected to external systems, external systems, external systems, can make a query and can make a query and can make a query and actually check: “ actually check: “ Hey, is Hey, is Hey, is this user allowed to this user allowed to this user allowed to do this? Does do this? Does the trace reflect the the trace reflect the access they had access they had ? So Harness as a judge ? So Harness as a judge ? So Harness as a judge essentially moves essentially moves essentially moves from simply calling a from simply calling a from simply calling a fundamental fundamental fundamental model that evaluates model that evaluates model that evaluates input and produces an input and produces an input and produces an output, to a more output, to a more output, to a more agent-based approach where agent-based approach where agent-based approach where you can call you can call you can call tools and tools and tools and discover those same " discover those same " unknown unknowns."

unknown unknowns." Steeply. So, we talked a little bit Steeply. So, we talked a little bit about Harness about Harness about Harness as a judge. How it would as a judge. How it would as a judge. How it would work in this work in this work in this scenario: evaluating scenario: evaluating scenario: evaluating what should have what should have what should have led to the error led to the error led to the error and deciding and deciding and deciding whether to pass the example whether to pass the example whether to pass the example or not. For example, or not. For example, or not. For example, confirm that the price of confirm that the price of confirm that the price of each returned each returned each returned item is item is item is within the requested within the requested minimum and minimum and maximum price range. maximum price range. maximum price range. It is an error if It is an error if any output any output any output exceeds the budget. He exceeds the budget. He exceeds the budget. He can actually go into the can actually go into the can actually go into the code and check if code and check if these filters are working correctly. This these filters are working correctly. This rubric will now rubric will now rubric will now detect this on detect this on detect this on every subsequent every subsequent every subsequent run, not just in the run, not just in the run, not just in the individual test individual test individual test case you case you case you coded earlier. coded earlier. So, to quickly So, to quickly summarize, we summarize, we summarize, we talked about talked about talked about detection, detection, detection, research, and research, and research, and implementation of implementation of implementation of a fix. The part a fix. The part a fix. The part about implementing about implementing about implementing the fix is ​​what I want the fix is ​​what I want the fix is ​​what I want to discuss now in the to discuss now in the to discuss now in the context of agent context of agent context of agent experiments. So, experiments. So, experiments. So, maybe we've found maybe we've found maybe we've found the problem, backed the problem, backed the problem, backed it up with real it up with real it up with real trace data, trace data, trace data, investigated it, know investigated it, know investigated it, know why it's occurring, and why it's occurring, and why it's occurring, and decided we have a decided we have a decided we have a fix fix fix worth implementing.

worth implementing. worth implementing. Before Before Before implementing implementing implementing a fix, it's a good idea a fix, it's a good idea a fix, it's a good idea to test it on a to test it on a to test it on a set of results. set of results. set of results. Not sure how you Not sure how you Not sure how you do it now, do it now, do it now, maybe deploy maybe deploy maybe deploy in a in a in a development environment and then development environment and then development environment and then test manually. Arize test manually. Arize test manually. Arize has offered has offered has offered agent experiments agent experiments that allow you that allow you that allow you to test this to test this to test this automatically through automatically through automatically through CI. I think we're all CI. I think we're all CI. I think we're all familiar with the concept of familiar with the concept of failure data sets that failure data sets that we collect over time. we collect over time. With With agent agent agent experiments, you experiments, you experiments, you can deploy a can deploy a can deploy a test agent on test agent on test agent on an endpoint and an endpoint and an endpoint and route a dataset to it route a dataset to it directly from Arize. directly from Arize. This dataset This dataset This dataset will consist of will consist of will consist of real traces real traces real traces where your agent where your agent where your agent made an error. made an error. In the previous In the previous example, I was showing example, I was showing example, I was showing Wonder Toys, where there were several Wonder Toys, where there were several Wonder Toys, where there were several tracebacks tracebacks tracebacks requesting requesting requesting birthday gifts birthday gifts birthday gifts under under under $30 that didn't $30 that didn't $30 that didn't work. I can work. I can work. I can use use use Arize's capabilities to Arize's capabilities to Arize's capabilities to create such a create such a create such a dataset. So, dataset. So, dataset. So, I'll have, let's say, 25 I'll have, let's say, 25 I'll have, let's say, 25 examples of failed examples of failed examples of failed searches with searches with searches with budget budget budget constraints. Then I constraints. Then I constraints. Then I ran a ran a ran a workflow with cloud workflow with cloud workflow with cloud code, analyzed code, analyzed code, analyzed the trace, found the the trace, found the the trace, found the bug, and have a bug, and have a bug, and have a fix in the fix in the fix in the form of a PR, or form of a PR, or form of a PR, or maybe Signal did maybe Signal did maybe Signal did it for me in the background it for me in the background it for me in the background . How do I . How do I . How do I test my test my test my agent with this agent with this agent with this dataset to dataset to dataset to make sure the make sure the make sure the proposed proposed proposed fix works? This is what fix works? This is what agent agent experiments allow us to do.

experiments allow us to do. experiments allow us to do. With agent With agent With agent experiments and the experiments and the agent sandbox in the product, you agent sandbox in the product, you can run a can run a can run a test for a patched test for a patched test for a patched version. You version. You version. You deploy it in a deploy it in a deploy it in a development environment, development environment, development environment, it becomes available it becomes available it becomes available on an endpoint, you on an endpoint, you on an endpoint, you pass the input pass the input pass the input data set to the agent data set to the agent , the agent responds, and , the agent responds, and , the agent responds, and you evaluate you evaluate you evaluate the result using the same the result using the same the result using the same evaluation rubric evaluation rubric evaluation rubric as in production. as in production. as in production. So you can So you can So you can compare side by side compare side by side what the agent does in what the agent does in what the agent does in development mode with the development mode with the development mode with the patch, patch, patch, versus what the versus what the versus what the agent did in agent did in agent did in production, production, production, and see if you're and see if you're and see if you're actually covering actually covering actually covering those edge cases. those edge cases. So, like other dataset experimentation workflows, you dataset experimentation workflows, you need to look at the before need to look at the before need to look at the before and and and after state, evaluate after state, evaluate after state, evaluate your judge—in this your judge—in this your judge—in this case, the judge of case, the judge of case, the judge of price correctness, price correctness, price correctness, look at examples look at examples look at examples and queries, see and queries, see and queries, see if if if latency or latency or latency or cost increases, perhaps, to get a cost increases, perhaps, to get a cost increases, perhaps, to get a sense of these sense of these sense of these operational metrics operational metrics , and then , and then , and then quantify the results of quantify the results of quantify the results of your experiment. your experiment.

And then maybe And then maybe going back a little bit going back a little bit going back a little bit to the idea of ​​a to the idea of ​​a to the idea of ​​a self-correcting self-correcting self-correcting loop as a coherent loop as a coherent loop as a coherent storyline, you storyline, you storyline, you now have an agent that is now have an agent that is now have an agent that is autonomously working with autonomously working with autonomously working with your traces, your traces, your traces, watching to see if there are any watching to see if there are any watching to see if there are any problems. problems. problems. So, I see that the So, I see that the So, I see that the user asked for a user asked for a user asked for a price of up to $30, and price of up to $30, and price of up to $30, and the answer was clearly the answer was clearly the answer was clearly over $30. over $30. over $30. Investigating the reason for Investigating the reason for Investigating the reason for this with reference to the this with reference to the this with reference to the codebase, the agent codebase, the agent codebase, the agent proposes a fix proposes a fix proposes a fix in the form of a in the form of a in the form of a merge request (PR) on merge request (PR) on merge request (PR) on how he thinks it how he thinks it how he thinks it can be improved. In can be improved. In can be improved. In this case, it was a this case, it was a this case, it was a filtering error. filtering error. Because our Because our experiments experiments experiments are exposed through are exposed through are exposed through skills and APIs, the agent skills and APIs, the agent skills and APIs, the agent can actually can actually can actually redeploy itself redeploy itself redeploy itself on a new PR and on a new PR and on a new PR and test itself on a test itself on a test itself on a dataset of dataset of dataset of real-world real-world real-world crash cases before crash cases before crash cases before notifying you or notifying you or notifying you or deciding it's worth deciding it's worth deciding it's worth your human your human your human review. So, you know, review. So, you know, review. So, you know, this full cycle this full cycle this full cycle happens happens happens completely autonomously. completely autonomously. The agent will actually collect the The agent will actually collect the crashes, create a crashes, create a crashes, create a dataset for itself, dataset for itself, dataset for itself, run it on a run it on a run it on a new version of itself, and new version of itself, and new version of itself, and prepare a response prepare a response prepare a response for you: “Hey, I for you: “Hey, I for you: “Hey, I think I found think I found think I found this problem. I this problem. I this problem. I examined her. I know examined her. I know examined her. I know why this can why this can why this can happen. I happen. I happen. I created a PR. I then created a PR. I then created a PR. I then ran this PR in a ran this PR in a ran this PR in a development environment development environment development environment on the on the on the detected failures dataset and detected failures dataset and detected failures dataset and saw, in this saw, in this saw, in this case, a 36% case, a 36% case, a 36% performance improvement over performance improvement over performance improvement over this particular this particular this particular estimator. And all this estimator. And all this estimator. And all this information is available to information is available to information is available to you directly in the PR you directly in the PR , so you can just , so you can just , so you can just read it, read it, read it, check the code, check the code, check the code, do a code review, do a code review, do a code review, and release

and release and release the update. So I the update. So I the update. So I think that, you know, think that, you know, think that, you know, brings us back to the idea of brings us back to the idea of brings us back to the idea of this cycle. I think that this cycle. I think that this cycle. I think that makes it makes it makes it really very really very really very powerful, allowing you powerful, allowing you powerful, allowing you to close the loop where to close the loop where to close the loop where you have a cycle of discovery, you have a cycle of discovery, you have a cycle of discovery, exploration, exploration, exploration, hypothesising, and hypothesising, and hypothesising, and correction. And correction. And correction. And you also have a validation cycle, you also have a validation cycle, you also have a validation cycle, right? So how do right? So how do right? So how do I actually I actually I actually know that this know that this know that this fix is fix is fix is correct? And it's not correct? And it's not correct? And it's not just, you know, an agent just, you know, an agent just, you know, an agent in a sandbox in a sandbox in a sandbox saying everything is okay. saying everything is okay. saying everything is okay. How do I actually How do I actually How do I actually test this on test this on test this on real failures and real failures and real failures and real edge real edge real edge cases from my cases from my cases from my working example. And working example. And working example. And how do I, you know, how do I, you know, how do I, you know, start start start setting this up and setting this up and setting this up and automatically automatically automatically testing it for myself. And testing it for myself. And testing it for myself. And so, I think, you know, so, I think, you know, so, I think, you know, we'll be back. we'll be back. we'll be back. We will also have time We will also have time We will also have time to help you to help you to help you during the rest of the workshop during the rest of the workshop during the rest of the workshop if the Wi-Fi works if the Wi-Fi works if the Wi-Fi works a little better. But I'll a little better. But I'll a little better. But I'll go back to the go back to the go back to the previous slide. previous slide. previous slide. So you can So you can So you can walk through this yourself. walk through this yourself. walk through this yourself. First: getting started First: getting started First: getting started with tracing. If with tracing. If with tracing. If you don't collect you don't collect you don't collect trace data, nothing trace data, nothing trace data, nothing else else else will work. It will work. It will work. It takes less than 10 takes less than 10 takes less than 10 minutes for your minutes for your minutes for your own agent.

own agent. own agent. We have the skills We have the skills We have the skills to tool. We to tool. We to tool. We support all support all support all major agent major agent major agent frameworks and all frameworks and all frameworks and all major major model providers. Activating model providers. Activating a signal and a signal and a signal and delving in or delving in or delving in or using using using cloud code to cloud code to cloud code to actually actually actually apply apply apply Arize's skills to conduct the Arize's skills to conduct the Arize's skills to conduct the investigation for you. investigation for you. investigation for you. Validation Validation Validation using a test using a test using a test system as a judge, and system as a judge, and system as a judge, and then measuring the then measuring the then measuring the results results results using agent using agent using agent experiments. So, experiments. So, experiments. So, before we wrap up, I'll before we wrap up, I'll before we wrap up, I'll come back to this and come back to this and come back to this and we'll all go around the we'll all go around the we'll all go around the room and try to room and try to room and try to help you help you help you set this up. If set this up. If set this up. If you have any problems you have any problems you have any problems with .env files, just with .env files, just with .env files, just search the search the search the repository or repository or repository or ask your coding ask your coding agent where the agent where the agent where the .env file is located, .env file is located, .env file is located, they will help you with they will help you with they will help you with it. Also, it. Also, it. Also, make sure you make sure you make sure you clone clone clone the repository and the repository and the repository and get your API get your API keys from the Arize platform, keys from the Arize platform, keys from the Arize platform, and then you can and then you can and then you can see how these see how these see how these traces traces traces come together. In the come together. In the come together. In the spirit of AIE and workshops, spirit of AIE and workshops, spirit of AIE and workshops, I think it works I think it works I think it works much better when much better when much better when you see it, you see it, you see it, touch it yourself, touch it yourself, touch it yourself, and try to and try to and try to do it do it do it yourself. So yourself. So yourself. So hopefully the Wi-Fi hopefully the Wi-Fi hopefully the Wi-Fi will work and I'll be able to will work and I'll be able to will work and I'll be able to walk around the room. walk around the room. walk around the room. If you have any If you have any If you have any questions questions questions or get or get or get stuck, don't stuck, don't stuck, don't hesitate to ask hesitate to ask hesitate to ask me. Chances are, me. Chances are, me. Chances are, someone else in this someone else in this someone else in this room is facing the room is facing the room is facing the same problem.

same problem. same problem. So I'll be happy to So I'll be happy to So I'll be happy to give you some advice as you go give you some advice as you go give you some advice as you go . By the way, . By the way, . By the way, if you are also having if you are also having if you are also having difficulty with difficulty with skill set, I have put it on skill set, I have put it on slides here. It's very slides here. It's very slides here. It's very simple: mpx skills add arize-ai/ simple: mpx skills add arize-ai/ arize-skills. This will give arize-skills. This will give arize-skills. This will give you access to an you access to an you access to an interactive interactive interactive mode where you can mode where you can mode where you can choose what you choose what you choose what you want to install, want to install, want to install, or just or just or just install everything install everything install everything at once. This includes at once. This includes at once. This includes skills in skills in skills in trace analysis, trace analysis, data set curation, and data set curation, and evaluators. So, as an evaluators. So, as an evaluators. So, as an example of some example of some example of some skills you skills you skills you can run: I can run: I can run: I can ask Qual can ask Qual can ask Qual to check the code: "Hey, to check the code: "Hey, to check the code: "Hey, check my check my check my traces for traces for traces for errors or errors or errors or anything I can anything I can anything I can fix." "Qual, fix." "Qual, fix." "Qual, review the code for the review the code for the review the code for the agent in this agent in this agent in this repository and repository and repository and use the use the use the agent's skills." We will send a agent's skills." We will send a agent's skills." We will send a recording of this event, as well recording of this event, as well recording of this event, as well as all the resources as all the resources as all the resources you need to you need to you need to get started. So, get started. So, get started. So, sorry about sorry about sorry about the internet, if the internet, if the internet, if everything doesn't work everything doesn't work everything doesn't work out today, you out today, you out today, you can still get started very quickly can still get started very quickly can still get started very quickly . . . Okay, friends, it's time for us Okay, friends, it's time for us Okay, friends, it's time for us to wrap up our to wrap up our to wrap up our seminar. Thank you again. seminar. Thank you again. seminar. Thank you again. I just want to I just want to I just want to leave you with this leave you with this leave you with this "discover, "discover, "discover, explore, fix, explore, fix, explore, fix, and test" slide, which and test" slide, which and test" slide, which covers the basic covers the basic covers the basic path of how you path of how you path of how you can get started can get started can get started with Arize and how to build with Arize and how to build self-improving agents.

self-improving agents. And again, feel free to And again, feel free to And again, feel free to reach out to me on reach out to me on reach out to me on Twitter or LinkedIn. And if Twitter or LinkedIn. And if Twitter or LinkedIn. And if you want to get a you want to get a you want to get a year of Arize Pro for free year of Arize Pro for free , just use , just use , just use the code on the screen. We have a the code on the screen. We have a the code on the screen. We have a free tier free tier free tier that is always that is always free to use. free to use. For this conference only, For this conference only, For this conference only, we are also providing Arize we are also providing Arize we are also providing Arize Pro for free, which gives Pro for free, which gives Pro for free, which gives you slightly higher you slightly higher you slightly higher usage limits. If you are usage limits. If you are usage limits. If you are running agents with running agents with running agents with thousands and thousands of thousands and thousands of thousands and thousands of traces. So, traces. So, traces. So, thank you very much. We are thank you very much. We are thank you very much. We are having another having another having another workshop at 2pm, workshop at 2pm, workshop at 2pm, hosted by Lori, where we will hosted by Lori, where we will hosted by Lori, where we will go over some more go over some more go over some more code examples. code examples. code examples. Hopefully the Wi-Fi Hopefully the Wi-Fi Hopefully the Wi-Fi will be a little better by then, and will be a little better by then, and will be a little better by then, and yes, we'll get started. Thank yes, we'll get started. Thank yes, we'll get started. Thank you. I appreciate you.