From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

Before we start, Before we start, if you want to if you want to if you want to work with work with work with me, this is a hands-on me, this is a hands-on me, this is a hands-on workshop, so I see a workshop, so I see a workshop, so I see a lot of laptops, lot of laptops, lot of laptops, great. Um, if you great. Um, if you great. Um, if you want to do everything want to do everything want to do everything exactly like me, you exactly like me, you exactly like me, you need to have a need to have a need to have a wandb account or a wandb account or a wandb account or a Weights Weights Weights and Biases account. If you and Biases account. If you and Biases account. If you don't have one, you can don't have one, you can don't have one, you can sign up and sign up and sign up and get one for get one for get one for free. We free. We free. We also provide you with also provide you with also provide you with withdrawal credits withdrawal credits withdrawal credits so you can conduct an so you can conduct an so you can conduct an evaluation, as well as evaluation, as well as evaluation, as well as access to the sandbox. access to the sandbox. access to the sandbox. And, um, if you want to And, um, if you want to And, um, if you want to use something use something use something else, like OpenRouter else, like OpenRouter , you can do that too , you can do that too , you can do that too . Or even . Or even . Or even another provider, whatever another provider, whatever another provider, whatever you you you want to want to want to use. I'll use. I'll use. I'll just just just use this use this use this as an example in the as an example in the as an example in the seminar. And if you seminar. And if you seminar. And if you have an AI agent, you can have an AI agent, you can have an AI agent, you can just tell it just tell it just tell it to change something and to change something and to change something and make it work make it work make it work with my setup. with my setup. So, let's go. Briefly So, let's go. Briefly about me. I'm Wolfram about me. I'm Wolfram about me. I'm Wolfram Reiters. And my slogan: Reiters. And my slogan: Reiters. And my slogan: I evaluate models for I evaluate models for I evaluate models for breakfast, build breakfast, build breakfast, build agents at night, and agents at night, and agents at night, and preach AI all preach AI all preach AI all day. I do this as an AI day. I do this as an AI evangelist at Corvus and evangelist at Corvus and evangelist at Corvus and Weights and Biases. I also Weights and Biases. I also Weights and Biases. I also co-host the co-host the co-host the Thursday AI podcast. Alex is here too.

Thursday AI podcast. Alex is here too. Thursday AI podcast. Alex is here too. He is the boss here. And He is the boss here. And He is the boss here. And if you have any if you have any if you have any technical problems technical problems technical problems during the seminar, you during the seminar, you during the seminar, you can contact can contact can contact him. He will help him. He will help him. He will help you so that we don't get you so that we don't get you so that we don't get interrupted. He is interrupted. He is interrupted. He is your man. Thanks, your man. Thanks, your man. Thanks, Alex, for your help. And Alex, for your help. And Alex, for your help. And yes, you can yes, you can yes, you can follow me on X follow me on X follow me on X or LinkedIn, and on wolfbench.ai. or LinkedIn, and on wolfbench.ai. or LinkedIn, and on wolfbench.ai. Easy to remember. Um, Easy to remember. Um, Easy to remember. Um, but this workshop is not but this workshop is not but this workshop is not just about wolfbench, it's just about wolfbench, it's just about wolfbench, it's about why and how I about why and how I about why and how I built it so you built it so you built it so you can deploy can deploy can deploy your own model and your own model and your own model and agent, even from scratch agent, even from scratch agent, even from scratch to leaderboard. Um, yeah to leaderboard. Um, yeah , my story, basically, , my story, basically, , my story, basically, I started from childhood, I I started from childhood, I I started from childhood, I loved AI from the very loved AI from the very loved AI from the very beginning. Then, when beginning. Then, when beginning. Then, when ChatGPT came along, I ChatGPT came along, I ChatGPT came along, I decided to focus decided to focus decided to focus my career on that. my career on that. my career on that. Um, if anyone Um, if anyone Um, if anyone remembers the good old remembers the good old remembers the good old days of local Llama days of local Llama days of local Llama when the community was when the community was when the community was quantizing models, I quantizing models, I quantizing models, I was testing these models was testing these models was testing these models or started that way. or started that way. or started that way. I was often asked I was often asked I was often asked to help with to help with to help with the removal and other the removal and other the removal and other things. So I became a things. So I became a things. So I became a consultant. And yes, consultant. And yes, consultant. And yes, last year I last year I last year I fully immersed myself in a fully immersed myself in a fully immersed myself in a German AI startup, German AI startup, German AI startup, and this year I and this year I and this year I joined Corvus. And I've joined Corvus. And I've joined Corvus. And I've been doing a podcast for been doing a podcast for been doing a podcast for quite some time now, right, quite some time now, right, quite some time now, right, Alex? But this is about you.

Alex? But this is about you. So, the seminar: who are you So, the seminar: who are you and why are you here. I think there and why are you here. I think there and why are you here. I think there are different types of people, and are different types of people, and are different types of people, and I'm curious to see which I'm curious to see which I'm curious to see which category you fall into category you fall into category you fall into . So, . So, . So, for example, for example, for example, agent developers. Are there agent developers. Are there agent developers. Are there anyone here who anyone here who anyone here who creates agents and creates agents and creates agents and wants to evaluate them? wants to evaluate them? Someone here, someone there, Someone here, someone there, several people. And new several people. And new several people. And new skeptics. Are there people skeptics. Are there people skeptics. Are there people who want to conduct a who want to conduct a who want to conduct a qualitative assessment but qualitative assessment but qualitative assessment but don't yet know how don't yet know how don't yet know how to do it and want to to do it and want to to do it and want to learn? Okay, there are learn? Okay, there are learn? Okay, there are more of those. What more of those. What more of those. What about about about model estimators? Who model estimators? Who model estimators? Who cares which model cares which model cares which model or agent can be or agent can be or agent can be trusted? Which one do I trusted? Which one do I trusted? Which one do I want to use? Is want to use? Is want to use? Is anyone interested in anyone interested in anyone interested in this? He's there and he's there. this? He's there and he's there. this? He's there and he's there. Perfectly. Platforms and Perfectly. Platforms and Perfectly. Platforms and their leaders. People who their leaders. People who their leaders. People who want to conduct want to conduct want to conduct comparative and comparative and comparative and reproducible evaluation. reproducible evaluation. reproducible evaluation. Someone is here. Several Someone is here. Several Someone is here. Several people are here. Perfectly. people are here. Perfectly. Trace debuggers. These are the people who Trace debuggers. These are the people who look at the actual look at the actual look at the actual execution traces to execution traces to execution traces to figure out what went wrong figure out what went wrong figure out what went wrong and how to improve it. and how to improve it. and how to improve it. Here. Perfectly. And Here. Perfectly. And Here. Perfectly. And now the skeptics. Who now the skeptics. Who now the skeptics. Who doesn't trust doesn't trust doesn't trust leaderboards? Well, now leaderboards? Well, now leaderboards? Well, now each of you should each of you should each of you should raise your hand, right?

raise your hand, right? raise your hand, right? But why is this the case and how But why is this the case and how But why is this the case and how can we change it? can we change it? can we change it? This is part of what This is part of what This is part of what we're talking about and we're talking about and how you can how you can how you can create your own create your own create your own tools. But tools. But tools. But first: my first: my first: my main principle is that main principle is that thinking is more important thinking is more important thinking is more important than skills, and skills are more than skills, and skills are more than skills, and skills are more important than important than important than tools. So the tools. So the tools. So the toolkit toolkit is what we is what we is what we use, and in AI use, and in AI use, and in AI it changes almost it changes almost it changes almost every week or month. every week or month. every week or month. Something new is Something new is Something new is emerging and everything is emerging and everything is emerging and everything is constantly changing. constantly changing. constantly changing. Skills are how we Skills are how we Skills are how we use the use the use the tools we tools we tools we have. This applies to have. This applies to have. This applies to various instruments. various instruments. various instruments. So, if you have So, if you have So, if you have great skills and great skills and great skills and know how know how know how to use the to use the to use the tools, you tools, you tools, you can switch can switch can switch to others. But thinking to others. But thinking is the most important thing because it is the is the most important thing because it is the is the most important thing because it is the hardest hardest hardest to change or improve. to change or improve. to change or improve. And it is precisely for evaluation that it is And it is precisely for evaluation that it is And it is precisely for evaluation that it is important to know why important to know why important to know why we conduct we conduct we conduct benchmarks and what to pay benchmarks and what to pay benchmarks and what to pay attention to. attention to. attention to. Skills are simply the Skills are simply the Skills are simply the ability to use ability to use ability to use tools. And tools. And tools. And toolkits toolkits —I'll show you a few, —I'll show you a few, —I'll show you a few, and you can create and you can create and you can create your own, like I your own, like I your own, like I did with Wolf Bench. And did with Wolf Bench. And did with Wolf Bench. And the premise is this: the premise is this: the premise is this: one assessment one assessment one assessment is not enough, because is not enough, because is not enough, because productivity is productivity is productivity is a distribution, not a point. a distribution, not a point. What I don't What I don't like about like about like about leaderboards and leaderboards and leaderboards and many of the scores many of the scores many of the scores I've seen is that the nice I've seen is that the nice I've seen is that the nice bars and numbers bars and numbers bars and numbers are just are just are just averages averages that don't really that don't really that don't really tell you much. That's why tell you much. That's why tell you much. That's why people often say, "I people often say, "I people often say, "I don't trust this, don't trust this, don't trust this, I have a different experience." Behind I have a different experience." Behind I have a different experience." Behind the numbers lie the numbers lie the numbers lie things that we can things that we can things that we can discover and discover and discover and consider. Because is consider. Because is consider. Because is this really a good this really a good this really a good model? Maybe she model? Maybe she model? Maybe she just got lucky with just got lucky with just got lucky with this launch? What are the this launch? What are the this launch? What are the hidden compromises hidden compromises ? And so on. And that's why ? And so on. And that's why ? And so on. And that's why I created

I created I created Wolf Bench, because I wanted to know Wolf Bench, because I wanted to know Wolf Bench, because I wanted to know which models could be which models could be which models could be used with used with used with which agents. The agent which agents. The agent which agents. The agent changed over time. In changed over time. In changed over time. In January it was Open Claw. In January it was Open Claw. In January it was Open Claw. In February it was Hermes Agent. February it was Hermes Agent. February it was Hermes Agent. And, uh, I use Codex a lot now And, uh, I use Codex a lot now And, uh, I use Codex a lot now . . . It used to be Cloud Code. And It used to be Cloud Code. And It used to be Cloud Code. And I still want to know which I still want to know which I still want to know which models models models work best with which work best with which work best with which settings. And so settings. And so , as I said, , as I said, , as I said, most most assessment results boil down assessment results boil down to just one score to just one score . And most . And most . And most leaderboards rank leaderboards rank leaderboards rank models. You don't even models. You don't even models. You don't even see the agent see the agent see the agent systems behind systems behind systems behind it. Or, yeah, different it. Or, yeah, different it. Or, yeah, different versions, settings, versions, settings, versions, settings, even if you even if you even if you can even see what, uh can even see what, uh , settings they , settings they , settings they used. And used. And used. And they they they vary greatly, so vary greatly, so vary greatly, so you see one you see one you see one result and someone result and someone result and someone else reports the else reports the else reports the same benchmark but same benchmark but same benchmark but uses different uses different uses different settings. So, settings. So, settings. So, can you can you can you actually compare them? actually compare them? actually compare them? And is there a way And is there a way And is there a way to verify the to verify the to verify the results results results these people got? So these people got? So these people got? So Wolf Bench is a framework. Wolf Bench is a framework. Wolf Bench is a framework. It has graphs, It has graphs, It has graphs, dashboards, and dashboards, and dashboards, and leaderboards. By the way, it's leaderboards. By the way, it's leaderboards. By the way, it's open source, and open source, and open source, and there's also a there's also a there's also a methodology there, and that's what methodology there, and that's what methodology there, and that's what I want to I want to I want to talk about the most talk about the most . So, you can . So, you can . So, you can use this use this use this methodology with methodology with methodology with your own your own your own tools. And for tools. And for tools. And for WolfBench, one thing that WolfBench, one thing that WolfBench, one thing that applies to all applies to all applies to all benchmarks is that benchmarks is that benchmarks is that you have to you have to you have to decide what metrics decide what metrics decide what metrics you are interested in. Do you you are interested in. Do you you are interested in. Do you want to look at the want to look at the want to look at the average? Do you average? Do you average? Do you want to look at the want to look at the want to look at the worst and best worst and best worst and best results? Do you want results? Do you want results? Do you want to look at to look at to look at what percentage of what percentage of what percentage of benchmarks it benchmarks it benchmarks it can solve at all can solve at all can solve at all ? And how ? And how ? And how consistently he consistently he consistently he does it, and also does it, and also does it, and also how much it costs. This is how much it costs. This is how much it costs. This is also something that not everyone also something that not everyone also something that not everyone takes into account.

takes into account. takes into account. How many tokens were How many tokens were How many tokens were burned? How much burned? How much burned? How much did it cost? And it could did it cost? And it could did it cost? And it could be that a be that a be that a cheaper and faster cheaper and faster cheaper and faster model runs longer and model runs longer and model runs longer and costs more than costs more than costs more than one of the larger one of the larger one of the larger models, simply because models, simply because models, simply because it used it used it used so many tokens. so many tokens. so many tokens. Also, it's very important to have Also, it's very important to have Also, it's very important to have uniform conditions so that uniform conditions so that uniform conditions so that you have the you have the you have the same same same settings every time, because settings every time, because settings every time, because if you change just if you change just if you change just one variable, one of one variable, one of one variable, one of many that exist, many that exist, many that exist, and we're looking at a bunch of and we're looking at a bunch of and we're looking at a bunch of them, if you them, if you them, if you change that, it's effectively a change that, it's effectively a change that, it's effectively a new benchmark. And new benchmark. And new benchmark. And also also also multi-agency: multi-agency: multi-agency: if you want if you want if you want to use to use to use different agents, different agents, different agents, who are you looking at? And who are you looking at? And who are you looking at? And always do multiple always do multiple always do multiple runs. One launch runs. One launch is a benchmark lottery is a benchmark lottery . This is a random . This is a random . This is a random result and is only an result and is only an result and is only an approximate number. Therefore, approximate number. Therefore, approximate number. Therefore, you should do you should do you should do several runs. I several runs. I several runs. I do five runs do five runs do five runs in WolfBench, but you can in WolfBench, but you can in WolfBench, but you can do more if you do more if you do more if you have the money, of course have the money, of course . It's always a compromise. . It's always a compromise. . It's always a compromise. Uh, transparency is also Uh, transparency is also Uh, transparency is also very important to me very important to me , because I publish , because I publish , because I publish all the results and all the results and all the results and people can people can people can see them. So I don't see them. So I don't see them. So I don't just show a just show a just show a rating and say, this is rating and say, this is rating and say, this is why this is my why this is my why this is my number one model and this is my number one model and this is my number one model and this is my number two model. I'm number two model. I'm number two model. I'm uploading everything, you uploading everything, you uploading everything, you can check it out, can check it out, can check it out, study it, or delegate study it, or delegate study it, or delegate it to your agent, which is it to your agent, which is it to your agent, which is quite logical given quite logical given quite logical given all the all the all the data we've collected. So, data we've collected. So, data we've collected. So, essentially, get essentially, get essentially, get the right mindset in place and the right mindset in place and the right mindset in place and the tools will become the tools will become the tools will become interchangeable. And interchangeable. And interchangeable. And the most important thing I think the most important thing I think people need to realize is that people need to realize is that you're not ranking the you're not ranking the you're not ranking the model. If you model. If you model. If you are testing a model, it is the are testing a model, it is the are testing a model, it is the same model, but same model, but same model, but you are actually testing a you are actually testing a you are actually testing a specific specific specific configuration. You configuration. You configuration. You use a use a use a dataset, and if you dataset, and if you dataset, and if you change the dataset, change the dataset, change the dataset, it's a new test.

it's a new test. it's a new test. You have a model, and behind it is the You have a model, and behind it is the inference mechanism and inference mechanism and inference mechanism and everything that happens everything that happens everything that happens at the endpoint, which at the endpoint, which at the endpoint, which you often can't you often can't you often can't control. But control. But control. But if they update the if they update the output software, the results output software, the results may may may change again and you change again and you change again and you won't be able to won't be able to won't be able to see it. Unless see it. Unless see it. Unless it's your own it's your own it's your own output on your system output on your system . Agent orange. Also Agent orange . Agent orange. Also Agent orange : if I test Blood Code : if I test Blood Code : if I test Blood Code or Codex, it's a different or Codex, it's a different or Codex, it's a different version if I version if I version if I test it the test it the test it the next week next week next week after the update. So this is after the update. So this is after the update. So this is also important also important also important to remember. The same to remember. The same to remember. The same goes for the sandbox goes for the sandbox . Even if you . Even if you . Even if you use a use a use a sandbox provider sandbox provider , and they change something , and they change something , and they change something in their backend, you in their backend, you in their backend, you might not even might not even might not even notice, and it could notice, and it could notice, and it could affect all the affect all the affect all the settings you settings you settings you chose yourself. So chose yourself. So chose yourself. So change any change any change any part and it's a part and it's a part and it's a different experiment. You different experiment. You different experiment. You must remember must remember must remember this. Um, okay, here's the this. Um, okay, here's the this. Um, okay, here's the glossary. You glossary. You glossary. You will get will get will get the presentation, so I wo the presentation, so I wo the presentation, so I wo n't go through all of it n't go through all of it n't go through all of it . . . Most of you are very Most of you are very Most of you are very familiar with this, and familiar with this, and familiar with this, and I will explain everything that will be I will explain everything that will be I will explain everything that will be needed in the needed in the needed in the process. So we're process. So we're process. So we're looking at a looking at a looking at a specific set of specific set of specific set of tools because tools because tools because this is a hands-on this is a hands-on this is a hands-on workshop, and this is one workshop, and this is one workshop, and this is one way to build, not the way to build, not the way to build, not the only way, and I'll show you only way, and I'll show you only way, and I'll show you different ways, and you different ways, and you different ways, and you can choose your own.

can choose your own. can choose your own. To follow the progress, To follow the progress, To follow the progress, you need a you need a you need a laptop, a terminal, laptop, a terminal, laptop, a terminal, Python, UV, Python, UV, Python, UV, and Git are recommended. We have files and Git are recommended. We have files and Git are recommended. We have files for download, for download, for download, so if you don't so if you don't so if you don't have time, you can have time, you can have time, you can download download download the results, the results, the results, import them, and import them, and import them, and still use still use still use this this this toolkit. So you toolkit. So you toolkit. So you don't have to don't have to don't have to follow everything, but follow everything, but follow everything, but if you want, you can if you want, you can if you want, you can do it and do it and do it and use use use your own results your own results your own results that way. Um, yes, that way. Um, yes, that way. Um, yes, you must have Wi-Fi, and an API you must have Wi-Fi, and an API key is optional. I key is optional. I key is optional. I showed this at the very showed this at the very showed this at the very beginning. I'll beginning. I'll beginning. I'll show it again in a show it again in a show it again in a minute. So you minute. So you minute. So you can get it. can get it. can get it. So, again, take a So, again, take a So, again, take a screenshot. screenshot. screenshot. Scan the QR code. Scan the QR code. Scan the QR code. Here you will find a form Here you will find a form where you can get where you can get where you can get free free free data output credits data output credits data output credits for the AI for the AI for the AI Engineer conference, and this Engineer conference, and this Engineer conference, and this requires a requires a requires a free free free One BE account. One BE account. One BE account. There you can There you can There you can register register register if you haven't already if you haven't already if you haven't already . If you do . If you do . If you do n't have it yet. Who already n't have it yet. Who already n't have it yet. Who already has an account? has an account? has an account? Does anyone here Does anyone here Does anyone here use One BE? use One BE? use One BE? A lot of people. Perfectly. A lot of people. Perfectly. A lot of people. Perfectly. Okay, and about Okay, and about Okay, and about sandboxes: I'm sandboxes: I'm sandboxes: I'm using the using the Core Weave serverless sandboxes from Core Weave serverless sandboxes from your suggestion, where your suggestion, where your suggestion, where each task each task each task has an isolated box, has an isolated box, has an isolated box, which is extremely which is extremely which is extremely important, because otherwise you important, because otherwise you important, because otherwise you would be changing your own would be changing your own would be changing your own system every time you system every time you system every time you run run run the assessment, and the the assessment, and the the assessment, and the result would be even more result would be even more result would be even more random. So, random. So, random. So, we look at six we look at six we look at six interchangeable interchangeable interchangeable levels and choose levels and choose levels and choose each of them, you can each of them, you can each of them, you can do the same. It's a do the same. It's a do the same. It's a set of skills, how set of skills, how set of skills, how to do it, and a worldview to do it, and a worldview to do it, and a worldview in the background, and now we're in the background, and now we're in the background, and now we're choosing different choosing different choosing different tools for tools for tools for this set that we're this set that we're this set that we're building. So we building. So we building. So we need a benchmark, need a benchmark, need a benchmark, that is, a data set.

that is, a data set. that is, a data set. We need an We need an evaluation tool to evaluation tool to run the dataset run the dataset run the dataset and the evaluation, and of course and the evaluation, and of course and the evaluation, and of course to orchestrate to orchestrate to orchestrate all of this and all of this and all of this and run it in sandboxes run it in sandboxes run it in sandboxes so it doesn't run so it doesn't run so it doesn't run on your system. And on your system. And on your system. And we also need an we also need an we also need an agent agent agent toolkit. toolkit. toolkit. What kind of agent do we What kind of agent do we What kind of agent do we want to want to want to use? Both use? Both use? Both the model and the service. This, the model and the service. This, the model and the service. This, of course, is usually what of course, is usually what interests people interests people interests people the most. How does the most. How does the most. How does this model work? The thing is, this model work? The thing is, this model work? The thing is, you're you're you're not just using a model, you're not just using a model, you're not just using a model, you're always always always using an using an using an endpoint. It is endpoint. It is endpoint. It is always the provider; always the provider; always the provider; if it's an Open Router, they if it's an Open Router, they if it's an Open Router, they route requests route requests route requests to other providers. There are to other providers. There are to other providers. There are many of them, and whether it's OpenAI, many of them, and whether it's OpenAI, many of them, and whether it's OpenAI, or Google, or XAI, or or Google, or XAI, or whoever, they have their own whoever, they have their own whoever, they have their own settings, and you settings, and you evaluate that to some extent as well. And also evaluate that to some extent as well. And also all the options that all the options that all the options that can be selected. So the can be selected. So the can be selected. So the first question is: what first question is: what first question is: what benchmark are we benchmark are we benchmark are we using? using? using? You can start with a You can start with a You can start with a benchmark, I did benchmark, I did benchmark, I did it this way because I wanted to it this way because I wanted to it this way because I wanted to test agent test agent test agent usage with usage with usage with different different different tools, so I tools, so I tools, so I decided I decided I decided I needed a benchmark needed a benchmark needed a benchmark that allowed this. So that allowed this. So that allowed this. So I looked at all the I looked at all the I looked at all the benchmarks that the benchmarks that the leading labs used , and I thought that Terminal , and I thought that Terminal , and I thought that Terminal Bench was almost all of them, and Bench was almost all of them, and Bench was almost all of them, and it's an agent benchmark.

it's an agent benchmark. it's an agent benchmark. This isn't just a This isn't just a This isn't just a coding benchmark, it's an coding benchmark, it's an coding benchmark, it's an agent benchmark with agent benchmark with agent benchmark with tasks like tasks like tasks like patching a patching a patching a git repository, cracking a git repository, cracking a git repository, cracking a password, repairing a password, repairing a password, repairing a database—things you would, database—things you would, database—things you would, for example, ask for example, ask for example, ask your agent your agent your agent to do in normal to do in normal to do in normal use. That's use. That's use. That's why I chose Terminal Bench why I chose Terminal Bench . But if you are . But if you are . But if you are interested in interested in interested in multilingualism, you multilingualism, you multilingualism, you can choose MMLU, or can choose MMLU, or can choose MMLU, or if you are interested in if you are interested in if you are interested in multimodality— multimodality— MMU Pro. If you MMU Pro. If you MMU Pro. If you are interested in are interested in are interested in using using using GUI agents, you should GUI agents, you should GUI agents, you should choose a different choose a different choose a different benchmark. Whether you're benchmark. Whether you're benchmark. Whether you're interested in math, interested in math, interested in math, programming, or programming, or programming, or anything else, anything else, anything else, there's a there's a there's a benchmark for everything; The main thing is benchmark for everything; The main thing is to find the one that is to find the one that is to find the one that is most interesting to you most interesting to you most interesting to you and comparable. and comparable. and comparable. Of course, you Of course, you Of course, you can also create your can also create your can also create your own. You can own. You can own. You can create your own create your own create your own dataset. As for Terminal Bench, it is a dataset. As for Terminal Bench, it is a dataset. As for Terminal Bench, it is a general general general benchmark, which is why I benchmark, which is why I benchmark, which is why I chose it for chose it for chose it for testing agents. testing agents. testing agents. There are versions 2.0 and There are versions 2.0 and There are versions 2.0 and 2.1, and version 3.0 is currently in 2.1, and version 3.0 is currently in 2.1, and version 3.0 is currently in development. I chose 2.0 development. I chose 2.0 development. I chose 2.0 because version 2.1 was because version 2.1 was because version 2.1 was not available when I not available when I not available when I started. If you're started. If you're started. If you're starting now, starting now, starting now, go with the latest go with the latest go with the latest version 2.1, but I'm version 2.1, but I'm version 2.1, but I'm using 2.0 because using 2.0 because using 2.0 because upgrading would upgrading would upgrading would be a new be a new be a new benchmark and I would benchmark and I would benchmark and I would n't be able to n't be able to n't be able to compare compare compare results anymore. It results anymore. It results anymore. It only corrects 28 out of 89 only corrects 28 out of 89 only corrects 28 out of 89 tests, but that would be a tests, but that would be a tests, but that would be a different benchmark for different benchmark for different benchmark for all models. I would have to all models. I would have to retest everything retest everything I had already done. And I I had already done. And I I had already done. And I 've done a lot, although it's never 've done a lot, although it's never enough, but I enough, but I 've done a lot, so I 've done a lot, so I 've done a lot, so I don't want to don't want to don't want to create a new create a new create a new version of Wolf Bench right now. I'll version of Wolf Bench right now. I'll version of Wolf Bench right now. I'll do it later, do it later, do it later, maybe with the release of 3.0 maybe with the release of 3.0 maybe with the release of 3.0 or if there's a or if there's a or if there's a big leap in big leap in

big leap in capabilities and capabilities and capabilities and another another another benchmark is needed. benchmark is needed. Evaluation harness Evaluation harness . This is the stage where you have . This is the stage where you have . This is the stage where you have already chosen the benchmark and already chosen the benchmark and already chosen the benchmark and dataset you dataset you dataset you want to want to want to use. use. use. Now you need Now you need Now you need something that can something that can something that can run this run this run this evaluation, and Harbor is the evaluation, and Harbor is the evaluation, and Harbor is the standard framework standard framework standard framework for Terminal Bench. There are for Terminal Bench. There are for Terminal Bench. There are others. Inspector is a great others. Inspector is a great others. Inspector is a great toolkit from the toolkit from the UK AI Security Institute UK AI Security Institute that has many that has many that has many different benchmarks. different benchmarks. different benchmarks. It is also very It is also very It is also very popular. popular. popular. I've used the LM Evaluation Harness I've used the LM Evaluation Harness I've used the LM Evaluation Harness for years, it's been around forever for years, it's been around forever for years, it's been around forever , and OpenAI , and OpenAI , and OpenAI has its own has its own has its own evaluation tools. You can evaluation tools. You can evaluation tools. You can create your create your create your own "launcher" own "launcher" own "launcher" or use or use or use others. Choose what others. Choose what others. Choose what you like. Some people you like. Some people you like. Some people first choose a first choose a first choose a tool to tool to tool to launch, and only then launch, and only then launch, and only then choose one of the choose one of the choose one of the benchmarks it benchmarks it benchmarks it supports. This is also a supports. This is also a supports. This is also a completely valid completely valid completely valid approach. Now approach. Now approach. Now let's move on to the first let's move on to the first let's move on to the first practical step— practical step— installing Harbor. In installing Harbor. In installing Harbor. In this case, I'm this case, I'm this case, I'm using the current using the current using the current version. It version. It version. It was relevant yesterday. Not was relevant yesterday. Not was relevant yesterday. Not sure if sure if sure if it's relevant right it's relevant right it's relevant right now. This now. This now. This changes often, but I changes often, but I changes often, but I fix the version, and I fix the version, and I fix the version, and I strongly strongly strongly recommend that everyone recommend that everyone recommend that everyone stick to one stick to one stick to one version, because any version, because any version, because any change in variables, if change in variables, if change in variables, if it affects it affects it affects the estimate, and many do, the estimate, and many do, the estimate, and many do, means that means that means that you are already changing the you are already changing the you are already changing the parameter, and parameter, and parameter, and comparability comparability comparability suffers. Even if suffers. Even if suffers. Even if there is an error, there is an error, there is an error, at least you are testing at least you are testing at least you are testing all models with the all models with the all models with the same error.

same error. same error. But this is utopian But this is utopian But this is utopian thinking, of course, because thinking, of course, because thinking, of course, because sometimes you have to sometimes you have to sometimes you have to upgrade, if a upgrade, if a upgrade, if a new version of the new version of the assessment tool is supported by assessment tool is supported by another agent or another agent or another agent or provider, then you provider, then you provider, then you may be forced to may be forced to may be forced to upgrade. For those who are upgrade. For those who are serious about this, I serious about this, I recommend recommend recommend forking the repository, forking the repository, forking the repository, maintaining your own maintaining your own maintaining your own fork, and simply fork, and simply fork, and simply backporting everything you backporting everything you backporting everything you need. And if you need. And if you need. And if you install it, this is it. install it, this is it. Line break is missing here. This is uv tool Line break is missing here. This is uv tool install harbor. So, it install harbor. So, it install harbor. So, it is installed in a is installed in a is installed in a uv environment, so it does not uv environment, so it does not uv environment, so it does not affect your affect your affect your system, and you can system, and you can system, and you can easily remove it. easily remove it. easily remove it. Then you should have this Then you should have this Then you should have this version. Next version. Next version. Next step: agents step: agents step: agents need a sandbox need a sandbox need a sandbox because if you because if you because if you just run this just run this just run this locally on your own locally on your own locally on your own system, the second agent system, the second agent system, the second agent will see the already modified will see the already modified will see the already modified system and interfere with system and interfere with system and interfere with the work, so I wouldn't the work, so I wouldn't the work, so I wouldn't recommend recommend recommend doing this. You can doing this. You can doing this. You can use Docker. use Docker. use Docker. This is very common. But This is very common. But This is very common. But this is very difficult this is very difficult this is very difficult to parallelize, depending to parallelize, depending to parallelize, depending on what system on what system on what system you are using. you are using. you are using. So you need So you need So you need a lot of resources. That's a lot of resources. That's a lot of resources. That's why cloud why cloud why cloud providers exist, and I have a providers exist, and I have a providers exist, and I have a big big big thank you to Daytona. They thank you to Daytona. They thank you to Daytona. They supported me in supported me in supported me in the beginning, before the beginning, before the beginning, before QualiHub had its own QualiHub had its own QualiHub had its own sandboxes, and that's where I wanted to sandboxes, and that's where I wanted to sandboxes, and that's where I wanted to start.

start. start. They supported me a lot They supported me a lot They supported me a lot , and , and , and I conducted many assessments I conducted many assessments I conducted many assessments with Daytona. Now with Daytona. Now with Daytona. Now we have our own we have our own we have our own offer, so I offer, so I offer, so I use my own use my own use my own product, and that's why product, and that's why product, and that's why I recommend these two I recommend these two I recommend these two options. options. options. QualiHub sandboxes are designed for QualiHub sandboxes are designed for QualiHub sandboxes are designed for those using the those using the those using the Kubernetes service, and the “ Kubernetes service, and the “ golden” option golden” option golden” option here is here is here is QualiHub serverless sandboxes. You QualiHub serverless sandboxes. You QualiHub serverless sandboxes. You just need a just need a just need a Vent B account. Vent B account. Vent B account. What was that? Oh, okay. What was that? Oh, okay. We also have Um, I We also have Um, I Can you, I I turned off Can you, I I turned off Can you, I I turned off Slack so it doesn't Slack so it doesn't Slack so it doesn't show up here. If show up here. If show up here. If anyone wants to get anyone wants to get anyone wants to get the PDF immediately, just the PDF immediately, just the PDF immediately, just contact Alex. contact Alex. contact Alex. This is probably the easiest way at the moment This is probably the easiest way at the moment This is probably the easiest way at the moment . . . Um, okay. So, Um, okay. So, Um, okay. So, after you after you after you install this, uh, install this, uh, install this, uh, Harbor is installed. Harbor is installed. Harbor is installed. Now we need a Now we need a Now we need a sandbox client, sandbox client, sandbox client, which is part of Harbor. And which is part of Harbor. And which is part of Harbor. And in this case, it's in this case, it's in this case, it's part of several part of several part of several things. Here is the QR code for things. Here is the QR code for things. Here is the QR code for the presentation. Yes.

the presentation. Yes. the presentation. Yes. Good. Um, the command "uv Good. Um, the command "uv Good. Um, the command "uv tool install harbor" tool install harbor" tool install harbor" installs the provider installs the provider installs the provider for using " for using " sandboxes" and you sandboxes" and you sandboxes" and you don't don't don't need to do anything else. need to do anything else. need to do anything else. Once installed, Once installed, Once installed, you will only need you will only need you will only need your API key. You your API key. You your API key. You configure this in a YAML configure this in a YAML file, and here's the file, and here's the file, and here's the next step. You next step. You next step. You don't need to don't need to don't need to copy it. There is a copy it. There is a copy it. There is a link and a short link and a short link and a short URL that you can URL that you can URL that you can simply enter with the simply enter with the simply enter with the full configuration a full configuration a full configuration a few slides few slides few slides further down. This is just so further down. This is just so further down. This is just so you can see it you can see it . But if you . But if you . But if you follow along follow along follow along with me, just with me, just with me, just type the command "uv tool type the command "uv tool type the command "uv tool install" to install the install" to install the install" to install the Harbor provider if Harbor provider if Harbor provider if you're you're you're using it. using it. Let's continue. Let's continue. The next step is an The next step is an agent. This is the most interesting agent. This is the most interesting agent. This is the most interesting part for me because there are part for me because there are part for me because there are so many agents and so many agents and so many agents and I want to check how I want to check how I want to check how this agent works with this agent works with this agent works with this model. Here is a this model. Here is a this model. Here is a list of some list of some list of some agents. These are all agents. These are all agents. These are all agents, and there are even more agents, and there are even more agents, and there are even more supported by the supported by the supported by the Harbor framework. So Harbor framework. So , we have Cursor, Claude, Klein, , we have Cursor, Claude, Klein, , we have Cursor, Claude, Klein, Codex, Open Claude Hermes, and Terminus Codex, Open Claude Hermes, and Terminus Codex, Open Claude Hermes, and Terminus 2—this is the default agent 2—this is the default agent 2—this is the default agent for for for Terminal Bench and the Terminal Bench and the Terminal Bench and the Harbor framework. And this is not a real Harbor framework. And this is not a real Harbor framework. And this is not a real agent. You can't agent. You can't agent. You can't use it use it use it for coding, or for coding, or for coding, or at least you shouldn't.

at least you shouldn't. at least you shouldn't. It's more like It's more like It's more like simulating a simulating a simulating a CLI environment for an actual CLI environment for an actual CLI environment for an actual model: it sends model: it sends model: it sends keystrokes and the keystrokes and the keystrokes and the agent executes them. This is agent executes them. This is agent executes them. This is one way to one way to one way to do it. So, this is a do it. So, this is a do it. So, this is a simple agent that is simple agent that is simple agent that is standard: if you standard: if you standard: if you see see see Terminal Bench ratings somewhere and no one Terminal Bench ratings somewhere and no one Terminal Bench ratings somewhere and no one indicated that they indicated that they indicated that they tested with another tested with another tested with another agent, then agent, then agent, then they used they used they used it. And then— it. And then— which model do you want which model do you want which model do you want to test? You to test? You to test? You need to choose a need to choose a need to choose a model and choose model and choose model and choose a provider. If you a provider. If you a provider. If you are using GPT 5.5, are using GPT 5.5, are using GPT 5.5, you will be you will be you will be using OpenAI. using OpenAI. using OpenAI. If you If you If you use Claude, use Claude, use Claude, Fable (lucky if Fable (lucky if Fable (lucky if you you you use it), then use it), then use it), then you will you will you will use use use Anthropic. Google also provides Anthropic. Google also provides Anthropic. Google also provides some of the Anthropic models. some of the Anthropic models. And these are just some of them; And these are just some of them; OpenRouter provides almost all OpenRouter provides almost all , probably all, even , probably all, even , probably all, even with DDR enabled, so with DDR enabled, so with DDR enabled, so you don't send you don't send you don't send any data to the any data to the model providers. And we also have our model providers. And we also have our own own own inference engine with inference engine with inference engine with serverless inference, where serverless inference, where serverless inference, where you also get a you also get a you also get a selection of selection of selection of open source models. And in open source models. And in open source models. And in this case, I wanted to this case, I wanted to this case, I wanted to test GLM 5.2, which test GLM 5.2, which test GLM 5.2, which was the last was the last model I tested and also the model I tested and also the best model with best model with best model with open weights.

open weights. So I just chose it So I just chose it for demonstration. for demonstration. for demonstration. Of course, you can Of course, you can Of course, you can choose any choose any choose any other. And now the other. And now the other. And now the main thing is the main thing is the main thing is the settings. There are settings. There are settings. There are a a a lot of settings. Some of them are lot of settings. Some of them are lot of settings. Some of them are well known to people, well known to people, well known to people, some perhaps less so. some perhaps less so. some perhaps less so. So the size of the So the size of the So the size of the context window is context window is context window is also very important, also very important, also very important, because you always see because you always see that the model has such and that the model has such and such a such a such a context window size, context window size, context window size, but each provider... but each provider... Yes, some providers Yes, some providers Yes, some providers decide to provide a decide to provide a decide to provide a different size, smaller, different size, smaller, different size, smaller, for for for performance reasons. performance reasons. For example, our For example, our own inference own inference own inference also doesn't offer the also doesn't offer the also doesn't offer the full million G of GLM 5.2, full million G of GLM 5.2, full million G of GLM 5.2, only 256K, but this allows only 256K, but this allows only 256K, but this allows us us us to serve it to serve it to serve it faster, and the quality faster, and the quality faster, and the quality of the model degrades of the model degrades of the model degrades after a certain after a certain after a certain number of tokens. number of tokens. So, essentially, half of So, essentially, half of the context—it's already the context—it's already the context—it's already very, uh, very very, uh, very very, uh, very noticeable. And the thing is noticeable. And the thing is noticeable. And the thing is , it's , it's , it's expensive if you're expensive if you're expensive if you're constantly constantly constantly using a using a using a million and then million and then million and then compressing. So I compressing. So I compressing. So I personally think personally think personally think 256K is what I 256K is what I 256K is what I recommended to our recommended to our recommended to our inference team inference team inference team to use in to use in to use in this situation. Uh, I this situation. Uh, I this situation. Uh, I also use also use also use the same thing from Codex. Uh, the same thing from Codex. Uh, the same thing from Codex. Uh, basically, right now it gives you the basically, right now it gives you the basically, right now it gives you the best quality, it's the best quality, it's the best quality, it's the best best best balance of price, balance of price, balance of price, performance, and performance, and performance, and speed. Efforts for speed. Efforts for speed. Efforts for reflection and reflection and reflection and reasoning. This is a reasoning. This is a reasoning. This is a big factor, and I've big factor, and I've big factor, and I've done different done different done different assessments with different assessments with different assessments with different mindsets, and mindsets, and mindsets, and they really do have a they really do have a they really do have a big impact on the big impact on the big impact on the model, and sometimes a very model, and sometimes a very model, and sometimes a very strange impact, as you strange impact, as you strange impact, as you increase the depth of increase the depth of increase the depth of thinking, the scores thinking, the scores thinking, the scores drop, but I'll drop, but I'll drop, but I'll talk about that when talk about that when talk about that when we get to the slide.

we get to the slide. we get to the slide. So, the system So, the system So, the system prompt. You don't always prompt. You don't always prompt. You don't always have the option have the option have the option to change the system to change the system to change the system prompt or even prompt or even prompt or even check it, but check it, but check it, but it's a very important it's a very important it's a very important factor. And if you factor. And if you factor. And if you think about an agent think about an agent think about an agent like Open Law Hermes, you have a like Open Law Hermes, you have a like Open Law Hermes, you have a sole MD where you put in sole MD where you put in sole MD where you put in a lot of the instructions. a lot of the instructions. a lot of the instructions. Now imagine that you Now imagine that you Now imagine that you use this with use this with use this with a benchmark, and if you a benchmark, and if you a benchmark, and if you give the instruction "you are a give the instruction "you are a give the instruction "you are a helpful assistant," helpful assistant," helpful assistant," the result will be the result will be the result will be completely different than completely different than completely different than if you give if you give if you give the instruction "you are an the instruction "you are an the instruction "you are an autonomous agent autonomous agent autonomous agent that performs a that performs a that performs a task." I've seen task." I've seen task." I've seen results where a model results where a model results where a model or benchmark or benchmark or benchmark gave a bad gave a bad gave a bad result because result because result because the model tried the model tried the model tried to query the user to query the user , but the user , but the user , but the user wasn't there and it didn't wasn't there and it didn't wasn't there and it didn't know it. So, if I know it. So, if I know it. So, if I change the system change the system change the system prompt and tell her that prompt and tell her that prompt and tell her that you can't you can't you can't ask ask ask the user, she the user, she the user, she will behave will behave will behave differently and get a differently and get a differently and get a better score, I'm better score, I'm better score, I'm sure of that. And that sure of that. And that sure of that. And that will completely change what will completely change what will completely change what I test. So, there are I test. So, there are I test. So, there are so many variables here. so many variables here. so many variables here. Also Also Also tool configuration. If you tool configuration. If you tool configuration. If you give the agent give the agent give the agent network access, I network access, I network access, I have seen cases where have seen cases where have seen cases where the model went the model went the model went online and searched for a online and searched for a online and searched for a solution for the solution for the solution for the benchmark, since benchmark, since benchmark, since Terminal Bench is a Terminal Bench is a Terminal Bench is a public benchmark. And public benchmark. And public benchmark. And you need to you need to you need to look in the logs to look in the logs to look in the logs to see what see what see what happened here. This is actually not happened here. This is actually not happened here. This is actually not a bad thing, if a bad thing, if a bad thing, if you want to get a you want to get a you want to get a smart agent, it smart agent, it smart agent, it has to be smart. And has to be smart. And has to be smart. And if a solution already exists for the problem if a solution already exists for the problem if a solution already exists for the problem , why would , why would , why would he try to he try to he try to invent one when invent one when invent one when it's already it's already it's already published? So, it published? So, it published? So, it all depends on all depends on what exactly you want what exactly you want what exactly you want to measure and whether to measure and whether to measure and whether you want to provide you want to provide you want to provide access to it. And not all of them access to it. And not all of them access to it. And not all of them do it. Or not do it. Or not do it. Or not every time.

every time. every time. Sandbox resources, all that Sandbox resources, all that Sandbox resources, all that stuff. If you give stuff. If you give stuff. If you give less CPU or less CPU or less CPU or memory, they may memory, they may memory, they may time out or time out or time out or the work will slow down the work will slow down , causing execution , causing execution , causing execution to take longer or to take longer or to take longer or even even even fail altogether, and you will get a fail altogether, and you will get a fail altogether, and you will get a bad result. And bad result. And bad result. And this is not the model's fault, this is not the model's fault, this is not the model's fault, it is a consequence of the it is a consequence of the sandbox configuration. sandbox configuration. Task timeout, each Task timeout, each Task timeout, each task has task has task has a time limit. a time limit. a time limit. By default, By default, By default, they they they seem to range from 5-15 seem to range from 5-15 seem to range from 5-15 minutes to over 3 minutes to over 3 minutes to over 3 hours. One lasts 3 or hours. One lasts 3 or hours. One lasts 3 or so hours, and the other so hours, and the other about 2 hours. And that about 2 hours. And that about 2 hours. And that means you means you means you need to decide need to decide need to decide how much time you are how much time you are how much time you are willing to invest. willing to invest. When I started with Wolf When I started with Wolf Bench, I gave each Bench, I gave each Bench, I gave each task a fixed 2 task a fixed 2 task a fixed 2 hours because I didn't want to hours because I didn't want to hours because I didn't want to limit limit limit open-source models, open-source models, open-source models, which often run on which often run on which often run on less powerful less powerful output endpoints than models output endpoints than models from leading from leading from leading labs. So if labs. So if labs. So if you run DeepSeek on a you run DeepSeek on a you run DeepSeek on a local system, local system, local system, it usually doesn't run it usually doesn't run it usually doesn't run as fast as GPT on as fast as GPT on as fast as GPT on OpenAI servers. And it would OpenAI servers. And it would OpenAI servers. And it would distort the assessment distort the assessment distort the assessment if you didn't provide if you didn't provide if you didn't provide unlimited time. But unlimited time. But unlimited time. But I noticed that with a 2- I noticed that with a 2- hour limit, hour limit, hour limit, if a model can't if a model can't if a model can't handle handle handle a task, it sometimes a task, it sometimes a task, it sometimes just goes around in circles just goes around in circles : trying, : trying, : trying, trying, trying, trying, trying, and trying, and trying, and failing over and over again failing over and over again . It was very expensive . It was very expensive . It was very expensive and took a long time, because and took a long time, because and took a long time, because if you allocate 2 if you allocate 2 if you allocate 2 hours per task, and there are hours per task, and there are hours per task, and there are 89 of them, and I parallelize 89 of them, and I parallelize 89 of them, and I parallelize them depending on them depending on them depending on the degree of the degree of the degree of competition, competition, competition, which will be discussed later—it which will be discussed later—it which will be discussed later—it takes too much takes too much takes too much time. It takes time. It takes time. It takes a lot of time and is a lot of time and is a lot of time and is expensive, and expensive, and expensive, and you don't get the result because you don't get the result because you don't get the result because the model simply can't the model simply can't the model simply can't do the job.

do the job. do the job. So I've noticed that 1 So I've noticed that 1 So I've noticed that 1 hour is a good hour is a good hour is a good guideline. If you guideline. If you guideline. If you use use use the standard Terminal Bench, the the standard Terminal Bench, the testing time will increase. testing time will increase. Competitiveness is Competitiveness is Competitiveness is exactly what I meant exactly what I meant exactly what I meant . You don't want to . You don't want to . You don't want to wait 89 hours or wait 89 hours or wait 89 hours or even twice as long. even twice as long. even twice as long. You want to You want to You want to use use use concurrency to concurrency to concurrency to execute multiple execute multiple execute multiple attempts at the same time. But attempts at the same time. But attempts at the same time. But then the question arises: then the question arises: then the question arises: how high how high how high can it be? If I can it be? If I can it be? If I set set set the concurrency to 89, the concurrency to 89, the concurrency to 89, I can run all I can run all I can run all 89 Terminal Bench tasks 89 Terminal Bench tasks 89 Terminal Bench tasks simultaneously. But if simultaneously. But if simultaneously. But if my output endpoint isn't my output endpoint isn't fast enough and I get fast enough and I get a limit on a limit on a limit on query rate, I'll query rate, I'll query rate, I'll end up with end up with end up with poor ratings or poor ratings or poor ratings or none at all. Same none at all. Same none at all. Same with sandboxes. with sandboxes. with sandboxes. If I have a If I have a If I have a sandbox provider, sandbox provider, sandbox provider, I get resources, and I get resources, and I get resources, and if I use if I use if I use 8GB per sandbox, then 8GB per sandbox, then 8GB per sandbox, then yes, it yes, it yes, it adds up quickly and adds up quickly and adds up quickly and you might you might you might go over the go over the go over the allowed limit. allowed limit. allowed limit. Retry policy Retry policy Retry policy , how often , how often , how often to retry something. This is to retry something. This is to retry something. This is also important: if I also important: if I also important: if I don't don't don't retry the task retry the task retry the task and it fails and it fails and it fails due to a due to a due to a sandbox crash, the model sandbox crash, the model sandbox crash, the model will get a low will get a low will get a low score. But it was score. But it was score. But it was n't the model's fault. n't the model's fault. n't the model's fault. So, there are mistakes that are So, there are mistakes that are So, there are mistakes that are worth worth worth trying again, and trying again, and trying again, and mistakes that are mistakes that are mistakes that are not worth trying again. For example, not worth trying again. For example, not worth trying again. For example, if the model if the model if the model causes itself to fail causes itself to fail causes itself to fail by doing something stupid, that's by doing something stupid, that's by doing something stupid, that's exactly the kind of failure exactly the kind of failure exactly the kind of failure you want you want you want to measure.

to measure. to measure. Temperature, a Temperature, a Temperature, a common thing. Hmm, common thing. Hmm, common thing. Hmm, this is a tricky point this is a tricky point this is a tricky point for the benchmark, because what for the benchmark, because what for the benchmark, because what temperature should temperature should temperature should I use? I use? I use? Some Some Some model providers provide a model providers provide a model providers provide a recommended recommended recommended temperature for temperature for temperature for operating their operating their operating their model. Some don't. And model. Some don't. And model. Some don't. And some agents have some agents have some agents have default settings default settings default settings , others , others , others decide based on a decide based on a decide based on a model. There are so model. There are so model. There are so many combinations that many combinations that many combinations that you have to decide: you have to decide: you have to decide: do you want to look for the do you want to look for the do you want to look for the recommended recommended recommended value for each value for each value for each model? If there is model? If there is model? If there is none, what should none, what should none, what should I use? How do I use? How do I use? How do you make sure that people you make sure that people running the running the running the same benchmark same benchmark same benchmark are using the are using the are using the same settings as same settings as same settings as you? This is difficult. And you? This is difficult. And you? This is difficult. And also Top P: should also Top P: should also Top P: should I use 0.95, I use 0.95, I use 0.95, or one, or any or one, or any or one, or any other value? This is other value? This is other value? This is just one just one just one setting, and there are setting, and there are setting, and there are many others. And many others. And many others. And the maximum the maximum the maximum number of new number of new number of new tokens: how many tokens: how many tokens: how many tokens are tokens are tokens are models allowed to generate? You models allowed to generate? You models allowed to generate? You have to decide have to decide have to decide on this, as there are on this, as there are on this, as there are recommendations from recommendations from recommendations from providers, there are providers, there are providers, there are restrictions from restrictions from restrictions from hosting providers. hosting providers. hosting providers. You need to You need to You need to choose something, and usually it choose something, and usually it choose something, and usually it needs to be configured. needs to be configured. needs to be configured. And if possible, And if possible, And if possible, you can also you can also you can also set a seed to set a seed to set a seed to reduce reduce reduce randomness. When I randomness. When I randomness. When I started testing, I started testing, I started testing, I used used used a temperature of zero, a temperature of zero, a temperature of zero, so there was less so there was less so there was less chance of randomness. But chance of randomness. But chance of randomness. But randomness never randomness never randomness never completely disappeared.

completely disappeared. completely disappeared. I've seen even I've seen even I've seen even small things small things small things like like like driver updates change the driver updates change the driver updates change the results I get. results I get. results I get. Even if everything was Even if everything was Even if everything was deterministic or deterministic or deterministic or looked that looked that looked that way. Because we have way. Because we have floating point rounding errors floating point rounding errors , , , performance optimizations—so performance optimizations—so performance optimizations—so many effects that many effects that many effects that change the result, change the result, change the result, even at even at even at zero temperature. And zero temperature. And zero temperature. And today, when the model today, when the model today, when the model recommends recommends recommends a temperature of one, a temperature of one, a temperature of one, of course, you shouldn't of course, you shouldn't of course, you shouldn't set it to zero. Otherwise, set it to zero. Otherwise, set it to zero. Otherwise, you will go against you will go against you will go against the recommendation. In the recommendation. In the recommendation. In this case, you can this case, you can this case, you can set a seed, but set a seed, but set a seed, but even if you even if you even if you choose one like 42, choose one like 42, it is very popular. it is very popular. it is very popular. You know why. You know why. You know why. The fact is that The fact is that The fact is that one model may have one model may have one model may have more luck with more luck with more luck with this seed, while another may have less this seed, while another may have less . These are all choices you . These are all choices you . These are all choices you have to make. Now have to make. Now have to make. Now regarding this choice of regarding this choice of regarding this choice of which inference we which inference we which inference we want want want to use: to use: to use: on the left I have a on the left I have a on the left I have a regular regular regular inference core with inference core with inference core with serverless serverless serverless inference. If you inference. If you inference. If you want to follow want to follow want to follow me and want to me and want to me and want to use Open use Open use Open Router because you didn't get your Router because you didn't get your Router because you didn't get your own key, you own key, you own key, you can use can use can use it. Anyway it. Anyway , you just , you just , you just use the env use the env file and insert file and insert file and insert the variable at the bottom here. You the variable at the bottom here. You the variable at the bottom here. You just put a just put a just put a variable, a key in the env file variable, a key in the env file variable, a key in the env file and we will use that and we will use that and we will use that with the agent. Um, if you with the agent. Um, if you with the agent. Um, if you use Open use Open use Open Router, you need to Router, you need to Router, you need to set some set some set some settings to settings to settings to use use use our provider, One our provider, One our provider, One API. This is very important API. This is very important API. This is very important if you if you if you are using Open are using Open are using Open Router to test the Router to test the Router to test the model. There are so model. There are so model. There are so many providers out there, and many providers out there, and many providers out there, and by default you by default you by default you get one of get one of get one of them. They them. They them. They decide for themselves which is decide for themselves which is decide for themselves which is best at that best at that best at that moment. So, if you moment. So, if you moment. So, if you run a single run a single run a single test on 89 tasks, you test on 89 tasks, you test on 89 tasks, you might get

might get might get half the tasks from half the tasks from half the tasks from one provider and one provider and one provider and half from another. half from another. half from another. Therefore, you Therefore, you Therefore, you must specify must specify must specify which one you want to which one you want to which one you want to use. And use. And use. And after all that, you after all that, you after all that, you chose one part chose one part chose one part for each of the six for each of the six for each of the six levels we levels we levels we went through. We chose went through. We chose went through. We chose Terminal Bench for our Terminal Bench for our Terminal Bench for our evaluation, chose Harbor and evaluation, chose Harbor and evaluation, chose Harbor and its specific its specific its specific version, chose our version, chose our sandboxes, and Terminus 2 sandboxes, and Terminus 2 sandboxes, and Terminus 2 as the agent. GLM 5.2, and as the agent. GLM 5.2, and as the agent. GLM 5.2, and we have specific we have specific we have specific settings. If you settings. If you settings. If you are conducting an assessment, you are conducting an assessment, you are conducting an assessment, you will do the same. Whatever will do the same. Whatever will do the same. Whatever framework you framework you framework you use, whatever use, whatever use, whatever model you choose, you model you choose, you model you choose, you have to make a choice and have to make a choice and have to make a choice and clearly understand what you are clearly understand what you are clearly understand what you are using. using. using. This is important when you This is important when you This is important when you publish your publish your publish your results. So that others results. So that others results. So that others know what you know what you know what you actually actually actually used and used and used and measured. So, here's measured. So, here's measured. So, here's another QR code. It another QR code. It another QR code. It provides full provides full provides full configuration so you configuration so you configuration so you don't have to don't have to don't have to copy from slides, copy from slides, copy from slides, but here the formatting is but here the formatting is but here the formatting is a bit off. a bit off. a bit off. Sorry about that. Um, Sorry about that. Um, Sorry about that. Um, if you download if you download if you download this configuration, I'll this configuration, I'll this configuration, I'll show you on the next show you on the next show you on the next slide how to slide how to slide how to run it, and it'll take run it, and it'll take run it, and it'll take about 5 minutes to about 5 minutes to about 5 minutes to get the results, get the results, get the results, I'll do the same. So I'll do the same. So , if you want to , if you want to , if you want to download the config, download the config, download the config, here is the QR code. And I'll move on here is the QR code. And I'll move on here is the QR code. And I'll move on to the next one. So to the next one. So to the next one. So let me not let me not let me not just show it, but just show it, but just show it, but also run it. So take also run it. So take also run it. So take a look at this. We a look at this. We a look at this. We connect the .env file connect the .env file connect the .env file to get the API key, to get the API key, to get the API key, which can be found which can be found which can be found two slides earlier, and two slides earlier, and two slides earlier, and then we run this with then we run this with then we run this with the command “uv run harbor run the command “uv run harbor run the command “uv run harbor run config,” passing in config,” passing in config,” passing in the configuration that you the configuration that you the configuration that you downloaded or

downloaded or downloaded or created, and then we created, and then we created, and then we can get the can get the can get the results. I'll do results. I'll do results. I'll do it right now. In it right now. In it right now. In the terminal. So, the terminal. So, the terminal. So, first, we install the first, we install the first, we install the Harbor framework. Harbor framework. Harbor framework. Executing the command. I'll Executing the command. I'll Executing the command. I'll zoom in a little zoom in a little zoom in a little . So, if . So, if . So, if my Wi-Fi is working properly my Wi-Fi is working properly , everything is fine. OK. , everything is fine. OK. , everything is fine. OK. Let's leave it to Let's leave it to Let's leave it to work. I'll be work. I'll be work. I'll be back in a minute. back in a minute. So let's continue So let's continue the presentation. We the presentation. We the presentation. We run the process and run the process and run the process and it will only run one it will only run one it will only run one test. I didn't want to test. I didn't want to test. I didn't want to use 89 use 89 use 89 tests for this, tests for this, tests for this, otherwise we would be otherwise we would be otherwise we would be delayed by an delayed by an delayed by an hour, and that's just the time hour, and that's just the time hour, and that's just the time I allocate for I allocate for I allocate for each test. Afterwards, each test. Afterwards, the verifier should run to the verifier should run to check check check the correctness of the correctness of the correctness of the execution. This also the execution. This also the execution. This also takes time, so it takes time, so it takes time, so it lasts even longer. But lasts even longer. But lasts even longer. But Harbor pulls up Harbor pulls up Harbor pulls up the task, the the task, the the task, the Terminal Bench 2.0 dataset, Terminal Bench 2.0 dataset, Terminal Bench 2.0 dataset, using it using it using it from the configuration. Here is from the configuration. Here is from the configuration. Here is the part where the part where the part where the configuration specifies the configuration specifies the configuration specifies Terminal Bench version 2.0.

Terminal Bench version 2.0. Then it launches the " sandbox" sandbox" sandbox" using the using the using the chosen provider. chosen provider. chosen provider. It creates a " It creates a " sandbox" for sandbox" for sandbox" for each task each task each task depending on the level of depending on the level of depending on the level of parallelism. Next, it parallelism. Next, it parallelism. Next, it launches an agent that launches an agent that launches an agent that enters the enters the sandbox, sandbox, sandbox, installs the agent, and installs the agent, and installs the agent, and performs the task, performs the task, performs the task, providing providing providing the user with a system prompt with the user with a system prompt with the user with a system prompt with instructions on what exactly instructions on what exactly instructions on what exactly needs to be done. needs to be done. needs to be done. The agent does this. The agent does this. After that, the results from the sandbox are collected before it sandbox are collected before it sandbox are collected before it is destroyed. It also is destroyed. It also is destroyed. It also gets trajectories— gets trajectories— what exactly is the agent doing? what exactly is the agent doing? what exactly is the agent doing? This way you This way you This way you get a lot of get a lot of get a lot of information. This is a information. This is a information. This is a small task, small task, small task, one of the smallest: there is a one of the smallest: there is a one of the smallest: there is a Git repository where Git repository where Git repository where something has been lost, and something has been lost, and something has been lost, and you need to find it in the you need to find it in the you need to find it in the Git tree and Git tree and Git tree and restore it. This takes restore it. This takes restore it. This takes 3 to 5 minutes. 3 to 5 minutes. 3 to 5 minutes. Let's see how we Let's see how we Let's see how we progress. Already progress. Already progress. Already installed. Then we installed. Then we installed. Then we install our install our install our One B client. The next One B client. The next One B client. The next step is to get my step is to get my step is to get my API key from OnePassword. And API key from OnePassword. And API key from OnePassword. And then I write then I write then I write the configuration here. This is the one the configuration here. This is the one the configuration here. This is the one you can you can you can download. I just download. I just download. I just redirect it to a redirect it to a redirect it to a file via "cat". And file via "cat". And file via "cat". And I download the .env file.

I download the .env file. I download the .env file. Now the API key is in my Now the API key is in my Now the API key is in my environment variables and environment variables and environment variables and I just...this is the I just...this is the I just...this is the launch command. We are launching launch command. We are launching launch command. We are launching the evaluation. So, the evaluation. So, the evaluation. So, now it starts now it starts now it starts working, starts the working, starts the sandbox, sandbox, sandbox, installs the agent and installs the agent and installs the agent and runs it. And it all runs it. And it all runs it. And it all happens here. happens here. And while that's And while that's happening, I'll happening, I'll happening, I'll get back to the get back to the get back to the presentation. presentation. Next part. Next part. Run as evidence, Run as evidence, Run as evidence, not just an estimate. not just an estimate. not just an estimate. The formatting here is The formatting here is The formatting here is completely messed up. I'll completely messed up. I'll completely messed up. I'll fix the presentation fix the presentation and we'll upload it. and we'll upload it. and we'll upload it. This is simply a catalog of what This is simply a catalog of what This is simply a catalog of what is is is in the evaluation folder. in the evaluation folder. in the evaluation folder. We can create this We can create this We can create this live as soon as live as soon as live as soon as it's it's it's finished. Of course, finished. Of course, finished. Of course, it will take about 5 it will take about 5 it will take about 5 minutes. minutes. The environment that The environment that creates sandboxes is now running. creates sandboxes is now running. creates sandboxes is now running. So just let it So just let it So just let it work. A. It's too work. A. It's too work. A. It's too early. And when we early. And when we early. And when we receive it, we will be given receive it, we will be given receive it, we will be given one assignment that one assignment that one assignment that will be graded. It will be graded. It will be graded. It will receive a score of either will receive a score of either will receive a score of either one or zero. Success one or zero. Success one or zero. Success or failure? If or failure? If or failure? If you do this for 89 you do this for 89 you do this for 89 tasks, you get tasks, you get tasks, you get the number of the number of the number of tasks completed, and then tasks completed, and then tasks completed, and then you do this, you do this, you do this, for example, five for example, five for example, five times, you get times, you get times, you get the percentage and you know the percentage and you know the percentage and you know what percentage of Terminal what percentage of Terminal what percentage of Terminal Bench the model received. And Bench the model received. And Bench the model received. And that's usually the estimate that that's usually the estimate that labs report.

labs report. Most people Most people Most people simply give an simply give an simply give an average of average of average of five runs. And this is five runs. And this is five runs. And this is the installation of Wolf Bench the installation of Wolf Bench the installation of Wolf Bench Toolkit. What we did Toolkit. What we did Toolkit. What we did before was the before was the before was the Harbor framework and Harbor framework and Harbor framework and Terminal Bench, and that's what I Terminal Bench, and that's what I Terminal Bench, and that's what I programmed, what I programmed, what I programmed, what I built. Wolf Bench Toolkit built. Wolf Bench Toolkit is a way to work with is a way to work with is a way to work with this data, because this data, because this data, because otherwise you have a few otherwise you have a few otherwise you have a few files and some kind of score files and some kind of score , but how do you , but how do you , but how do you analyze it? So this is a analyze it? So this is a analyze it? So this is a manifold, this is what manifold, this is what manifold, this is what Wolf Bench does, and I... it would be Wolf Bench does, and I... it would be best if I best if I best if I showed it when I showed it when I showed it when I opened the program. So opened the program. So opened the program. So let me let me let me come back here. come back here. come back here. While that's happening, While that's happening, While that's happening, I'll just run this in a I'll just run this in a I'll just run this in a new terminal. We new terminal. We new terminal. We just clone it just clone it just clone it and go inside. and go inside. and go inside. So, it's open So, it's open So, it's open source, Apache 2 license. source, Apache 2 license. source, Apache 2 license. You can You can You can use it use it use it for your own projects. for your own projects. for your own projects. Of course, it will Of course, it will Of course, it will have to have to have to be adapted if you be adapted if you be adapted if you use something use something use something other than Terminal Bench, but other than Terminal Bench, but other than Terminal Bench, but it's easy it's easy it's easy to do now. Just to do now. Just to do now. Just tell your tell your tell your agent. Quick agent. Quick agent. Quick question. Who question. Who question. Who uses agents uses agents uses agents for things like this? Not for things like this? Not for things like this? Not everyone? Oh, I'm surprised.

everyone? Oh, I'm surprised. everyone? Oh, I'm surprised. By the way, I still see a By the way, I still see a By the way, I still see a lot of notebooks. lot of notebooks. lot of notebooks. Because some people Because some people Because some people now just now just now just pull out their phone, pull out their phone, pull out their phone, take a take a take a screenshot, and say, "Okay screenshot, and say, "Okay , agent, go ahead." " , agent, go ahead." " Do everything he Do everything he Do everything he shows on stage." shows on stage." shows on stage." Next year. So Next year. So , it's loading , it's loading , and I'm also loading , and I'm also loading , and I'm also loading all the results I all the results I all the results I got from all the got from all the got from all the runs for today. runs for today. runs for today. So a quick spoiler. So a quick spoiler. So a quick spoiler. I have done 430 runs I have done 430 runs I have done 430 runs so far. Hmm, so far. Hmm, so far. Hmm, where is this? Uh, that was 430 where is this? Uh, that was 430 where is this? Uh, that was 430 launches. But launches. But anyway, here are the anyway, here are the anyway, here are the agents I've agents I've agents I've tested so far tested so far tested so far . Cloud code, Hermes agent, . Cloud code, Hermes agent, . Cloud code, Hermes agent, open clock cursor, Codex. Client has open clock cursor, Codex. Client has open clock cursor, Codex. Client has not yet been published. He's not yet been published. He's not yet been published. He's on my list, as are on my list, as are on my list, as are some other agents. some other agents. some other agents. If you create If you create If you create an agent, contact an agent, contact an agent, contact me if you want it to me if you want it to me if you want it to appear too. And appear too. And appear too. And some of the models I some of the models I some of the models I tested, many of tested, many of tested, many of them on W&B. This was based on them on W&B. This was based on them on W&B. This was based on our own our own our own inference. inference. inference. So, the final So, the final So, the final cost, unfortunately, cost, unfortunately, cost, unfortunately, was not as high was not as high as shown here, as shown here, as shown here, almost 40 thousand. So, almost 40 thousand. So, almost 40 thousand. So, some of this is based on some of this is based on some of this is based on our own our own our own judgment, but I judgment, but I judgment, but I just used the just used the just used the published prices.

published prices. published prices. So, let's see what So, let's see what So, let's see what we have here. It we have here. It we have here. It loaded. So, loaded. So, loaded. So, 13 megabytes—these are the 13 megabytes—these are the 13 megabytes—these are the results. If you results. If you results. If you want to see all want to see all want to see all the trajectories, they are the trajectories, they are the trajectories, they are loaded into Weave and loaded into Weave and loaded into Weave and you can just you can just you can just browse through them. You don't browse through them. You don't browse through them. You don't need to need to need to download download download gigabytes of data. gigabytes of data. Now if you want to Now if you want to use Wolf use Wolf use Wolf Bench you can Bench you can Bench you can adjust some adjust some adjust some variables, it's also a variables, it's also a variables, it's also a bootloader but I'll bootloader but I'll bootloader but I'll show you once I show you once I show you once I run it. This run it. This run it. This uses the uses the uses the Marine notebook to provide a Marine notebook to provide a Marine notebook to provide a dashboard dashboard , a convenient way to , a convenient way to , a convenient way to process all process all process all assessment data. So, assessment data. So, assessment data. So, minute by minute. minute by minute. minute by minute. Here it is. So, this is a Here it is. So, this is a Here it is. So, this is a dashboard. dashboard. dashboard. You can enter your You can enter your You can enter your information where information where information where you want to upload you want to upload you want to upload the trace if the trace if the trace if you have Weave data. Then you have Weave data. Then you have Weave data. Then you can find the you can find the you can find the virtual machines. virtual machines. virtual machines. If you If you If you are using XE dev are using XE dev are using XE dev or Hetzner, it will find or Hetzner, it will find or Hetzner, it will find all your virtual all your virtual all your virtual machines and you can machines and you can machines and you can filter which filter which filter which ones are evaluation. ones are evaluation. ones are evaluation. Because I usually don't Because I usually don't Because I usually don't run Harbor Framework on run Harbor Framework on run Harbor Framework on my laptop. I'm my laptop. I'm my laptop. I'm running it on a running it on a running it on a virtual machine, virtual machine, virtual machine, so I can turn off my so I can turn off my so I can turn off my laptop, or my Wi-Fi laptop, or my Wi-Fi laptop, or my Wi-Fi won't turn off and won't turn off and won't turn off and the connection to the the connection to the the connection to the sandboxes won't sandboxes won't sandboxes won't be interrupted, because that be interrupted, because that be interrupted, because that could also could also could also affect something. In this affect something. In this affect something. In this case, everything is local case, everything is local case, everything is local once I once I once I load the data, which I load the data, which I load the data, which I did. So I can did. So I can did. So I can just scan just scan just scan my local my local my local directory and it will find directory and it will find directory and it will find all the data from runs all the data from runs all the data from runs I've already run. Here are I've already run. Here are I've already run. Here are 443 unique launches.

443 unique launches. And here And here the filter is shown, so every the filter is shown, so every the filter is shown, so every run that has a run that has a run that has a score is measured. score is measured. score is measured. He has to pass all the He has to pass all the He has to pass all the tests. If any tests. If any tests. If any tests were not tests were not tests were not run, this is not a run, this is not a run, this is not a valid valid valid run result. It was run result. It was run result. It was just a test just a test just a test run. It run. It run. It shouldn't have too shouldn't have too shouldn't have too many errors or many errors or many errors or timeouts. And then I timeouts. And then I timeouts. And then I get a list of get a list of get a list of all of that, and I can all of that, and I can all of that, and I can choose which choose which choose which models I want to models I want to models I want to create a nice create a nice create a nice graph for. In this graph for. In this graph for. In this case, I'll just case, I'll just case, I'll just use them all. use them all. use them all. Then you can also Then you can also Then you can also change the change the change the model names, and sometimes you model names, and sometimes you model names, and sometimes you have to change the have to change the have to change the "thinking" as well, "thinking" as well, "thinking" as well, because the parameter because the parameter because the parameter you set can you set can you set can be changed by be changed by be changed by the agent. For example, the agent. For example, the agent. For example, if you if you if you are using the are using the are using the "thinking extra high" mode or "thinking extra high" mode or "thinking extra high" mode or want to use the want to use the want to use the "thinking max" mode and your "thinking max" mode and your "thinking max" mode and your agent doesn't agent doesn't agent doesn't support it, support it, support it, it may be called it may be called it may be called something else. And then you something else. And then you something else. And then you need to change that so that need to change that so that need to change that so that it really it really it really reflects what reflects what reflects what was was was actually used. Now that actually used. Now that actually used. Now that all that is done, all that is done, all that is done, I can just I can just I can just create my create my create my results file. You results file. You results file. You can download can download can download it from We view if it from We view if it from We view if you have the key. I didn't you have the key. I didn't you have the key. I didn't set up the key, so I do set up the key, so I do set up the key, so I do n't have n't have n't have that button right now. Now I that button right now. Now I that button right now. Now I create a create a create a save file and it is a single save file and it is a single save file and it is a single JSON file that contains JSON file that contains JSON file that contains all the results. You all the results. You all the results. You can share can share can share it with other people. And it with other people. And it with other people. And finally, I create finally, I create finally, I create a diagram, which is an a diagram, which is an a diagram, which is an artifact that I artifact that I artifact that I generate. So, now from generate. So, now from generate. So, now from all the all the all the results I get, I results I get, I results I get, I get a new get a new get a new chart. This is the same thing chart. This is the same thing chart. This is the same thing that I then, in the that I then, in the that I then, in the next step, next step, next step, also have a also have a also have a bootloader.

bootloader. bootloader. Once the chart Once the chart Once the chart is generated, I simply is generated, I simply is generated, I simply upload it to upload it to upload it to wolfbenchmark.ai and it wolfbenchmark.ai and it wolfbenchmark.ai and it updates. If I updates. If I updates. If I activate this option to activate this option to activate this option to replace index.html, it replace index.html, it replace index.html, it instantly comes instantly comes instantly comes online. So, online. So, online. So, back to the back to the back to the present moment present moment . Oh, let's just . Oh, let's just . Oh, let's just check how the check how the check how the benchmark went. Oh, benchmark went. Oh, benchmark went. Oh, it's over. it's over. it's over. You see, he's ready. You see, he's ready. You see, he's ready. Tests: one, Tests: one, Tests: one, average score: 1.0. So, average score: 1.0. So, average score: 1.0. So, he had one he had one he had one task, he task, he task, he completed it, that is, everything completed it, that is, everything completed it, that is, everything worked out. He worked out. He worked out. He finished the work and finished the work and finished the work and fixed the Git fixed the Git repository. This repository. This repository. This task is now in task is now in task is now in my catalog and I my catalog and I my catalog and I could could could import it, but it's import it, but it's import it, but it's not a full run, not a full run, not a full run, so so so it it it doesn't show up by default. And doesn't show up by default. And doesn't show up by default. And once we once we once we do that and do that and do that and download it, we download it, we download it, we can see can see can see what the Wolf benchmark has to offer. what the Wolf benchmark has to offer. what the Wolf benchmark has to offer. This is again a This is again a This is again a question of thinking: question of thinking: question of thinking: what do we want what do we want what do we want to look at? And in this to look at? And in this to look at? And in this case, that's what we case, that's what we case, that's what we just did. I just did. I just did. I showed you the showed you the showed you the dashboard. dashboard. dashboard. And now we need And now we need And now we need to look at the to look at the to look at the leaderboard and what leaderboard and what leaderboard and what it means, because the it means, because the it means, because the leaderboard itself, whatever leaderboard itself, whatever leaderboard itself, whatever it generates, it generates, it generates, is not just is not just is not just an answer, it an answer, it an answer, it helps you find a helps you find a helps you find a better question.

better question. better question. For example, what are you For example, what are you For example, what are you looking at? Are you looking at? Are you looking at? Are you interested in the interested in the interested in the model's capability model's capability model's capability or its reliability? And I'll or its reliability? And I'll or its reliability? And I'll show you what that show you what that show you what that means by going means by going means by going to wolfbenchmark.ai. And the thing is, to wolfbenchmark.ai. And the thing is, to wolfbenchmark.ai. And the thing is, I'm just going to I'm just going to I'm just going to make it a little make it a little make it a little smaller. So, they're smaller. So, they're smaller. So, they're sorted from left sorted from left sorted from left to right, and to right, and to right, and I have all the information here I have all the information here . Most of the tests . Most of the tests . Most of the tests you see you see you see only show a only show a only show a total score and total score and total score and only an only an only an average. So, this is an average. So, this is an average. So, this is an interactive interactive interactive page that page that page that is used not is used not is used not only to only to only to display display display results, but also to results, but also to results, but also to create graphs create graphs create graphs that can then be that can then be that can then be copied, pasted, copied, pasted, copied, pasted, and used and used and used elsewhere. There's a screenshot icon here elsewhere. There's a screenshot icon here elsewhere. There's a screenshot icon here , so I , so I , so I can show this, can show this, can show this, for example, in X. If I for example, in X. If I for example, in X. If I want to show how want to show how want to show how this model did, this model did, this model did, I can just I can just I can just mark it. I can mark it. I can mark it. I can delete some delete some delete some columns or change columns or change columns or change their order. So, their order. So, their order. So, it's interactive. So it's interactive. So it's interactive. So I can create my I can create my I can create my report like this. And report like this. And report like this. And this is also reflected in the this is also reflected in the this is also reflected in the URL. So if URL. So if URL. So if you send someone you send someone you send someone this link, they this link, they this link, they will see the same thing will see the same thing will see the same thing you see on their screen.

you see on their screen. you see on their screen. So, I'm switching So, I'm switching So, I'm switching here. And sometimes it here. And sometimes it here. And sometimes it looks a little looks a little looks a little complicated because there's complicated because there's too much information in one column. As I said, you can As I said, you can filter filter filter the information to the information to the information to leave a simple leave a simple leave a simple column, but I column, but I column, but I like to like to like to see it this way because the dark see it this way because the dark see it this way because the dark part at the bottom is the part at the bottom is the part at the bottom is the percentage of problems that percentage of problems that percentage of problems that the model solved in the model solved in the model solved in each of the five each of the five each of the five runs. This is what I runs. This is what I runs. This is what I call stability call stability . This model and this . This model and this . This model and this agent agent agent solved 60% of solved 60% of the tasks of the entire test every time. the tasks of the entire test every time. the tasks of the entire test every time. This means that if This means that if This means that if the score is very high, the score is very high, the score is very high, the model is very the model is very the model is very reliable, because if reliable, because if reliable, because if it solves the it solves the it solves the problem, you can problem, you can problem, you can expect it to expect it to expect it to always do so. always do so. always do so. Whereas the yellow score Whereas the yellow score is the average, is the average, is the average, the same result the same result the same result that is usually that is usually that is usually published. And the published. And the published. And the up and down arrows around it up and down arrows around it up and down arrows around it are the best and are the best and are the best and worst results worst results worst results out of five runs. If out of five runs. If out of five runs. If they are very close, they are very close, they are very close, then the variance is then the variance is then the variance is small. So, you small. So, you small. So, you can expect the can expect the can expect the same score. If same score. If same score. If they are very far apart they are very far apart they are very far apart , as we , as we , as we see here (79% to 60% see here (79% to 60% ), this means a lot of ), this means a lot of ), this means a lot of variability. And this is a variability. And this is a variability. And this is a sign that there is a sign that there is a sign that there is a lot of randomness in the model's work lot of randomness in the model's work lot of randomness in the model's work . You . You . You cannot fully cannot fully cannot fully trust her to trust her to trust her to complete the task.

complete the task. complete the task. Even if the score is Even if the score is Even if the score is high, her work is high, her work is high, her work is somewhat haphazard. And the somewhat haphazard. And the somewhat haphazard. And the highest mark is an highest mark is an highest mark is an asterisk, it's a "ceiling." asterisk, it's a "ceiling." asterisk, it's a "ceiling." What percentage of tests What percentage of tests What percentage of tests did she solve did she solve did she solve at all, at least at all, at least at all, at least once? This is the once? This is the once? This is the number of tasks number of tasks number of tasks that shows that in the that shows that in the that shows that in the ideal case, ideal case, ideal case, this is the score that the model this is the score that the model this is the score that the model should receive should receive should receive constantly if it is constantly if it is constantly if it is perfect and always perfect and always perfect and always copes. Although that would copes. Although that would copes. Although that would also be an also be an also be an indicator that indicator that indicator that she was probably she was probably she was probably trained specifically for trained specifically for trained specifically for this test. Um, but this test. Um, but this test. Um, but that shows you how that shows you how that shows you how far it can far it can far it can go. So what you go. So what you go. So what you need is a very need is a very need is a very high ceiling, a very high ceiling, a very high ceiling, a very stable stable stable average, and a very average, and a very average, and a very solid base. And if solid base. And if solid base. And if the model has the the model has the the model has the same score and they are same score and they are same score and they are almost the same, you almost the same, you almost the same, you can still can still can still see that the reliable see that the reliable see that the reliable base is different. And base is different. And base is different. And in that case, I would choose the one with the in that case, I would choose the one with the in that case, I would choose the one with the higher higher higher reliable base. Like here, reliable base. Like here, reliable base. Like here, from 73% to 74%. It's almost the same thing from 73% to 74%. It's almost the same thing from 73% to 74%. It's almost the same thing . If . If . If you look at just the you look at just the you look at just the GPA, it's GPA, it's GPA, it's almost the same. There may almost the same. There may almost the same. There may be variability, be variability, be variability, just an error. But that's a just an error. But that's a just an error. But that's a bigger difference bigger difference . That's 6 percentage . That's 6 percentage . That's 6 percentage points, and that's a points, and that's a points, and that's a noticeable gap. So noticeable gap. So , you see, "extra- , you see, "extra- high thinking" is high thinking" is high thinking" is better than "average better than "average better than "average thinking," as we thinking," as we thinking," as we expected. But if expected. But if expected. But if we look at, um...

we look at, um... that's also very interesting. that's also very interesting. that's also very interesting. I will show this in I will show this in I will show this in the presentation. the presentation. the presentation. Let me Let me Let me switch to switch to switch to the presentation. So these the presentation. So these the presentation. So these were different ways were different ways were different ways to imagine the model. Ah, to imagine the model. Ah, to imagine the model. Ah, risk and risk and cost tolerance. I have to show you this cost tolerance. I have to show you this because it's something... because it's something... If you look at the If you look at the If you look at the second best second best second best model in my model in my model in my ranking, it was the ranking, it was the ranking, it was the Gemini 3.5 Flash. So this Gemini 3.5 Flash. So this Gemini 3.5 Flash. So this model, the flash model, is model, the flash model, is model, the flash model, is almost as good almost as good almost as good as GPT 5.5. So what's the as GPT 5.5. So what's the as GPT 5.5. So what's the matter? If I just matter? If I just matter? If I just look at the terms look at the terms look at the terms so we don't have so so we don't have so so we don't have so many columns, I many columns, I many columns, I can also say can also say can also say that I only want to compare that I only want to compare that I only want to compare these two. And yes, it is a these two. And yes, it is a these two. And yes, it is a great model, the Gemini 3.5 great model, the Gemini 3.5 great model, the Gemini 3.5 Flash. But if I Flash. But if I Flash. But if I look at how look at how look at how she achieved this, she achieved this, she achieved this, I have a 3D view that I have a 3D view that I have a 3D view that shows how many shows how many shows how many tokens she tokens she tokens she used to do used to do used to do it. And it took it. And it took it. And it took billions of tokens to billions of tokens to billions of tokens to get that score, get that score, get that score, while GPT 5.5 while GPT 5.5 while GPT 5.5 used significantly used significantly used significantly less. Let me less. Let me less. Let me zoom in. So, 136 zoom in. So, 136 zoom in. So, 136 million tokens million tokens million tokens for GPT 5.5 compared to for GPT 5.5 compared to for GPT 5.5 compared to 1.65 billion tokens 1.65 billion tokens 1.65 billion tokens for Gemini 3.5 Flash. And for Gemini 3.5 Flash. And for Gemini 3.5 Flash. And the thing is, if the thing is, if the thing is, if we also look we also look we also look at the price, which is another at the price, which is another at the price, which is another 3D view, it's the 3D view, it's the 3D view, it's the cost. And I cost. And I cost. And I can even show both at the can even show both at the can even show both at the same time. The cost same time. The cost same time. The cost is now displayed in a is now displayed in a is now displayed in a shadow. You see shadow. You see shadow. You see the cost is right here.

the cost is right here. the cost is right here. It overlaps It overlaps It overlaps because because because so many tokens were used, so many tokens were used, so many tokens were used, but it shows that Gemini but it shows that Gemini but it shows that Gemini 3.5 Flash is very close to 3.5 Flash is very close to 3.5 Flash is very close to GPT 5.5, but it GPT 5.5, but it GPT 5.5, but it used so used so used so many tokens that it many tokens that it many tokens that it cost even cost even cost even more than GPT 5.5, of course more than GPT 5.5, of course . So in this . So in this . So in this case it case it case it cost $620 cost $620 cost $620 compared to compared to compared to $497. So, from 500 $497. So, from 500 $497. So, from 500 to over 600. So it's to over 600. So it's to over 600. So it's not that... not that... Yeah, you should think about that Yeah, you should think about that Yeah, you should think about that . Do . Do . Do you want you want you want to use a to use a to use a fast and cheap fast and cheap fast and cheap model? But you have to model? But you have to model? But you have to compare whether compare whether compare whether it is really faster and cheaper, it is really faster and cheaper, it is really faster and cheaper, because all these tokens because all these tokens because all these tokens also take also take also take time. This is one moment. time. This is one moment. Returning to Returning to the presentation. And this is the the presentation. And this is the the presentation. And this is the surface of surface of surface of decision-making. This is not just a decision-making. This is not just a decision-making. This is not just a schedule. So, you schedule. So, you schedule. So, you upload your upload your upload your snapshot, you can add your snapshot, you can add your snapshot, you can add your own if you own if you own if you made one, and you run made one, and you run made one, and you run the code to see them. the code to see them. the code to see them. And then you can And then you can And then you can filter, as I filter, as I filter, as I showed, different showed, different showed, different metrics and models. You metrics and models. You metrics and models. You can change the can change the can change the order of columns and order of columns and order of columns and export your own export your own export your own data. And then you data. And then you data. And then you can show it, can show it, can show it, make a post on make a post on make a post on social media, or social media, or social media, or write an article, write an article, write an article, whatever. Um, you whatever. Um, you whatever. Um, you can do that. In can do that. In can do that. In this case, I was looking for this case, I was looking for which model was the which model was the which model was the best on this best on this best on this benchmark. So, my benchmark. So, my benchmark. So, my favorite right now is GBD 5.5 from favorite right now is GBD 5.5 from favorite right now is GBD 5.5 from Codex. It's even better Codex. It's even better Codex. It's even better than Fable, because if you than Fable, because if you than Fable, because if you ask what happened ask what happened ask what happened to Fable, that's also interesting to Fable, that's also interesting to Fable, that's also interesting to consider. If I to consider. If I to consider. If I add Fable here, it add Fable here, it add Fable here, it had 13 rejections. So, had 13 rejections. So, had 13 rejections. So, she refused. And she refused. And she refused. And if you look at if you look at if you look at the cost, it's the cost, it's the cost, it's off the charts. This was my off the charts. This was my off the charts. This was my most expensive benchmark most expensive benchmark . Don't tell anyone because

. Don't tell anyone because . Don't tell anyone because testing Fable testing Fable testing Fable cost 11,000 and it cost 11,000 and it cost 11,000 and it failed because there were failed because there were failed because there were too many rejections. too many rejections. too many rejections. Oh. And that, of course, is something Oh. And that, of course, is something Oh. And that, of course, is something you have to decide. you have to decide. you have to decide. When people say, "Oh, When people say, "Oh, When people say, "Oh, I don't trust this I don't trust this I don't trust this benchmark, this model is benchmark, this model is benchmark, this model is in first place," in first place," in first place," look what look what look what happened here. The benchmark should happened here. The benchmark should happened here. The benchmark should be transparent so that you be transparent so that you be transparent so that you can review can review can review the traces and see what the traces and see what the traces and see what happened, then everything happened, then everything happened, then everything becomes clear. becomes clear. OK. Going OK. Going back, as I said, the back, as I said, the back, as I said, the best model best model best model was Codex with GPT 5.5, but as was Codex with GPT 5.5, but as was Codex with GPT 5.5, but as you can see, it was very you can see, it was very you can see, it was very expensive. Let's expensive. Let's expensive. Let's put it in perspective, it put it in perspective, it put it in perspective, it cost almost a cost almost a cost almost a thousand dollars. thousand dollars. thousand dollars. Whereas Cursor with the same Whereas Cursor with the same Whereas Cursor with the same model cost model cost model cost only 282. It was only 282. It was only 282. It was significantly cheaper significantly cheaper significantly cheaper compared to the other one and compared to the other one and compared to the other one and it used slightly it used slightly it used slightly less tokens. But less tokens. But less tokens. But yes, you could say that yes, you could say that yes, you could say that this is the best this is the best this is the best combination, but combination, but combination, but perhaps another model is perhaps another model is perhaps another model is better in terms of price better in terms of price better in terms of price and performance. And and performance. And and performance. And Terminus also had the Terminus also had the Terminus also had the highest ceiling, but highest ceiling, but highest ceiling, but if the results are very if the results are very if the results are very close, it could be close, it could be close, it could be true, so you would true, so you would true, so you would have to do 10 have to do 10 have to do 10 or more runs or more runs or more runs to be sure, to be sure, to be sure, but at some point you have to but at some point you have to but at some point you have to make a choice and make a choice and make a choice and work with it. This is work with it. This is work with it. This is also what I was just also what I was just also what I was just showing: Gemini 3.5 flash showing: Gemini 3.5 flash showing: Gemini 3.5 flash compared to GPT 5.5— compared to GPT 5.5— lower score, higher lower score, higher lower score, higher score, so in this score, so in this score, so in this case it makes more sense case it makes more sense case it makes more sense to use a different to use a different to use a different model. And this was

model. And this was model. And this was also very interesting and also very interesting and also very interesting and quite quite quite surprising to me when I surprising to me when I surprising to me when I tested Opus 4.6: I tested Opus 4.6: I tested Opus 4.6: I set the level of set the level of set the level of thinking to maximum, thinking to maximum, thinking to maximum, and the result decreased, and the result decreased, and the result decreased, it really got it really got it really got worse. I did worse. I did worse. I did the same with Claude code, the same with Claude code, the same with Claude code, since I since I since I was also using it at the time was also using it at the time was also using it at the time , and saw the , and saw the , and saw the same effect. When I same effect. When I same effect. When I do this with GPT, if I do this with GPT, if I do this with GPT, if I increase the level of increase the level of increase the level of thinking from "off thinking from "off " to "very high" " to "very high" " to "very high" or from "medium" to or from "medium" to or from "medium" to "very high", "very high", "very high", the model becomes the model becomes the model becomes smarter, as smarter, as smarter, as expected. So why did expected. So why did expected. So why did Opus get worse? This Opus get worse? This Opus get worse? This was unexpected for was unexpected for was unexpected for me, and the difference, if me, and the difference, if me, and the difference, if you look at the logs, you look at the logs, you look at the logs, is that is that is that the model is thinking too much the model is thinking too much the model is thinking too much . She is being . She is being . She is being too clever. And that too clever. And that too clever. And that means I give her an means I give her an means I give her an hour of time. And if hour of time. And if hour of time. And if the model thinks for a minute the model thinks for a minute the model thinks for a minute or two and then or two and then or two and then decides to execute decides to execute decides to execute a command that a command that a command that isn't yet on the machine, then isn't yet on the machine, then isn't yet on the machine, then it thinks for another two or three it thinks for another two or three it thinks for another two or three minutes: "How do I minutes: "How do I minutes: "How do I install this?" Whereas an install this?" Whereas an install this?" Whereas an agent who doesn't even agent who doesn't even agent who doesn't even think just does the think just does the think just does the job. So, first job. So, first job. So, first impulse: I run impulse: I run impulse: I run this command, I get an this command, I get an error message. I run error message. I run this command, I get an this command, I get an error message. I install error message. I install it, get an error, it, get an error, it, get an error, fix it—and it gets fix it—and it gets fix it—and it gets more tries. So, more tries. So, more tries. So, she can do she can do she can do more. It's the difference more. It's the difference more. It's the difference between thinking for between thinking for between thinking for an hour and an hour and an hour and doing nothing, or just doing nothing, or just doing nothing, or just trying, trying, trying, trying, trying, trying, and trying through and trying through and trying through trial and error. And trial and error. And trial and error. And interestingly, the interestingly, the interestingly, the default GPT models did default GPT models did default GPT models did n't seem as n't seem as n't seem as smart as Opus. Opus is smart as Opus. Opus is smart as Opus. Opus is very functional very functional very functional right out of the box.

right out of the box. right out of the box. Even with his thinking turned off, Even with his thinking turned off, Even with his thinking turned off, he already had a he already had a he already had a great result, great result, great result, if you compare here. if you compare here. Even with the mode turned off, it's Even with the mode turned off, it's better than the better than the better than the average here in GPT 5.5. And average here in GPT 5.5. And average here in GPT 5.5. And even the baseline even the baseline even the baseline was higher. So he was higher. So he was higher. So he ran more ran more ran more tests during tests during tests during each run. Yes, each run. Yes, each run. Yes, this is something to this is something to this is something to pay attention to, and pay attention to, and pay attention to, and if the result is surprising, if the result is surprising, if the result is surprising, look at the logs look at the logs look at the logs to find out what's to find out what's to find out what's going on here. The model is going on here. The model is going on here. The model is overthinking. By the way overthinking. By the way , I saw the same thing with Fable, , I saw the same thing with Fable, , I saw the same thing with Fable, where the model was very where the model was very where the model was very intelligent and seemed to intelligent and seemed to know about it. So, know about it. So, know about it. So, she decided to she decided to she decided to move in a certain move in a certain move in a certain direction, and even direction, and even direction, and even if the test required a if the test required a if the test required a different way, she different way, she different way, she would say, "No, I know would say, "No, I know would say, "No, I know better, I'll do it better, I'll do it my way." And then my way." And then my way." And then the verifier the verifier the verifier ran the check at the end ran the check at the end ran the check at the end , and it , and it , and it failed. So, failed. So, failed. So, in some cases, in some cases, in some cases, the model was too the model was too the model was too smart to its own smart to its own smart to its own detriment. And this is also an detriment. And this is also an detriment. And this is also an important feature that important feature that important feature that I have to show you. I have to show you. I have to show you. The bar of the chart The bar of the chart is the statement, the log is the statement, the log is the proof. I keep is the proof. I keep is the proof. I keep talking about logs. So talking about logs. So talking about logs. So we have a column, this is what we have a column, this is what we have a column, this is what we see on Wolf we see on Wolf we see on Wolf Bench. You have nice Bench. You have nice Bench. You have nice columns, and the great thing is columns, and the great thing is columns, and the great thing is that you can click on that you can click on that you can click on any of them, any of them, any of them, just click and you'll just click and you'll just click and you'll go to our go to our go to our One page, where there's Reef, One page, where there's Reef, One page, where there's Reef, which allows you which allows you which allows you to explore all the to explore all the to explore all the traces. In this traces. In this traces. In this case, I just case, I just case, I just clicked on one clicked on one clicked on one column and saw all column and saw all column and saw all five runs that five runs that five runs that made up that score.

made up that score. made up that score. Now I can select Now I can select Now I can select any of the launches any of the launches , go here, and uh, , go here, and uh, , go here, and uh, I should remove this. I should remove this. I'll show it later. Uh-huh , disappear. So, we , disappear. So, we , disappear. So, we have 89 entries. These are 89 have 89 entries. These are 89 have 89 entries. These are 89 tasks that I tasks that I tasks that I ran, and you ran, and you ran, and you can see all the can see all the can see all the traces here. There are a traces here. There are a traces here. There are a lot of them, so lot of them, so lot of them, so loading takes loading takes loading takes a little time. You a little time. You a little time. You can then go into can then go into any of them, see the any of them, see the any of them, see the task name, task name, task name, reward, whether reward, whether reward, whether it was successful, and it was successful, and it was successful, and review all the review all the review all the steps. You see the steps. You see the steps. You see the initial prompt, the initial prompt, the user message: “You are an assistant. Your an assistant. Your an assistant. Your job is to do it job is to do it ." Next you see what ." Next you see what ." Next you see what LLM did as the first LLM did as the first LLM did as the first team, what analysis it team, what analysis it team, what analysis it conducted, what it wants to conducted, what it wants to conducted, what it wants to do, then the do, then the do, then the team it team it team it sends, and so on. sends, and so on. sends, and so on. You can You can You can examine everything that examine everything that examine everything that happened in detail and happened in detail and happened in detail and analyze it. analyze it. analyze it. Or use Or use Or use your agent to your agent to your agent to do it. We have a do it. We have a do it. We have a new feature—an agent new feature—an agent new feature—an agent built built built directly into the directly into the directly into the platform. So you have an platform. So you have an platform. So you have an agent inside Reef agent inside Reef agent inside Reef that you can simply that you can simply that you can simply give a command to. My give a command to. My give a command to. My colleague, Tim Sweeney, colleague, Tim Sweeney, colleague, Tim Sweeney, will be giving a talk on will be giving a talk on will be giving a talk on this on July 1st. You this on July 1st. You this on July 1st. You can can can find out about it there. I often find out about it there. I often find out about it there. I often use agents use agents use agents because reading all because reading all because reading all the traces takes the traces takes the traces takes a lot of time. And if an a lot of time. And if an a lot of time. And if an agent gives you a agent gives you a agent gives you a link to Reef, and you link to Reef, and you link to Reef, and you see that his words see that his words see that his words are true, that's very are true, that's very are true, that's very good. So, good. So, good. So, to summarize, there are to summarize, there are to summarize, there are many variables that are many variables that are many variables that are important for important for important for benchmarks. This is the benchmarks. This is the benchmarks. This is the benchmark version, 2.0, 2.1, benchmark version, 2.0, 2.1, benchmark version, 2.0, 2.1, agent version, agent version, agent version, model service and route, model service and route, model service and route, even even even inference endpoint updates inference endpoint updates inference endpoint updates can change can change can change the results and

the results and the results and settings. settings. settings. Sandbox resources: As Sandbox resources: As Sandbox resources: As I said, if an agent I said, if an agent I said, if an agent runs out of resources runs out of resources , it fails , it fails , it fails and you get a and you get a and you get a low score for the low score for the low score for the model, even though it was model, even though it was model, even though it was n't the model's fault. n't the model's fault. n't the model's fault. Sampling, Sampling, Sampling, temperature, all the things temperature, all the things temperature, all the things we discussed. we discussed. Default thinking settings Default thinking settings , such as , such as , such as duration limits duration limits , if they are too , if they are too , if they are too long, and a smaller and long, and a smaller and long, and a smaller and faster faster faster agent response could agent response could agent response could work better. work better. work better. Timeout policy, Timeout policy, Timeout policy, retry policy retry policy retry policy . If you don't . If you don't retry and an infrastructure error occurs, you infrastructure error occurs, you will receive a low will receive a low will receive a low score. Therefore, after score. Therefore, after score. Therefore, after each run, I each run, I each run, I review all review all review all errors and check errors and check errors and check if there were any problems with the if there were any problems with the if there were any problems with the conclusions. You can conclusions. You can conclusions. You can tell if there are tell if there are tell if there are repeated characters repeated characters repeated characters or if the or if the or if the request pattern tokens request pattern tokens request pattern tokens are getting into are getting into are getting into the response—then you the response—then you the response—then you know it's a know it's a know it's a tokenizer or tokenizer or tokenizer or inference server error. And, inference server error. And, inference server error. And, especially with new especially with new especially with new models, this is a serious models, this is a serious models, this is a serious problem. If you problem. If you problem. If you test models test models test models right after they are right after they are right after they are released, they are released, they are released, they are so new that the so new that the so new that the inference tool inference tool inference tool may not yet fully support them may not yet fully support them may not yet fully support them correctly correctly correctly . So I . So I . So I did an evaluation did an evaluation did an evaluation where the new model was where the new model was where the new model was n't very good. A n't very good. A n't very good. A week later I week later I week later I did the same thing, everything did the same thing, everything did the same thing, everything was identical, but was identical, but was identical, but the result was the result was the result was better because the better because the better because the inference engine had been updated. This is inference engine had been updated. This is inference engine had been updated. This is also part of the also part of the also part of the process. So, process. So, process. So, use use use the message, the message, the message, bring your own.

bring your own. bring your own. You have the building You have the building You have the building blocks, watch blocks, watch blocks, watch the presentation, the presentation, the presentation, think about what you think about what you think about what you want to do, what want to do, what want to do, what assessment assessment assessment to use, what to use, what to use, what agent to choose, what agent to choose, what agent to choose, what model to test. model to test. model to test. Now you have an Now you have an Now you have an understanding of what to look for understanding of what to look for understanding of what to look for to to to create more robust create more robust create more robust leaderboards, leaderboards, leaderboards, charts, and scores charts, and scores charts, and scores than if you just than if you just than if you just launched something and launched something and launched something and posted a score. posted a score. So, run So, run your own reproducible your own reproducible benchmark configurations. Share benchmark configurations. Share the results. If the results. If the results. If you want to work with Wolf you want to work with Wolf you want to work with Wolf Bench, contact Bench, contact Bench, contact me. If you want me. If you want me. If you want to run some to run some to run some models, that would be great. models, that would be great. models, that would be great. We could We could We could collaborate and collaborate and collaborate and publish publish publish the results so I do the results so I do the results so I do n't have to n't have to n't have to run everything myself. run everything myself. run everything myself. And, if you want to And, if you want to And, if you want to branch out branch out , that's great too. , that's great too. , that's great too. Use a different Use a different Use a different framework, framework, framework, whatever. And if you are whatever. And if you are whatever. And if you are creating models or creating models or creating models or agents, feel free to agents, feel free to agents, feel free to contact me contact me contact me so I can so I can so I can test them. I am already test them. I am already test them. I am already working with some working with some working with some agent creators and will agent creators and will agent creators and will publish the publish the publish the results soon. Yes, this is results soon. Yes, this is results soon. Yes, this is effectively the end of effectively the end of effectively the end of the presentation. the presentation. the presentation. I hope you I hope you I hope you got something got something got something useful out of this. And if you useful out of this. And if you useful out of this. And if you have any questions, now is the have any questions, now is the have any questions, now is the time. Otherwise, time. Otherwise, time. Otherwise, you can find you can find you can find me at the booth, and me at the booth, and me at the booth, and a big thank you to a big thank you to a big thank you to Alex for his help.

Alex for his help. The podcast is The podcast is every Thursday morning, I'm every Thursday morning, I'm every Thursday morning, I'm from a different time from a different time from a different time zone. At 8:30 a.m. zone. At 8:30 a.m. zone. At 8:30 a.m. Pacific Time Pacific Time . Every week, because it's so . Every week, because it's so . Every week, because it's so hard to keep up with what's hard to keep up with what's hard to keep up with what's going on, and it's going on, and it's going on, and it's hard for us ourselves, so hard for us ourselves, so hard for us ourselves, so we have to do it we have to do it we have to do it to let to let to let you know. Okay, thank you.