Just-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel, Southbridge AI

I really I really like like like Gannes' article from DuckDB, where he Gannes' article from DuckDB, where he Gannes' article from DuckDB, where he asks whether asks whether asks whether we've wasted we've wasted we've wasted decades decades decades chasing chasing chasing distributed distributed distributed architectures architectures architectures based on the based on the based on the assumption that assumption that assumption that computing and computing and computing and storing data are storing data are storing data are slow and expensive. slow and expensive. slow and expensive. However, the capabilities of However, the capabilities of However, the capabilities of each era each era each era determine determine determine the architecture and the architecture and the architecture and tools, and the tools, and the tools, and the tools constantly tools constantly tools constantly lag behind. First lag behind. First lag behind. First we made we made we made computation cheaper computation cheaper computation cheaper than storage, then we than storage, then we than storage, then we gave it to everyone, and then we gave it to everyone, and then we gave it to everyone, and then we accelerated it. And each time, accelerated it. And each time, accelerated it. And each time, architectures undergo architectures undergo architectures undergo these massive changes. these massive changes. these massive changes. Our entire modern Our entire modern Our entire modern data infrastructure data infrastructure data infrastructure is built on the idea is built on the idea is built on the idea that transformation is that transformation is that transformation is expensive. What if that's expensive. What if that's expensive. What if that's not the case anymore? I believe we have not the case anymore? I believe we have not the case anymore? I believe we have overcome the last overcome the last overcome the last hurdle in the hurdle in the hurdle in the transformation— transformation— general intelligence general intelligence general intelligence or or or judgment—and made it judgment—and made it judgment—and made it available on demand available on demand available on demand along the entire along the entire along the entire Pareto frontier. This puts Pareto frontier. This puts Pareto frontier. This puts us on the verge of a us on the verge of a us on the verge of a paradigm shift. So, in this paradigm shift. So, in this paradigm shift. So, in this talk, we talk, we talk, we will look at why will look at why will look at why single ontologies single ontologies single ontologies always always always fail, the three roles of fail, the three roles of fail, the three roles of models in pipelines, the models in pipelines, the models in pipelines, the intersection of a priori intersection of a priori intersection of a priori knowledge of models for knowledge of models for knowledge of models for better results, better results, better results, and “Pareto serving.” How and “Pareto serving.” How and “Pareto serving.” How modern models modern models modern models can tune can tune can tune workloads workloads workloads for cost, for cost, for cost, speed, and speed, and speed, and accuracy simultaneously.

accuracy simultaneously. accuracy simultaneously. Here's a quick example. Here's a quick example. Here's a quick example. When Jeff launched, When Jeff launched, When Jeff launched, we already had an entity we already had an entity we already had an entity resolution pipeline running resolution pipeline running resolution pipeline running , , , processing processing Ohio campaign finance records to determine whether Ohio campaign finance records to determine whether they belonged to a single they belonged to a single they belonged to a single person, company, or person, company, or person, company, or committee. committee. committee. The projected The projected The projected cost was cost was cost was about $10,000. about $10,000. about $10,000. Our systems took this Our systems took this Our systems took this new model, new model, new model, identified where it identified where it identified where it fit, fit, fit, tested over tested over tested over a dozen approaches, and a dozen approaches, and a dozen approaches, and integrated it. integrated it. integrated it. 10 days after that 10 days after that 10 days after that first call, we first call, we first call, we had definitive, had definitive, had definitive, reliable reliable reliable information for information for information for every every every donor case in Ohio for donor case in Ohio for donor case in Ohio for less than less than less than $800. This is the power of $800. This is the power of Pareto serving and Pareto serving and Pareto serving and just- just- just- in-time transformations. So, in-time transformations. So, in-time transformations. So, let's get started. let's get started. Ontologies are the Ontologies are the corporate version of corporate version of corporate version of mind maps. This is the mind maps. This is the mind maps. This is the trap that traps trap that traps trap that traps us all, including us all, including us all, including me. I talked about a me. I talked about a me. I talked about a single universal single universal single universal scheme when I scheme when I scheme when I founded South Bridge 2 founded South Bridge 2 founded South Bridge 2 years ago. The popular years ago. The popular years ago. The popular belief—and it's not belief—and it's not belief—and it's not wrong, it's just a wrong, it's just a wrong, it's just a small small small part of the picture— part of the picture— is that is that is that migrations and migrations and migrations and ontologies ontologies ontologies fail due to fail due to fail due to lack of lack of lack of support. And that they are support. And that they are support. And that they are abandoned halfway, abandoned halfway, abandoned halfway, that organizations and that organizations and that organizations and people never people never people never switch to them.

switch to them. switch to them. All of this is true. 100%. All of this is true. 100%. All of this is true. 100%. But lack of But lack of But lack of support is not the support is not the support is not the main reason. main reason. main reason. For example, if you For example, if you For example, if you could integrate and could integrate and could integrate and transfer all your transfer all your transfer all your information into a single information into a single information into a single superschema in the blink of an eye superschema in the blink of an eye , you would still have to , you would still have to , you would still have to accept accept accept that the superschema that the superschema that the superschema does not exist. does not exist. does not exist. There are actually four schemes. There are actually four schemes. There are actually four schemes. The first belongs to The first belongs to The first belongs to the source from which the source from which the source from which the information comes the information comes . Let me take temperature . Let me take temperature . Let me take temperature as an example. as an example. as an example. Suppose I ask Suppose I ask Suppose I ask you to record you to record you to record the temperature. The source the temperature. The source the temperature. The source schema is all schema is all schema is all you have access to. Your you have access to. Your you have access to. Your phone shows phone shows phone shows the temperature in your the temperature in your the temperature in your area, a sensor on your area, a sensor on your area, a sensor on your desk—less accurate desk—less accurate desk—less accurate but more local—or, but more local—or, but more local—or, say, your watch say, your watch . But even your . But even your . But even your patience in fulfilling patience in fulfilling patience in fulfilling my request is my request is my request is part of this part of this part of this source scheme. The second is the source scheme. The second is the source scheme. The second is the storage scheme. You storage scheme. You storage scheme. You can only record a can only record a can only record a certain amount of certain amount of certain amount of data. You cannot data. You cannot data. You cannot write the firmware of write the firmware of write the firmware of the device you the device you the device you are using. You are using. You are using. You can't even can't even can't even record the time to record the time to record the time to the second because you won't the second because you won't the second because you won't have time to have time to have time to record it all in one record it all in one record it all in one second. This is a pattern that second. This is a pattern that storage imposes on you. The third storage imposes on you. The third belongs to the decision.

belongs to the decision. belongs to the decision. Why do I need you to Why do I need you to Why do I need you to record my record my record my temperature? Maybe temperature? Maybe temperature? Maybe you're figuring out what's you're figuring out what's you're figuring out what's comfortable for you, and then comfortable for you, and then comfortable for you, and then I need to get a I need to get a temperature reading throughout the temperature reading throughout the day, and I need to day, and I need to day, and I need to make sure it's make sure it's make sure it's you who's taking the reading. you who's taking the reading. you who's taking the reading. Maybe I'm creating Maybe I'm creating weather migration maps, and then I don't weather migration maps, and then I don't care who's measuring, care who's measuring, care who's measuring, but what matters is your but what matters is your but what matters is your exact exact exact location, location, location, time, device, and whether time, device, and whether time, device, and whether it's changed. Or it's changed. Or it's changed. Or maybe I'm deciding whether maybe I'm deciding whether maybe I'm deciding whether to install to install to install sensitive equipment and sensitive equipment and sensitive equipment and where, and you yourself where, and you yourself where, and you yourself become this variable become this variable become this variable that I need to that I need to that I need to eliminate. However, there is a eliminate. However, there is a eliminate. However, there is a fourth, which we fourth, which we fourth, which we will discuss when we will discuss when we will discuss when we touch on a priori touch on a priori touch on a priori knowledge. This is a recent knowledge. This is a recent knowledge. This is a recent addition. The real addition. The real addition. The real problem is that you will problem is that you will problem is that you will never know never know never know all four patterns all four patterns all four patterns at the same time, except at at the same time, except at at the same time, except at the moment of the moment of the moment of decision. In almost decision. In almost decision. In almost every business every business every business and every home, and every home, and every home, most data most data most data is recorded without is recorded without is recorded without understanding what understanding what understanding what it will be it will be it will be used for. used for. used for. Let's take the customer—the Let's take the customer—the first block in first block in any ontology. For the any ontology. For the any ontology. For the billing department, billing department, billing department, the customer is the one who the customer is the one who the customer is the one who pays. For pays. For pays. For support, this is the person support, this is the person support, this is the person who calls. For the who calls. For the who calls. For the compliance department, compliance department, this is the one for whom we are this is the one for whom we are responsible. And responsible. And you will only find out about it you will only find out about it you will only find out about it when when when the decision becomes known. Should I get a the decision becomes known. Should I get a refund for this refund for this order? Can order? Can order? Can we give this we give this we give this account more account more account more credit? Suddenly, what credit? Suddenly, what credit? Suddenly, what you need to know, you need to know, you need to know, how and to what extent, becomes how and to what extent, becomes how and to what extent, becomes crystal clear.

crystal clear. That's why I was wrong. There That's why I was wrong. There is no is no is no universal scheme. There universal scheme. There universal scheme. There is no concept of is no concept of is no concept of a customer a customer a customer separate from the separate from the separate from the question you question you question you want to want to want to answer. And answer. And answer. And combining these schemes combining these schemes is a process of is a process of is a process of transformation, transformation, transformation, turning turning turning information into something truly information into something truly information into something truly useful. Taking useful. Taking useful. Taking the context in which the context in which the context in which it was recorded and it was recorded and it was recorded and combining it with the combining it with the combining it with the context of the decision. And context of the decision. And context of the decision. And this requires this requires this requires insight. insight. insight. Doing this Doing this Doing this unification when the unification when the unification when the decision space decision space decision space is known, rather than is known, rather than is known, rather than a priori, is what a priori, is what a priori, is what I mean by I mean by I mean by operational operational operational ontology. So, we'll ontology. So, we'll ontology. So, we'll return to return to return to sanity with a sanity with a sanity with a story from my story from my story from my past job, where a past job, where a past job, where a significant part of my significant part of my significant part of my responsibilities responsibilities responsibilities was was was ensuring that ensuring that ensuring that certain numbers certain numbers certain numbers never never never appeared at all. Before appeared at all. Before appeared at all. Before that, a little about me and that, a little about me and that, a little about me and us. We are best us. We are best us. We are best described as obsessed with described as obsessed with described as obsessed with our work. our work. our work. Obsessed people who Obsessed people who Obsessed people who inject inject inject intelligence into intelligence into intelligence into data processing pipelines. data processing pipelines. data processing pipelines. Pipelines that are able Pipelines that are able Pipelines that are able to scale, to scale, to scale, run an run an run an infinite infinite infinite number of times to number of times to number of times to really take really take really take the load off us, and the load off us, and the load off us, and evolve in terms of evolve in terms of evolve in terms of cost, cost, cost, performance, and performance, and performance, and speed on their own.

speed on their own. speed on their own. And most of our And most of our And most of our data work data work data work falls into three falls into three falls into three modes. The first is " modes. The first is " write once, write once, write once, execute once." execute once." execute once." If you're in If you're in If you're in development, development, development, think of this as think of this as think of this as immediately callable immediately callable immediately callable functions of gigantic functions of gigantic functions of gigantic scale. They scale. They scale. They are created for a are created for a are created for a single single single use, use, use, such as to such as to such as to aggregate a aggregate a aggregate a dataset, mine dataset, mine dataset, mine information, or information, or information, or answer one answer one answer one specific question, specific question, specific question, and then they are and then they are and then they are deleted. The second deleted. The second is “write many, is “write many, is “write many, do once,” do once,” do once,” where many small and where many small and where many small and large tasks large tasks large tasks are created for are created for are created for one big one big one big common goal. common goal. common goal. The closest thing The closest thing The closest thing you might know to this you might know to this is large-scale is large-scale is large-scale data migrations. data migrations. data migrations. Years of archives Years of archives Years of archives are being transferred from are being transferred from are being transferred from systems that no systems that no systems that no one remembers the one remembers the one remembers the full information about anymore. full information about anymore. full information about anymore. Many tasks Many tasks Many tasks are created for are created for are created for one rule and one rule and one rule and checked separately checked separately checked separately in the source, checked in the source, checked in the source, checked against others, because all against others, because all against others, because all this happens only this happens only this happens only once. The third one—and once. The third one—and once. The third one—and this is our bread and butter, this is our bread and butter, this is our bread and butter, the most common—is “ the most common—is “ write a lot, write a lot, write a lot, do a lot.” These are do a lot.” These are do a lot.” These are iterative pipelines iterative pipelines that are created and that are created and that are created and maintained maintained maintained over over over time as time as time as new information is added. And at new information is added. And at new information is added. And at South Ridge, we've moved South Ridge, we've moved South Ridge, we've moved so much work from the so much work from the so much work from the main stream into these main stream into these main stream into these repeatable systems: repeatable systems: repeatable systems: from developing from developing from developing benchmarks, benchmarks, benchmarks, tracking, and tracking, and tracking, and encoding, to encoding, to encoding, to creating subtitles creating subtitles creating subtitles for the movies we for the movies we for the movies we want to watch.

want to watch. want to watch. Almost everything we Almost everything we Almost everything we do in today's do in today's do in today's world is world is world is data transformation data transformation data transformation or is limited by or is limited by or is limited by how difficult it is to how difficult it is to how difficult it is to transform data. transform data. transform data. And in all these cases, And in all these cases, And in all these cases, our job is simple. our job is simple. our job is simple. Make sure people Make sure people Make sure people can rely can rely can rely on the results. In just on the results. In just on the results. In just 2 years, we went 2 years, we went 2 years, we went from finding from finding from finding ways to make ways to make ways to make early 3.5 early 3.5 early 3.5 turbo models robust in turbo models robust in turbo models robust in areas like areas like areas like healthcare, aging, healthcare, aging, healthcare, aging, and mining and mining and mining research, to research, to research, to building Hank Wave, building Hank Wave, building Hank Wave, our our open source open source data tool platform, and data tool platform, and data tool platform, and then using then using then using it to it to it to collaborate on models at collaborate on models at collaborate on models at terabyte scale. terabyte scale. terabyte scale. Today we have Today we have Today we have self-optimizing self-optimizing self-optimizing systems: you give them a systems: you give them a systems: you give them a problem, a way problem, a way to make mistakes safely , a space to work, , a space to work, , a space to work, and they find and they find and they find the contours of a solution. the contours of a solution. the contours of a solution. They choose the They choose the They choose the right models, right models, right models, conduct pilot conduct pilot conduct pilot runs, runs, runs, scale up, and scale up, and scale up, and use Pareto use Pareto optimization optimization optimization to find the best to find the best to find the best possible option. possible option. So, a story from my So, a story from my past work: past work: past work: the question we heard the question we heard the question we heard most often— most often— most often— can we make this can we make this can we make this decision decision decision data-driven? And data-driven? And data-driven? And sometimes that was the sometimes that was the sometimes that was the right question right question right question when we had when we had thousands of vessels on the platform.

thousands of vessels on the platform. The problem is that we were The problem is that we were The problem is that we were installing it even when there installing it even when there were only 50 users. And if you were only 50 users. And if you know the statistics, you know the statistics, you know the statistics, you can can can explain well why A/B explain well why A/B testing for 50 testing for 50 testing for 50 users is a bad users is a bad users is a bad idea. Not because you won't idea. Not because you won't idea. Not because you won't get the result, get the result, get the result, but because you but because you but because you will. So the only will. So the only will. So the only real option real option real option was to not was to not was to not get any numbers at all, because get any numbers at all, because get any numbers at all, because as soon as as soon as as soon as a number appears, it becomes a number appears, it becomes a number appears, it becomes an argument in an argument in an argument in favor of the result. favor of the result. favor of the result. Today, however, Today, however, Today, however, the situation is worse. the situation is worse. the situation is worse. Interactive agents Interactive agents Interactive agents will always give you will always give you will always give you an answer. They an answer. They an answer. They will give it quickly. You will give it quickly. You will give it quickly. You ask for ask for ask for transformation, transformation, anything, and you get a anything, and you get a anything, and you get a ready-made result—" ready-made result—" yes, sir." This means yes, sir." This means yes, sir." This means that errors that errors that errors accumulate very accumulate very accumulate very quickly. A moment in a chat quickly. A moment in a chat quickly. A moment in a chat can become a can become a can become a nightmare just nightmare just nightmare just waiting to be waiting to be waiting to be discovered. The main discovered. The main discovered. The main reason for this is reason for this is reason for this is that that that intelligence in the context of intelligence in the context of intelligence in the context of data work data work data work today combines three today combines three today combines three separate roles that separate roles that separate roles that coding agents coding agents coding agents mix into a single mix into a single mix into a single context window. Let's context window. Let's context window. Let's separate them, because it will be separate them, because it will be separate them, because it will be very helpful when very helpful when very helpful when we move on to we move on to decision-making models, because they decision-making models, because they fit exactly into fit exactly into fit exactly into one of these three one of these three one of these three categories. The first is the categories. The first is the categories. The first is the creation and creation and creation and design design design of the pipeline: breaking down of the pipeline: breaking down of the pipeline: breaking down the problem, writing the problem, writing the problem, writing initial initial initial specifications, specifications, specifications, approving costs.

approving costs. approving costs. The second—often The second—often The second—often confused with the first and confused with the first and confused with the first and assigned to the same assigned to the same assigned to the same models—is the work models—is the work models—is the work inside the pipeline: inside the pipeline: inside the pipeline: performing tasks, performing tasks, performing tasks, reading PDFs, writing to reading PDFs, writing to reading PDFs, writing to databases, and checking databases, and checking databases, and checking results. Before results. Before results. Before we move on to we move on to we move on to the third, I want the third, I want the third, I want to clarify that these two to clarify that these two to clarify that these two roles are directed in roles are directed in roles are directed in completely different directions. completely different directions. completely different directions. The first requires The first requires The first requires slower models slower models slower models with heavier with heavier with heavier tokens: models tokens: models tokens: models that can that can that can think more, approach think more, approach think more, approach the problem from different the problem from different the problem from different angles, study angles, study angles, study the documentation, the documentation, the documentation, understand how understand how understand how pipelines are built, and pipelines are built, and pipelines are built, and predict the predict the predict the outcome before it even outcome before it even outcome before it even starts, like Opus or starts, like Opus or starts, like Opus or similar. The models similar. The models similar. The models needed inside the needed inside the needed inside the conveyors are completely conveyors are completely conveyors are completely different. You need different. You need different. You need cheaper, faster, cheaper, faster, cheaper, faster, specialized specialized specialized models that models that models that follow instructions well— follow instructions well— repeatable repeatable repeatable workers with workers with workers with minimal minimal minimal defects. defects. defects. You see the same thing in You see the same thing in You see the same thing in nature. A colony of nature. A colony of nature. A colony of leafcutter ants, leafcutter ants, leafcutter ants, for example, operates with for example, operates with for example, operates with four different four different four different sizes of workers sizes of workers , right? Small, medium, , right? Small, medium, , right? Small, medium, large, extra large. large, extra large. large, extra large. When you When you When you observe observe observe any system any system any system you design, you design, you design, analyze, or analyze, or analyze, or hear about, pay hear about, pay hear about, pay attention to what attention to what attention to what fits where in this fits where in this fits where in this multi-layered multi-layered multi-layered pie. This is of great pie. This is of great pie. This is of great importance. Because importance. Because importance. Because we had this we had this we had this separation, separation, separation, implementing new implementing new decision-making models was very decision-making models was very easy because they are easy because they are employees. They are employees. They are employees. They are designed for designed for designed for quick access to quick access to quick access to information and information and information and are deployed at the are deployed at the are deployed at the bottom of bottom of bottom of large large large workflows. So, the third workflows. So, the third workflows. So, the third category. These are what category. These are what category. These are what we call we call we call summit conquerors summit conquerors summit conquerors or servers. Models or servers. Models that can take an that can take an that can take an initial version and initial version and initial version and self-optimize.

self-optimize. self-optimize. With the necessary With the necessary With the necessary knowledge, instructions, and an knowledge, instructions, and an knowledge, instructions, and an incredibly stable incredibly stable incredibly stable base, they can base, they can base, they can try new things, try new things, try new things, change budgets change budgets change budgets to limit costs, to limit costs, to limit costs, quickly correct quickly correct quickly correct mistakes, and take mistakes, and take mistakes, and take notes, working their way notes, working their way notes, working their way not just to the not just to the not just to the right solution, right solution, right solution, but to the optimal one. We but to the optimal one. We but to the optimal one. We call it " call it " burrito surfing" burrito surfing" burrito surfing" and we'll talk about it at and we'll talk about it at and we'll talk about it at the end. But the end. But the end. But first, let's talk first, let's talk first, let's talk about our experience with Jev. about our experience with Jev. about our experience with Jev. Jev showed up on Jev showed up on Jev showed up on Tuesday when we Tuesday when we Tuesday when we were deep into the were deep into the entity recognition pipeline. You entity recognition pipeline. You worked on worked on worked on donors in donors in campaign finance in campaign finance in Ohio. At this point, Ohio. At this point, Ohio. At this point, many people many people many people have already seen this graph. We've have already seen this graph. We've have already seen this graph. We've talked about talked about talked about the results before and the results before and the results before and adapted some of the adapted some of the adapted some of the current work into a current work into a current work into a dataset so you dataset so you dataset so you can download can download can download it for yourself, complete it for yourself, complete it for yourself, complete with hints and with hints and with hints and everything. What we didn't everything. What we didn't everything. What we didn't talk about was talk about was talk about was everything behind that everything behind that everything behind that one arrow, and what one arrow, and what one arrow, and what happened when we happened when we happened when we launched the same launched the same launched the same idea for the entire state idea for the entire state . Here are all the architectures . Here are all the architectures . Here are all the architectures we tested. we tested. we tested. We started with three We started with three We started with three designs on a designs on a designs on a small small small development kit. A cheap development kit. A cheap development kit. A cheap reasoning model on its reasoning model on its reasoning model on its own, Jev with a own, Jev with a own, Jev with a reasoning model reasoning model reasoning model for everything for everything for everything she wasn't sure about she wasn't sure about , and a reasoning model , and a reasoning model , and a reasoning model that suggested that suggested that suggested answers and Jev answers and Jev answers and Jev checked them. On checked them. On checked them. On about 200 families, the about 200 families, the about 200 families, the cheap cheap cheap reasoning model on its reasoning model on its reasoning model on its own performed as well as own performed as well as own performed as well as Fable, Fable, Fable, at about 2% of at about 2% of the cost. Jev initially the cost. Jev initially the cost. Jev initially answered the answered the answered the basic questions correctly, basic questions correctly, basic questions correctly, but missed nine but missed nine household connections household connections because he confidently answered because he confidently answered because he confidently answered "no" to each of them.

"no" to each of them. "no" to each of them. So we rewrote So we rewrote So we rewrote the question. Without the question. Without the question. Without touching on the models touching on the models touching on the models or thresholds, the or thresholds, the household question household question now included a now included a now included a definition of what definition of what definition of what was considered sufficient was considered sufficient was considered sufficient evidence, and Jev changed evidence, and Jev changed evidence, and Jev changed the answers from “no” to “ the answers from “no” to “ yes” in all 13 yes” in all 13 yes” in all 13 test questions. And test questions. And test questions. And as we scaled as we scaled as we scaled statewide, we statewide, we statewide, we conducted over conducted over conducted over a dozen rounds of a dozen rounds of a dozen rounds of such qualifications. such qualifications. such qualifications. Each round Each round Each round refined the definitions refined the definitions : what is a committee, what does : what is a committee, what does : what is a committee, what does “ “ “ reporter” mean, when two reporter” mean, when two reporter” mean, when two people live at the people live at the people live at the same address—what does it same address—what does it same address—what does it mean? Each of mean? Each of mean? Each of Ohio's 203,317 cases now has a Ohio's 203,317 cases now has a Ohio's 203,317 cases now has a label indicating label indicating label indicating what what what the answer is based on and the answer is based on and the answer is based on and how thoroughly how thoroughly how thoroughly it has been reviewed. it has been reviewed. it has been reviewed. This is the hierarchy we This is the hierarchy we This is the hierarchy we discovered for this discovered for this discovered for this particular problem. particular problem. particular problem. Here's everything we Here's everything we Here's everything we learned about learned about learned about implementing implementing implementing decision-making models into decision-making models into decision-making models into pipelines. I'll leave this pipelines. I'll leave this pipelines. I'll leave this on the screen so you on the screen so you on the screen so you can take a can take a can take a screenshot. But where did the screenshot. But where did the screenshot. But where did the money go? All money go? All money go? All work within the state work within the state work within the state cost $795 cost $795 cost $795 with a budget of $800 with a budget of $800 set set in advance. in advance. in advance. The inspection cost The inspection cost The inspection cost about four about four about four times more than the times more than the times more than the execution. $545 execution. $545 execution. $545 for second opinions, for second opinions, for second opinions, dispute resolution, dispute resolution, dispute resolution, and audits versus and audits versus and audits versus $128 for first $128 for first $128 for first responses. Before Jev came along, responses. Before Jev came along, responses. Before Jev came along, we expected this we expected this we expected this job to cost job to cost job to cost around $10,000.

around $10,000. around $10,000. Now that the pipeline is Now that the pipeline is Now that the pipeline is built and built and built and tested, we tested, we tested, we expect expect expect audit costs for the audit costs for the audit costs for the next phase next phase next phase to be close to to be close to to be close to zero. The most amazing thing is zero. The most amazing thing is zero. The most amazing thing is that our agent 4A, that our agent 4A, that our agent 4A, who worked on Astra who worked on Astra who worked on Astra and later on Opus, and later on Opus, and later on Opus, found most of found most of found most of these approaches these approaches these approaches gradually and gradually and gradually and independently. By independently. By independently. By creating a system, creating a system, creating a system, launching it, launching it, launching it, debugging it, and debugging it, and debugging it, and testing something new. testing something new. testing something new. He worked for 10 days, He worked for 10 days, He worked for 10 days, testing over a testing over a testing over a dozen architectures. dozen architectures. dozen architectures. His tasks His tasks His tasks were run over were run over were run over 4,000 times with 90,000 4,000 times with 90,000 4,000 times with 90,000 calls to Jeff, he calls to Jeff, he calls to Jeff, he tested almost all tested almost all tested almost all models on the Pareto frontier models on the Pareto frontier , and the models within , and the models within , and the models within those runs those runs those runs processed about 6 processed about 6 processed about 6 billion tokens. And billion tokens. And billion tokens. And when all this was when all this was when all this was done, we wrote the done, we wrote the done, we wrote the acquired knowledge acquired knowledge acquired knowledge back into our back into our back into our agent agent agent component library. This component library. This component library. This work built on work built on work built on existing components, and existing components, and existing components, and we are adding new we are adding new we are adding new proven ones, such as a proven ones, such as a proven ones, such as a tool for tool for tool for validating emails. To validating emails. To validating emails. To me, this is the value of me, this is the value of me, this is the value of taking work outside the taking work outside the mainstream for both humans mainstream for both humans and agents, and agents, and agents, because each of these because each of these because each of these iterations was iterations was iterations was version version version controlled. They controlled. They controlled. They were easy to undo, were easy to undo, were easy to undo, understand, understand, understand, debug, and debug, and debug, and analyze. So, analyze. So, analyze. So, can models can models can models do this natively do this natively do this natively now? Not quite. They are now? Not quite. They are now? Not quite. They are missing four missing four missing four things, which we will things, which we will things, which we will consider. And there is something consider. And there is something consider. And there is something that doesn't disappear, even as that doesn't disappear, even as that doesn't disappear, even as models become models become models become bigger and bigger and bigger and more powerful—these are more powerful—these are more powerful—these are a priori beliefs a priori beliefs . Like humans, models . Like humans, models . Like humans, models bring to each bring to each bring to each task their own task their own task their own a priori ideas a priori ideas a priori ideas about what the about what the about what the world should be like. This is the fourth world should be like. This is the fourth world should be like. This is the fourth scheme I mentioned scheme I mentioned scheme I mentioned earlier, the one that the model earlier, the one that the model earlier, the one that the model brings with it.

brings with it. Larger models Larger models strangely sometimes strangely sometimes strangely sometimes do worse with this do worse with this do worse with this . They . They . They hold on to their hold on to their hold on to their beliefs and beliefs and beliefs and defend them even more defend them even more defend them even more strongly. Here are strongly. Here are strongly. Here are a few cases we have a few cases we have a few cases we have encountered. encountered. encountered. One conveyor was looking for "in One conveyor was looking for "in One conveyor was looking for "in kind", written with an kind", written with an kind", written with an underline. In underline. In underline. In the file, it was the file, it was the file, it was written with a written with a written with a capital "I" and a capital "I" and a capital "I" and a hyphen. Therefore, the model hyphen. Therefore, the model hyphen. Therefore, the model counted 15,658 counted 15,658 in-kind contributions as in-kind contributions as regular money without regular money without double-checking it. double-checking it. Loan repayments also Loan repayments also Loan repayments also disappeared in the same disappeared in the same disappeared in the same way. Another pipeline way. Another pipeline way. Another pipeline tested four tested four tested four date formats when a fifth was used in the date formats when a fifth was used in the date formats when a fifth was used in the file file . 99.4% of . 99.4% of transaction dates were transaction dates were transaction dates were empty, but empty, but empty, but the number of lines and the number of lines and the number of lines and amounts matched, so the amounts matched, so the amounts matched, so the agent continued agent continued agent continued working. Several working. Several working. Several runs noticed that the runs noticed that the runs noticed that the credit amounts were credit amounts were credit amounts were negative, decided negative, decided negative, decided that the sign was unnecessary, and that the sign was unnecessary, and that the sign was unnecessary, and kept the absolute kept the absolute kept the absolute value. And the net value. And the net value. And the net balance of balance of balance of $784,000 turned $784,000 turned $784,000 turned into $5.3 million in into $5.3 million in into $5.3 million in debt, which debt, which debt, which was confirmed by was confirmed by was confirmed by several models. And several models. And several models. And this is quickly becoming the this is quickly becoming the this is quickly becoming the largest class of largest class of largest class of mistakes I see mistakes I see mistakes I see companies making when companies making when companies making when working with data, working with data, working with data, as models take over as models take over . Sometimes this can be . Sometimes this can be solved by not solved by not solved by not relying on relying on relying on intuition, but intuition, but intuition, but by checking the by checking the by checking the results and results and results and applying human applying human applying human understanding. And what you understanding. And what you understanding. And what you actually do actually do actually do when you do it is when you do it is when you do it is interesting. You interesting. You interesting. You intersect your intersect your intersect your a priori beliefs a priori beliefs a priori beliefs with the beliefs of with the beliefs of with the beliefs of the models. In that the models. In that the models. In that first example, first example, first example, anyone would anyone would anyone would have noticed the missing have noticed the missing have noticed the missing in-kind contributions in-kind contributions in-kind contributions if they knew anything about if they knew anything about campaign finance or knew what campaign finance or knew what to check. So, we to check. So, we to check. So, we found it. But people, found it. But people, found it. But people, speaking as a person about speaking as a person about speaking as a person about this specific this specific this specific task, are task, are task, are expensive. They are slow

expensive. They are slow , they slow down , they slow down , they slow down deadlines, they are even deadlines, they are even deadlines, they are even more more more indeterministic than indeterministic than indeterministic than models, and they models, and they models, and they hate interruptions. And hate interruptions. And hate interruptions. And it turns out that it turns out that it turns out that significant significant significant progress can be made by progress can be made by progress can be made by intersecting the a priori intersecting the a priori intersecting the a priori knowledge of different models. knowledge of different models. knowledge of different models. Of our 200 Of our 200 Of our 200 deferred families, deferred families, deferred families, Jev and Luna agreed in 194 Jev and Luna agreed in 194 Jev and Luna agreed in 194 cases. And none of cases. And none of cases. And none of these common these common these common answers were answers were answers were wrong. All wrong. All wrong. All the errors were contained in the the errors were contained in the the errors were contained in the six families where six families where six families where they did not agree. they did not agree. they did not agree. Models from different Models from different Models from different families, with different families, with different families, with different origins, origins, origins, tend to have more tend to have more tend to have more divergent opinions divergent opinions divergent opinions and different a priori and different a priori and different a priori knowledge than models from the knowledge than models from the knowledge than models from the same group. same group. same group. The exception, I won't The exception, I won't The exception, I won't say who, are models say who, are models say who, are models that were that were that were distilled outside of distilled outside of distilled outside of one one one company. So, company. So, company. So, thanks to such work, the thanks to such work, the scope of problems can be significantly narrowed. And scope of problems can be significantly narrowed. And with it, with it, with it, errors, reducing them to a errors, reducing them to a errors, reducing them to a much smaller but much smaller but much smaller but constant class of constant class of constant class of problems inherent in problems inherent in problems inherent in LLM as a species. For example, in LLM as a species. For example, in LLM as a species. For example, in one benchmark, one benchmark, one benchmark, four different four different four different families of models families of models families of models produced the same produced the same produced the same final final final result because result because result because they all had a common they all had a common sign convention that was not present in the sign convention that was not present in the answer key. And answer key. And answer key. And once you get once you get once you get to that point, the only way to that point, the only way to that point, the only way forward is to forward is to forward is to list them, list them, list them, monitor them, and monitor them, and monitor them, and implement implement implement concrete solutions.

concrete solutions. Going back, Going back, what exactly are we what exactly are we what exactly are we missing for missing for missing for self-optimization? self-optimization? self-optimization? The first is a stable The first is a stable deployment environment. We deployment environment. We use use use Hank Weave for this—a Hank Weave for this—a tool that takes the tool that takes the tool that takes the evolving foundation of evolving foundation of agent runtimes like cloud code agent runtimes like cloud code and pie and provides a and pie and provides a and pie and provides a consistent, consistent, consistent, proven proven proven logging and API interface logging and API interface logging and API interface with built-in with built-in with built-in checkpoints checkpoints , rollbacks, and , rollbacks, and , rollbacks, and budget control. Something that budget control. Something that can be can be can be built on. But there are other built on. But there are other built on. But there are other solutions that fit solutions that fit solutions that fit this description. this description. this description. Honestly, for Honestly, for Honestly, for headless headless headless data work, Hank Weave, I would data work, Hank Weave, I would data work, Hank Weave, I would say, remains say, remains say, remains the oldest, the oldest, the oldest, most stable, and most stable, and most stable, and most open. I'll most open. I'll most open. I'll leave a link leave a link leave a link if you want to if you want to if you want to try it. The second is a try it. The second is a try it. The second is a way to limit way to limit way to limit your experiments. your experiments. your experiments. You can't You can't You can't test test if you don't control if you don't control if you don't control costs, time, and costs, time, and costs, time, and access as resources. access as resources. access as resources. The third is know-how and The third is know-how and The third is know-how and methodology. We methodology. We methodology. We found that this new found that this new found that this new class of models can class of models can class of models can take everything we take everything we take everything we learned in 2 years of learned in 2 years of learned in 2 years of deployment, agent deployment, agent deployment, agent patterns, patterns, patterns, debugging techniques, debugging techniques, debugging techniques, prompting, and prompting, and prompting, and apply it to apply it to apply it to complex problems. With the complex problems. With the complex problems. With the right level of right level of right level of abstraction, this can be abstraction, this can be abstraction, this can be expressed very expressed very expressed very efficiently in efficiently in efficiently in terms of tokens, and terms of tokens, and terms of tokens, and models will be able to models will be able to models will be able to implement it.

implement it. implement it. The fourth and The fourth and The fourth and most important thing is the most important thing is the most important thing is the failure plane. failure plane. failure plane. Archimedes said: " Archimedes said: " Give me a Give me a Give me a fulcrum and I will turn fulcrum and I will turn fulcrum and I will turn the Earth." And in our the Earth." And in our the Earth." And in our case, we didn't case, we didn't case, we didn't encounter any problems encounter any problems encounter any problems or limitations that we or limitations that we or limitations that we couldn't solve couldn't solve couldn't solve iteratively, having a iteratively, having a iteratively, having a stable stack to stable stack to stable stack to build on and room build on and room build on and room to catch to catch to catch bugs. This last one bugs. This last one bugs. This last one was the hardest was the hardest was the hardest to put into words, to put into words, to put into words, tools, and tools, and tools, and instructions that instructions that instructions that help models help models help models succeed. The succeed. The succeed. The current best current best current best models barely models barely models barely manage this, manage this, manage this, they are just sparks of they are just sparks of they are just sparks of intelligence, so to intelligence, so to intelligence, so to speak. The entire process, speak. The entire process, speak. The entire process, however, involves however, involves however, involves six six six stages. First, we stages. First, we stages. First, we check the task check the task check the task and divide it. For and divide it. For and divide it. For Jeff, this means Jeff, this means Jeff, this means dividing about 800,000 dividing about 800,000 dividing about 800,000 rows into 200,000 smaller rows into 200,000 smaller rows into 200,000 smaller specialized specialized specialized cases, each of cases, each of cases, each of which is divided into separate which is divided into separate which is divided into separate question types. What type of question types. What type of question types. What type of donor, one donor, donor, one donor, donor, one donor, you know, different you know, different you know, different questions that questions that questions that needed to be needed to be needed to be answered. answered. answered. Then we define Then we define Then we define the space for the space for the space for error detection. In error detection. In error detection. In this case, we this case, we this case, we ran good ran good ran good models on a small models on a small models on a small portion of the data, portion of the data, portion of the data, checking some of checking some of checking some of it manually to it manually to it manually to establish a establish a establish a baseline for baseline for baseline for error control. In other error control. In other error control. In other cases, we cases, we cases, we have set up have set up have set up automated automated error tracking systems, error tracking systems, audit trails, and have also audit trails, and have also audit trails, and have also embraced behavior as a embraced behavior as a embraced behavior as a criterion for criterion for criterion for measurement, but it is measurement, but it is measurement, but it is important to define important to define important to define the room for error the room for error the room for error before starting work. And before starting work. And before starting work. And now the most interesting now the most interesting now the most interesting part. We need part. We need part. We need minimal limited minimal limited minimal limited pilot projects to pilot projects to pilot projects to test test test the architecture. And once the architecture. And once the architecture. And once something starts something starts something starts working, we working, we working, we transfer those tags to the transfer those tags to the transfer those tags to the production setup. The code production setup. The code that is executed between that is executed between that is executed between limited limited limited segments of work is what we segments of work is what we segments of work is what we call codons.

call codons. call codons. These codons are then These codons are then reused, reused, looped looped looped independently, independently, independently, checked, and checked, and sequenced. Eventually, working Eventually, working pipelines pipelines pipelines are deployed into are deployed into are deployed into production, where they production, where they production, where they can gradually can gradually can gradually improve. And it is improve. And it is improve. And it is in this middle in this middle in this middle part that the part that the part that the "surfing" takes place. We "surfing" takes place. We "surfing" takes place. We find find find an architecture that gives a an architecture that gives a an architecture that gives a reliable result, reliable result, reliable result, use it as a use it as a use it as a base, replace base, replace base, replace models and other models and other models and other parts with faster and parts with faster and parts with faster and cheaper options, cheaper options, cheaper options, compare with compare with compare with known intermediate known intermediate known intermediate data, increase data, increase data, increase the requirements for success, the requirements for success, the requirements for success, each time getting closer each time getting closer each time getting closer to the ideal, and simply to the ideal, and simply to the ideal, and simply watch the watch the boundaries move. Being able to boundaries move. Being able to break a complex break a complex break a complex problem down into parts problem down into parts problem down into parts means that each means that each means that each part can take part can take part can take its best place, its best place, its best place, as Adam as Adam as Adam Smith would say. In such a system, Smith would say. In such a system, Smith would say. In such a system, ontologies exist as an ontologies exist as an ontologies exist as an intermediate link between intermediate link between intermediate link between intelligence and intelligence and intelligence and output, output, output, facilitating their facilitating their facilitating their reuse. This is the reuse. This is the reuse. This is the second half of what second half of what second half of what I mean by I mean by I mean by "just-in- "just-in- "just-in- time ontologies." It time ontologies." It time ontologies." It is built when is built when is built when a task needs it, a task needs it, a task needs it, stored with all stored with all stored with all the data, and the data, and reused when the reused when the next task next task next task needs the same needs the same needs the same thing. And once thing. And once thing. And once you have them, you you have them, you you have them, you can make these can make these can make these things cheaper. You things cheaper. You things cheaper. You can run them in can run them in can run them in parallel on different parallel on different parallel on different datasets for datasets for datasets for scaling. I scaling. I scaling. I conclude with an apology: conclude with an apology: conclude with an apology: the answer is that there the answer is that there is no single ontology.

is no single ontology. At least At least At least universal. It universal. It universal. It never existed. never existed. never existed. The good news is The good news is The good news is that this makes any that this makes any that this makes any ontology and schema ontology and schema ontology and schema valid, as long as you valid, as long as you valid, as long as you can label and can label and can label and preserve the context preserve the context preserve the context around them. And if we around them. And if we around them. And if we can harness can harness can harness this cheap intelligence this cheap intelligence this cheap intelligence that is beginning that is beginning that is beginning to spread around the world, to spread around the world, to spread around the world, we will quickly we will quickly we will quickly approach a approach a approach a future where future where any information in any information in any format becomes any format becomes any format becomes maximally useful maximally useful . And at Outpace, we are working hard . And at Outpace, we are working hard to make to make this future a this future a this future a reality. Thank you.