Your Company Should Train Its Own AI Models - Will Brown, Prime Intellect

Hey, how's it going? I'm Will Brown. I uh lead applied research at Prime Intellect and uh hope you are all having a a great AI engineer conference. This is always one of my favorites. Uh, it's always just so cool to see what everyone's working on and building and I feel like it is a place where people get to talk about what the future looks like and like start seeing the trends in the industry and seeing the progress and what works and what doesn't. Um, and that's kind of the theme I want to think about for this talk today is like what comes next from what we're doing right now and what are some of the things that we've been working on and also just been seeing and research that's unfolding that kind of points at what we think the future of AI might look like um at the organization level. uh because we've seen a lot of very cool progress in uh agents and interfaces and bots and things like that over the past uh several months, years. Uh but I there's always like there's always more coming. Things are just moving so fast and I think it's a very exciting time to just imagine and speculate and plan for uh what comes next. Um, and so I don't know if any of you remember back in the early chatbt days, OpenAI had kind of spelled out their framing of like what are the different stages of AI progress on the road to AGI. Um, and I think we've moved along this faster than we thought we

would. uh we like I remember 18 months ago I was uh here in New York at a engineer and giving a talk where we had gotten reasoners and chat bots had been around for a while and reasoners were like the new thing but we didn't really have agents yet. Um, and now we've climbed pretty quickly through uh agents and innovators. And the question next seems to be, okay, what happens for organizations? Like what does it mean for an organization to truly like be AI native, not just as like a a startup that uses a lot of agents and builds everything with cloud code, but like at the like large enterprise scale? um what do companies of the future look like that involve uh like deep integration of agents throughout the life cycle of the organization um and what do we need to build on the path there uh so a bit about prime just for some backdrop um we build a lot of different things around AI infrastructure for training deploying and continuously improving uh models and especially open source models uh and open source tooling and so we do a lot of things And like we roughly do the sorts of things that you would do inside of a big lab. And these are our products. And the reason we do this is because we are imagining a future that is coming pretty quickly it seems where open source models are readily available to train and deploy on your own workflows on your own data in

your own harnesses and can be the sorts of things that organizations can have grow with them over time that allow compounding their institutional knowledge, their institutional advantage from their workflows, from their data, from their people and their processes uh in addition to uh using the models from the Frontier Labs. And so this talk will be a bit about how we see that evolving and unfolding where like what is the role of training in AI native enterprise? How does this coexist with the frontier models? Uh and like what do we think is going to be the version of this that really unlocks the most value for people and what does it look like? Um, and so organizations have lots of knowledge and like the longer an organization's been around and the kind of more established it is, the more implicit knowledge there is that kind of doesn't ever get written down. Um, there are things that do get written down. There's code bases, there's emails, there's processes. Some of these are very explicit. They're like rules of like you must do this, this thing works like that. But a lot of things kind of live in people's heads and they live in between the lines of emails, of the meeting notes, of the kind of who talks to who. Like a lot of companies will have like a a on paper org chart and then a real org chart of like who actually makes decisions and like how does change get adopted throughout the company, what sorts of mandates are likely to take effect. Uh and people have kind of built up this knowledge from being at companies for a long time of like oh this is kind of how things work. This is where the friction points are. And I think in many cases uh and I think this is really one of the big

opportunities for AI at the organizational level is is that many individual people are kind of not in a position where they can change the things that are broken in an organization. Um because no one is actually spelling it out. No one is is like taking a bird's eye view of looking at everything that happens in an organization and saying hey this over here is inefficient. This could be better. this thing keeps happening across all these different teams, but nobody's figured out that the solution has actually been discovered by that team. And these are the sorts of things that like agents are great for. Like agents are very capable at combing through large amounts of textual data that's unstructured and finding these kinds of patterns and turning them into something useful uh and and surfacing them as like, okay, this workflow has a problem because it is wasting time and it's inefficient and it could be different instead. Um, and we don't want to just do this once though. We want this to kind of become the way that organizations adopt AI broadly where uh, agents are able to get the context they need, but also they're able to improve both the agents themselves as well as the organizations. And we want agents to be more like our co-workers, but um, human co-workers learn on the job.

Agents currently learn on the job in relatively primitive ways. Um, and they do this via incontext learning. Um, and oops sorry. I'm going to go back a few seconds so we can edit this on the recording. Um, agents learn in relatively primitive ways through like in context learning. Um, and this typically looks like them kind of having a skill file written by somebody or like looking around a codebase, but they don't get the institutional level latent knowledge that human employees really do. Um, and so they relearn a lot. They have to rediscover things. They spend a lot of input tokens. And I think nowadays people are realizing that like cash to input is like the bottleneck in many agent costs. Um, and even if you do get great cash rates and cash costs go down, you still have to like reprocess a lot of input to redo a workflow every single time. Um, and so to get around this, you kind of want to be able to bake in more into the model, especially for things that are harder to write down as rules.

So I I like thinking about the spectrum from like rules to patterns as well as like conceptually like code versus data. And so lots of things live as like code and like skills are kind of like this, rules are kind of like this. But things like preferences and friction and heruristics and trends. These are all data and institutions have so much data that they are like running on and the employees of the organization implicitly learn that models never really get a chance to learn. And this is the sort of thing that we see as blocking them from becoming true co-workers. Um, and so there is a question of like, okay, what goes in context? What goes in the weights? And part of it is you kind of need to earn it. Like things don't just go in the weights right away. But if you if you see kind of every new generation of models, the models get better not because the prompt is better, but because they train them more. Like training is useful. Training works. Training is why we see the progress we do. And the model is roughly like trained on lots of data that is aiming to approximate things that makes that the labs expect will be like high value repeatable workflows uh at organizations. Um and I like this like I think there's a a process that currently it works. It's how the models get better but if you like spell it out it's relatively inefficient. So like the way that a model would learn how to get really good at like using some combination of like Salesforce tools for example let's say is someone who knows how those tools get used has to be looking for work uh and maybe they

previously worked at a place that were using these tools and now they are uh browsing kind of expert network sites to make some money in their free time and they are sharing their knowledge to someone who works at an expert network company. this person at this network company will then turn this uh take this data from the human and uh pass it to someone else whose job is to create RL environments. This RL environment might get sold then to by someone who doesn't have this expert knowledge to a researcher at a lab who also really isn't the expert in the domain who then will be refining it to put it into a training pipeline uh to go into the next frontier model run to make a model that has kind of through this like game of telephone gotten some knowledge of that human originally. But there's so many steps in this pipeline and it's a relatively secret pipeline. Like people people talk at across two stages of it, but like the human who knows this thing is not supervising the training run is not really directly inputting like how the model gets trained in the way that a human could if they were doing it for their coding agent that they're working with day-to-day. Um, and so what we see as an opportunity and kind of also where we would hope to go is like to close this gap and to allow the person who the same way that when we work with our coding agents, we're prompting them like the same things every time, telling them to like not repeat the same mistakes.

Um, that this is how you could train models. You can tell the models things uh and they will learn these things durably, not just as like a line in your agents MD which kind of accumulates and then you have to refactor it all the time. You can do some of that, but a lot of times what you really want is you're giving kind of contextual preference. You're saying in this setting, this thing that you did was not exactly what I wanted and I would have preferred this instead. And we'd want systems that can compound this into like durable learning. Uh and training is a very effective way of doing this because the shape of data you're giving there is quite close to what you would want uh from like an actual training pipeline. And so there's there's several different types of learning. There's things that people do. There's things that you can do at the harness layer with skills and memory. Uh there's knowledge that just lives uh throughout an organization and file systems and databases uh that you retrieve uh with with tools and with code. Uh then there's the models themselves which have this really broad world knowledge but like we all have I'm sure experienced uh frustration when using a model whose like cut off of knowledge is like a while back and like you can't really talk to the models about the news of the past 6 months nowadays like they don't really they have to go like relearn a world model in context every single time and this just does not allow them to really operate as fluently as they can discuss things that happened longer ago. And so organizations to get the most value out of models, we imagine they will want to

increasingly have models that really understand what is going on in the organization at an intuitive level. And this is the sort of thing that you want to train for. Um, and so how do you get there? Like what is the road for an organization to go from like having this idea in the abstract of like we're going to train models on our data to actually getting there? And I think one really useful starting point is eval as essentially test cases. I think like with code we've gotten to a point of maturity around like testing and life cycles and software development life cycles for ensuring that uh code bases grow healthily over time usually and agents are doing so much of this work for us. And the version of continual learning that I I think is like most practical for most organizations is just treating a model kind of like a codebase where it is the sort of thing that you can have a life cycle around you can add test cases to, you can add features to, you can add uh assets to and this is the sort of thing that the process of using agents like initially you collect traces. These traces have your prompts and they have your feedback to things that models did. And this just from the process of using these models, you are accumulating the assets you need to create evals. It's just that making evals historically has been a little bit painful and a lot of organizations have not necessarily wanted to do it manually. But the agents are quite quickly getting to the to the point where thankfully we we have robots that can can do a lot of the eval curation

for us, especially as we train them more on the best practices of making evals in ways that are robust and amendable to learning. Um, and so I think another useful uh example here is the decision model craze that has kind of been taking the world by storm in the past couple weeks with with Jev and also the parallels around routing and people being very excited about cost optimization because we see that like it's possible to spend very large amounts of money on cloud every month and we figure out like when should we use a cheaper model. Um, and this is the sort of thing that I think people are recognizing needs to kind of live in the harness. Um, but it also is a an attribute that looks relatively similar to uh decision models. And the reason I bring both these up together is because there's things with very clear labels where there's a right answer and there's an opportunity for a human or another agent to give feedback. You can have the big model and a small model do the same task, judge whether the small model was as good as the big model. If so, then we this is now data of like for this kind of thing, we can start routing to the small model more. Same for a decision model that's learning to kind of mimic the way that a human in a certain situation would make a judgment. Uh this is a great labelable action. Um where the human might be doing the approval the same way that like we approve our tool calls in cloud code sometimes, but I think most people have turned on like auto uh tool acceptance. Um but every organization for more bespoke things that aren't just about like tool security are going to have their own rules and preferences. And these are the sorts of things that you could then compound into a decision model. And so this is like a very believable form of

continual learning that a lot of people could just plug in tomorrow and uh see immediate value from. Um but more broadly what we see people wanting to do and this is a lot of the work that that we do at PrimeClect is working with organizations that are looking to do relatively ambitious training projects around turning uh their traces from deploying a model in production for a high volume use case into rich robust environments that are blended real and synthetic that kind of capture the complexity of the real tasks they want to be using longunning agents for. and agents are very useful with sufficient orchestration around them, sufficient tooling and humans in the loop at turning these into environments that are uh rich enough to train in. Um, and we see that this is getting more and more automated with with every new generation of open and closed models. And this means that we have an opportunity to start having this whole thing of taking the stuff that we're already doing and turning it into a self-improvement loop just kind of fall out magically from applying agents to curation of the data and creating interfaces for humans to to give signal about what they want. Um, this is a idea that I've been for my own setup been like playing around with a lot that I think will become a a popular interface for interacting with models more in the future. You're seeing it a little bit in some of the bots, but I think it especially now that models are good at like writing code on the fly.

These kind of widget card features that are like I think a a very practical form of dynamic UI that are also really useful for eliciting human decisions. So, they're kind of like the the tool call approval thing, but much more personalized, much more specific, and also kind of get around the problem of like, okay, you have 20 different bots that are going, and you have to remember which ones are where, and who is curating all this for you, and then people are like, do you want one bot? And then you're like, oh, one's great, but I actually want two. Then you have two, then you have 20. And I think we kind of are oscillating back and forth between this the single stream and the multiream agent threads. But what we really want to do are make decisions. We want to give ideas to the agents, let them run with the ideas and then come back to us when we need to make decisions. And the agents hopefully should make good choices about when to bring us decisions. And if agents can create interfaces on the fly for us that allow us to very directly give them the information they need. One, this kind of is much more ergonomic for us as humans uh than like uh reading trying to parse through a long thread message to see like, okay, where was the question? It's also kind of like the human dual side of the the decision model uh Jev style. Uh but it also is perfect data. And so it's a setting where we've been experimenting with this as a way of training models to kind of get this feedback loop going from having human input with these kinds of natural interfaces be the sort of thing that directly acts as a label and is the sort of thing that you that a model can directly improve on because

there is a very explicit target. uh and it gets around a lot of the problems that we kind of hear about in reward hacking and whatnot. Those things are worth addressing too. But like this is like where we see a very useful starting point being for a lot of these problems. And this can be done doesn't have to be with every model in your stack. It can be the orchestration layer models. It can be the decision models. But it is an entry point for kind of starting to have this compounding of preferences and latent knowledge happen in the models. Um, and so I think there was a recent example today. I just added this uh earlier from a company called a paper instrument that uh they use some of our tooling to do this. They're they're friends of ours. Um, but they trained a model for uh 300K that is kind of paro frontier optimal for knowledge work for things like spreadsheets and slides. And it's like an employee scale investment for an org to do. Uh where like you could like go hire a software engineer or you could like train a model. Um and you can also spread this out over a longer period of time if you're kind of compounding the knowledge over time. But it's the sort of thing that is increasingly becoming quite practical.

Uh and it's the sort of thing that a lot of organizations we see wanting to do now and we kind of see the trend of how much more the demand for this is growing month after month. uh as people start recognizing one the value of this and how kind of the the open models are good enough that you can get them to be much better and also much more personalized by turning your knowledge for your domain, your harness, your environment into a training setting for them. Um but also it's the sort of thing that naturally lends itself to a a learning loop. And so this is a a prototype of something we're doing internally around kind of how we're imagining people can really deploy continual learning in the world which is that you can have traces you can start collecting these and you can then like every night see the proposed experiment look at the things that have been flagged as issues and kick off a training run for the night. Maybe your inference capacity in the day is serving your agents at night employees are not using them as much and you can use that capacity for training instead. And so this is a very natural way that we see organizations able to like quite soon actually do for real continual learning in the weights of the models and start compounding their knowledge into uh better and better capabilities for their agents.

Um and so this is kind of the where we see a lot of this culminating towards there's so many things you do. First you discover your workflow, you uh create skills, you make evals, you do routing, and then eventually you start to want to get the economies of scale, get the compounding benefits that come from the data inside of your companies, your organizations, and and have the organizations become increasingly self-improving in ways that uh get better and better over time. Um, that's what I got. Uh, I can maybe take questions if there's any time like I don't know, no questions. Um, and uh, yeah, we're at Primes. We are working a lot of the stuff. A lot of it is open source and public and we uh, blog about it and tweet about it, but also we work with companies and we're hiring and growing and so get in touch if you want to get in touch. Thanks. [applause]