Your AI Agents Shouldn't Care Where They Run - Nikhil Harithas, Factory

Uh good afternoon, everybody. Um I know this is one of the the last talks of the afternoon, so so hopefully people got some energy left. Um my name is Nikhil. Uh I lead forward deployed engineering here at Factory. Um and you know, uh this is this is this episode of Agents and Infrastructure is uh basically came out of like notes from the field. And you know, so we um we really believe that uh deployed engineering is um you know, part of the sensory nervous system of an engineering org. And essentially, it's trying to figure out uh if for our most complex customers, uh what's difficult, what we need to build, and use that kind of inform our product strategy. And the customer in question uh today is going to be Blackstone. Uh some of you may have heard of them. They're uh you know, tricky little large financial institution, uh trillion dollars in assets, and kind of represent this kind of like fractally complex enterprise, right? Where they have a lot Whoa.

Have a lot of uh have a lot of regulations, have a lot of security constraints, and are really really opinionated about how to deploy uh non-deterministic software in uh in their environments. And it's you really one of those uh you know, scenarios where like none of our defaults like work. It's kind of like, you know, they couldn't run agents in uh their their laptop. They couldn't uh they they wanted to bring their own compute and their own inference, uh their own identity, and and single tenant everywhere, right? Um and I think um I I realize now that I'm I should probably pause and probably explain a little bit about what we do in just to make sure that uh in case nobody else nobody knows. But um we we bring autonomy to software engineering. Uh we're in the business of um you know, doing that by uh our coding agents named Droid, and using that to compose software factories. And so you can imagine this requires a lot of permissions. And that is a particularly tricky thing to do when the thing that we were trying to do with Blackstone is basically doing delegation. So you can imagine through Slack or through Jira or through kind of your kind of client of choice basically saying you know assigning something to Droid and Droid is the name of our agent and so Droid going off and can basically completing a task saying okay, I want to assign the Jira ticket to Droid and it'll come back to you with like a PR. But I think that the but things get complicated when you have all these constraints of an enterprise right?

The two fundamental questions that kept coming back again and again is who is an agent acting on behalf of and where does it run? It is you know especially because so many of agentic products that say over the last kind of year 18 months were able to piggyback off of our individual identities. So like maybe you spun up a a sandbox and it just it had like your identity attached to it and it was able to use like the the permissions that were attached to that sandbox. But I mean ultimately like we really think that um that is like was like a kind of kind of temporary idea because ultimately agents can't just keep borrowing your identity and they're ultimately they're just applications. Agents are are basically just like slightly non-deterministic applications and you know we don't need to reinvent the wheel here. We were doing this for maybe decades of software engineering and it's about using the same systems of the past but kind of maybe like slightly repositioning them for the future. And so you know another you know at this conference there's um plenty of people offering you know agent sandboxes. Maybe some of you want to run them in EC2 fleets and we just found that this problem of combination of where does an agent run and who is it who is running as gets com- was coming up again and again and again.

And we found that even in the middle of our engagement with Blackstone like you know there was we were going to go with one of these one of these providers, but then they kind of changed some like key constraints that meant that we wanted to maybe move to an EC2 machine. And and this just kept coming up to our to our customers again and again, where it is like somebody wants to run it on Kubernetes, somebody wants to run on EC2 or G Cloud or whatever it is. And we just needed a way to kind of organically move our product to each one of these things. And and in the past we had a bring your own compute um product. But what we found is that just installing an agent on a piece of on a computer was insufficient because that means that our customer still needed to deal with the compute life cycle. It turns out that's a like pretty complex problem. So what we came up with was was Open Compute. Open Compute is a to be open sourced project that is basically trying to abstract the compute life cycle for any agent. That is we obviously built this for Droid, but it is the idea is that this should be able to work for any arbitrary agent in your customer's environment. And this is I mean this is a really important idea for us because um one of our, you know, key differentiators is that we care deeply about building you a custom fit suit to deploy our product in your environment on your terms. So that means, you know, your security constraints, you know, on the compute of choice, and like totally in your VPC without it ever kind of like leaving your environment, right? Where

as much as possible is on your terms. Um so this gets you a lot of the advantages of what maybe an on-prem deployment could give you without also having like the pain of also having to port over your control layer. Like if you think about like in the previous world you would have your data plane and your control plane, and this is the kind of an emergent idea of like a compute plane really needs to be kind of spun out into its own thing for agents. So what you end up with is this kind of pattern, right? I mean this is just an an example simplified pattern of you have maybe a client of some kind, you have maybe you're delegating from you know, a third party of some kind and you can you basically uh talk to our back end which has these two concepts of a compute gateway and a relay and the compute gateway kind of tells this open our open compute driver to say somebody wants to start a new session which spins up compute in their VPC which could be kind of your micro VM of choice it could be in EKS it could be EC2 and then that sandbox uh is the is the machine that actually talks to the relay which talks to the client.

That was a lot of words so I'm going to let's let's do that real slow. So let's say you're an agent and so let's say you're a client which could be our CLI which could be our product it could be on frankly one of your products and you have something that says I want to start a new agent session. And so that session talks to our back end and says so that's that request talks to our back end and says I want to get compute to start this new session. And so we go to our compute gateway which is also an abstraction of like you know maybe we're using open compute to get compute but maybe we're also using a sandbox you know so it's an it's an it's an it's another abstraction to make this kind of even clearer to say okay and now this this gateway is going to say I'm going to talk to open compute which was and this is like a which already has an open connection to the control plane of the open compute application in the customer's VPC and says hey I want to create a new I want to create a new computer.

And so what open compute is going to do this control controller is going to do and say is going to say I'm going to talk to this idea of a driver. A driver is a kind of like a wrapper for a given kind of compute so this could be you can imagine like one driver for EC2 one driver for Kubernetes you know where your pod lives it could also be you know maybe for a sandbox provider like Daytona or Modal. So like it is like totally um like it is an abstraction that could wrap around your compute layer of choice. So, Open Compute would talk to your driver of choice in that in this example on EC2 machine, an EC2 driver, and say, "I want a new you know, I want a new EC2 machine." So, then we spin that up, and then and then we we connect that EC2 machine that reaches out with to a to what we call our relay, which is something in our control plane, and keeps an open web socket. So, and this is kind of actually an important idea because one thing that we hear from customers all the time is that they're very kind of like nervous about ingress, but less nervous about egress.

So, basically, what we promise in the system is that once set up, we never need to make an inward connection. It's always the customer that's making an outward connection and establishing web socket connection. Excuse me. So, essentially, so once um so once and there's a new like EC2 machine that is going to be like where our agent daemon starts, it's going to establish a open connection to our relay. And then when you have, you know, our CLI or or in Slack or or our, you know, our web application, it's when you send a hello world, it's going to go to our relay, which is able to relay this like literally relay this information to this this request to the sandbox. And they're able to kind of have a back and forth. And to kind of like just kind of close out this idea, in this particular example, we can assume that the the actual LLM call could be established like a BYOK situation where let's say a company, I mean, this is kind of only relevant for I'd say fairly mature engineering organizations, which may already have a deal with, you know, one of the labs, so you can they can like bring their own key and you can imagine like a bedrock situation or something like that to do your LLM calls. So, even that stays within sight of VPC.

So, in this way, without kind of having like a deploying factory entirely on prem, like where you need to have each one of these pieces kind of like loaded in your environment, you're able to get the most of of what you get from that, of like the control of the non-deterministic part of the agent loop, and you're going to feel like you have full observability and controllability, and make sure that like the blast radius of anything that you think is like too spicy for an agent to do is going to be contained in inside a customer's VPC. And this is I mean, we've seen like I'd say pretty significant success with this, just because this kind of gets you really far in in kind of gaining trust in in kind of a customer relationship. We believe that also kind of like long-term that like owning your sovereign is going to become a more and more important idea. And I think that that's often talked about with respect to models, but I mean, I think that that the entire stack is something that eventually large enterprises are going to want to own. And so, you can imagine um you know, us building out components to make it easy for a company to do all of that on their own terms, which is everything from owning their own eval, owning their own model, to owning their own compute layer, to also hosting like the product in in in their VPC, in their in their environments, and being kind of being able to tell the entire story on their terms.

So, so basically that you know, in a few weeks, we plan to open source this, and [snorts] I think that it's going to be useful for people just because the abstractions like have nothing to do with factory. They're to do with managing compute, and like being friendly for an agent to kind of interact, right? So, it is the the unit of work is is a computer. So, you can imagine all the kind of classical things that you'd want to do in a compute life cycle, which is, you know, being able to pause and resume, destroy and start, and all of this is based off of like templates that you can control in terms of you know, what you install, and what sha you can have like basically snapshots of you know, the exact requirements of your build, your your containers of choice. And and that So, that's So, the what you see here is kind of like the spec of like what our factory compute gateway calls. Like this is like the abstraction that you need to kind of um kind of convert to if if you're interested in this pattern. And and again a fairly standard pattern of what you do in a sandbox is to you know suspend or create a snapshot or to fork.

And especially the fork is is like a really important idea because you can imagine especially as fleets of agents and swarms of agents are are kind of gaining steam. I think that one session delegating to another and having sub agents and sub agents and sub agents this is like fractal idea means that being able to you know save a snapshot, fork it, and hand off like a subtask of your problem to another agent is going to like be a really important pattern. So, yeah snapshotting is also going to be really important. And we also plan for this to be another thing that a customer can own. So, we've talked a lot about compute, but you can also imagine like being able to save snapshot on S3 or your kind of like database of choice and eventually also being able to hydrate it with secrets in your secret store of choice. So, being giving customers like the optionality to kind of like hydrate, you know, their compute layer however they want and wherever they want. Um and so yeah, drivers are basically this idea of um the different compute stories. Let's see. Yeah. And I think that um I think an important idea here is that we are that this is a project to manage computers and not sessions. And I think that was a really important idea for us because the moment you tie in the idea of a session and by session I don't mean like a a web session. I mean like a factory back and forth like a factory session is that you get tied into it it becomes

like factory specific where I think that that makes it harder for for a customer to feel comfortable with with hosting that software. Whereas this is like eventually going to be open source and a totally like abstract concept that they could use for for you know for products beyond factory. Um and I think just to give an idea of like the kind of complexity that this solves, like the [snorts] in in dealing with uh like compute life cycle, you might, you know, fire off a request saying, um I want to start a new session, and maybe the maybe the AWS says that uh you know, I received your request, uh but it never acts with like the new sandbox of sorts. Um you maybe your request is maybe uh mid-session, one of your requests is dropped, and you don't get an ack back. Just to come something to manage like the bookkeeping of uh did I start a sandbox for you? Uh what was it called? Where how do I route requests from a given user to a given uh sandbox?

And um and being just being able to retry to make sure that like the client uh and the sandbox understand where they are in a given conversation, and to just like do that bookkeeping for you. Um and I think that you can imagine that this is um I mean, managing arbitrary compute is like a fairly complex problem. I mean, one example is uh you know, the fact that uh you know, micro VMs in AWS have like have an have an 8-hour session cap. Um and one of our products is uh is missions, which can run for like multiple days of like multi-agent orchestration. And so, you know, how do you uh save uh a sandbox uh with a snapshot and rehydrate a new one and kind of like swap in place as if nothing happened? It's like, you know, pretty difficult problems. And uh you know, some of the stuff that we've had to kind kind of uh slog through. And and additionally, just kind of like, you know, while this has been in kind of like private preview, um how do we actually communicate this idea like while it's not open source? Um so, like versioning, uh Terraform, and sharing you know, these Terraform packages to help folks stand it up has been has been interesting.

Um and yeah, I mean, I think that um another um kind of interesting thing that came out of like the way this problem was defined is that if you uh you can just kind of provide the spec to to Droid, uh and just say just make me a driver that that fits the spec and it and it kind of like is its own test like if like if something can it's almost like a it's a very good version of test driven and spectrum development where if it satisfies a given this given spec it's it's basically ready to go. So it's very easy to kind of contribute new drivers and add new kind of compute types so this can just kind of keep getting flex more and more flexible over time. But yeah, I mean ultimately just to kind of close out agents shouldn't care where they're on. This is something that comes up for people again and again and this is the way we solved it at factory and and how we are able to give a really high level flexibility to our deployments for our customers around the world and that's all. Thank you.