Stop Giving Your AI Agents API Keys - Kim Maida, Keycard & Mason Egger, Temporal

[music] >> My name is Kim Maida. I work at Keycard and Keycard provides agent access identity and access at runtime for agents. Oh my. >> Hi everyone. My name is Mason Egger. I'm a solutions architect at Temporal and Temporal provides durable execution to your software applications up into and including agents. So, not just agents, but pretty much the entire the entire stack. So. >> So, everything you're going to need for the workshop is at the link that's on the screen right now. You actually do not need to sign up for Keycard ahead of time because the sign up will be included like within the curriculum, but in order to get an account for your own personal use, if you want to play with Keycard on your own after the workshop, you can go to the link that is provided at the bottom and enter that registration code in order to sign up for a personal account.

Okay. So, we'll come back to this slide, but I am going to start out by talking a little bit about agent access. So, one of the things that we do a lot of these days is give API keys to agents to give them access to resources, right? And there's several problems with doing things this way. One being credentials aren't attributed to a user identity or an agent identity. The agent has unrestricted access to any and all permissions in that API key. And finally, we can't just sort of stick human in the loop everywhere because humans make mistakes, too, and many agents also run autonomously. So, the way we solve this Keycard is by using an open standard called token exchange. This This standard has actually been around since 2020 and it extends OAuth 2 and I am going to show you what we're going to be doing today. So, in order to give identity to an agent and the user behind the agent, we need to know who the user is as first order of business. So, an authorization server verifies identities and issues tokens. Identity providers like Google, Okta, Auth0, etc. provide authorization servers.

And if we want to get an identity chain in the agent execution path, then we need the user to log in first. After logging in, the authorization server prompts user for their consent to delegate some portion of their access to resources with a subset of their permissions called scopes. Now, this is kind of the first narrowing of access, right? We're only delegating some of the user's total access to the agent. And the authorization server issues a token I that identifies the user and includes their level of access. So, now if we go into what actually happens in the agent execution path, we have an authenticated user now with a known identity who wants to use an LLM to interact with resources. Now, an agent is a control loop that calls an LLM and in this workshop, that agent is an expense desk agent. That's what we're going to be working with.

It's going to call in an expense MCP server which sits between the agent and a ledger API. This provides the agent with a defined set of tools and it also provides the connection to that downstream resource. Now, an MCP client takes the agent's proposed tool calls and it dispatches them to its MCP server. The runtime is a process that runs with the agent loop and executes calls. So, in our case today, the Expense Desk application is the runtime. We're going to use a LangChain agent that's implemented in this custom Expense Desk app. So, if we follow a prompt through this execution path, the user submits the prompt to the runtime, which calls the model and sends it to the LLM. The model proposes tool calls, which are then dispatched by the MCP client to its MCP server, which executes the tools, and then calls the resource API. The API returns a response to the MCP server.

The MCP client delivers the response to the runtime, which calls the model, and this loop repeats until the model is satisfied and returns the response to the user. Now, if you remember, we have an authenticated user with a token identifying who they are. And this token becomes what we call a subject token as identifies the subject on whose behalf the agent is going to act. So, in order to support token exchange, we need an OAuth client that's capable of executing code. So, that might be a gateway between the MCP client and maybe a third-party MCP server, or a custom agent app, or a CLI wrapper around an off-the-shelf coding agent. In this case today, the Expense Desk agent is a custom agent app that implements token exchange for access to the Expense MCP server. This MCP server also supports token exchange for the next downstream resource, which is the ledger API.

This connects to the database where the expenses are actually going to be stored. Now, the prompt and the subject token are sent to the Expense Desk agent app. The agent loop runs, and the model proposes a tool call. The Expense Desk agent authenticates with Keycloak's security token service. Now, this is a special authorization server. And the Expense Desk agent can authenticate using its client credentials or a workload identity. It also sends the user's subject token that declares the user's identity and their level of authorization. Now the expense desk OAuth client creates a token exchange request asking permission to access the MCP server for that tool call, but only that one specific tool call. So now we basically have three pieces of information. We have the identity of the agent that's requesting access. We have the identity of the user on whose behalf the agent is acting.

And we have the level of access that the user granted to the agent. So now we need to decide if the requested token should be issued or not. And we can do this with policy, which is evaluated against what access is requested from what resource, who the agent is, and who the agent is acting for. Now if the delegation chain and the requested access all fall within the policy, then Keycard issues an access token that can be only only be used at the MCP server. Now this issued token usually expires within minutes and you should never store it. The token is sent to the expense desk MCP client, which makes a tool call to the expense MCP server using that token as a bearer credential. The MCP server validates that token and then uses it to authorize a tool call. The tools need access to the ledger API resource. So at that point the MCP server then initiates a second hop. This is where it sends its own token exchange request to Keycard.

And the request to exchange that token is received from the expense desk for a token to access the ledger API, which is the downstream resource. Now Keycard evaluates request against policy and then mints the credential for the resource scoped only for the access that's needed for that specific tool call. The MCP server receives this token and then it uses it to call the ledger API. The API validates the token and then grants the appropriate requested access. Now, the MCP server never stores the token and it discards it as soon as the call is done. The API response then flows back up and the agentic loop repeats until the model returns the answer to the user. So, at a high level, this is what you're going to see us implement today with Keycard.

And I'm going to pull up this slide again. So, again, I'm going to basically demo or Mason and I are going to demo everything step-by-step on screen. Uh you are free to follow along or if you would prefer to just watch, you can also do that. Uh the workshop is going to use a pre-provisioned shared Keycard organization. So, if you do participate, you are going to be in the same organization as everybody else in the room. So, you'll be able to see what everybody else is doing. Okay, so if you go to the link that is that was provided, you're going to see this. This is the uh workshop running in essentially a sandbox. So, we're going to go ahead and start this. This is going to take a few min a minute or two to to uh get everything all set up. But, basically, kind of the premise here is we're going to start with an expense desk agent that does not require a login. It doesn't know who is acting at any point and everybody who uses the agent is using an API key.

So, I have some folks uh here who are going to be helping to populate the demo, essentially. Um if you want to follow along as well, you would are going to eventually need uh admin access in key cards. So, as we go through the workshop, when we hit that part, uh they'll be watching to see people show up uh in the user list, and they will be granting you admin access as that happens. So, when you begin the workshop, this is what you're going to see. So, basically, this is the application where we don't know anybody's identity. Everybody is using the same MCP server, but they're all running their own agents. They are all using an API key. They might be using the same one, they might be using different ones. We, you know, don't really have a way of knowing that. But, if we look at the expense desk, it's we kind of see your your classical uh expense. You file an expense, and then it either gets approved or rejected, um and then paid out, which is not actually going to happen, so you're not going to receive money. Um But, basically, everything in this expense desk operates through the agent.

So, the first thing we are going to do is we're going to submit an expense. So, I'm going to tell the agent I want to submit a $75 expense for snacks for the workshop. It submits that expense, and then its status is pending. Now, if we go down to the activity here, we can see that uh this expense was created using access granted by an API key. Right? So, we don't have any user identity, we don't know the agent's identity, the agent doesn't really have an identity yet on its own. And what I'm going to do is I'm going to refresh this. And I'm going to approve or deny somebody else's expense. So, let's see this one.

If I hit approve, it's basically uh just going to send that as a prompt to the agent. It went ahead and approved that expense, and now we have the created by was created by this API key. So, I don't know who created that expense, right? Everybody's just using API keys. I approved it, but we also don't have any record of who approved it, either, because again, all we have is an API key. So, the next thing that we want to do is actually give the agent an identity, so that it can act on its own behalf. Uh so, the first thing we are going to do now is we are going to join that shared Keycard organization.

So, if you click the link, if you're following along, if you click the link in the instructions, you will be able to you will get a prompt to log in with GitHub, and you can sign up with GitHub to join Keycard. When you get in, you should say see this workshop AI NYC in the upper left corner, indicating that you're in the right uh organization, and let me move this around so we can see things better. Okay. So, if you're following along, when you join, you're likely to see a couple of consent prompts. These are what I kind of talked about earlier in the presentation, where you are delegating some subset of your access to the agent to use on your behalf. So, this is the key card console. The first thing I want to talk about is applications. Now, applications and resources are registrations.

So, something can be both an application and a resource. And basically, what this means is applications are things that are going to be able to make requests for tokens. They're going to ask for token exchanges in order to then exchange on down the line to access other resources. So, our The first thing we want to do is we want to create an application registration for our agent. So, what we are going to do here is I'm going to call this agent app. And then in order because we're all in the same uh organization here, we want to name things so that we can find our own applications and resources that we created. So, I'm going to use the my GitHub handle. Every app's identifier needs to be unique. So, we are going to do something similar here.

We're going to use implicit consent. And then we need to grab our redirect URL. Now, this is something that we can get out of our uh sandbox image. So, if we go into this. I don't know why that one won't come to the front. Go to the terminal. We are going to run this. This gives you the redirect URL that is specific to your instance of the sandbox. So, I'm just going to copy that. I'm going to Oops, don't need to add this as a redirect URL. So, after there's a login event, this is where the user is going to be redirected back to.

And I'm going to say create application. So, now we have a registration that says our expense desk agent can request tokens, right? So, but our expense desk agent can also receive tokens because it's going to receive the subject token from the user when the user logs in. And in order to do that, we basically need to do another registration, and we need to say that this agent application can also act as a resource, which means it is a target recipient for tokens. So, the authentication token is the token that this agent is going to be receiving. So, if we go into resources, we can add another resource. We are going to call this one The first one was agent app, this one is agent resource.

I'm going to use my GitHub name. Let's use a similar identifier. This is the agent resource. I'm going to set the credential provider to zone provider, which essentially means that Keycloak is the entity that's going to be handling the token exchange and issuing uh tokens for this resource. And then I'm going to come down here, and I'm going to say this resource is being provided by my application, the agent app came out of. So, basically, this is the relationship that tells Keycloak that uh this application can exchange sign-in tokens addressed to this resource. If we don't make this association, then the exchange requests are going to be rejected. So, I'm going to save that.

And now I want to add the things that my agent application depends on. So, if I come back to applications, I find my application. I'm going to go into dependencies, and then I'm going to look at the list of dependencies, and I'm going to add something here. So, the first one I'm going to add is the agent resource, and we need this because our agent application is going to request token exchanges to access uh downstream resources. We also want to be able to request token exchanges for the expense MCP server. So, this is a resource that's already been preconfigured for you. Everybody is using the same one, so we don't need to create a bunch of them that are all named after ourselves. Additionally, we have an agent, and we want to have the API key for that agent stored in Keycard so that we don't have its secret on disk. So, Keycard also provides a vault, and we have that API key configured in the vault already for you.

So, we can select the LLM API. And finally, we're going to select this user info endpoint as well, because this is going to be able to tell us who the logged-in user is. It's going to be able to provide us with that identity information. So, I'm going to say connect. Now, the next thing I need to do is I need to give this agent app an its own identity, and I can't really do that unless it has some way to identify itself to essentially log in. Uh so, I'm going to click on add credential. I'm going to click on client ID and secret. Now, wait. We're going to go into the editor.

We are going into the agent folder. We're going to find the end file. And we are going to paste in that client ID and that client secret. This is the secret. Now, we're doing this because this is essentially like a sandbox local environment. If you were using this in a production environment, you would probably use work something like workload identity. Also, Keycard provides a CLI which can populate secrets for you dynamically and not require them to be on disk. And then finally, I'm going to tell it what the identifier for the agent resource is. So, that was my your an agent resource and then my GitHub name. Going to wait until this says changes saved. And at this point, we have configured Keycard to be able to handle the token exchanges and exchange requests.

Now, what we need to do is make sure that our expense desk application can actually handle that. So, essentially, when we go and look at the code here, we find that in the agent folder, we have this agent off. If we go down, we find Okay, this thing is in the way. I'm trying to move that. This we're going to find the checkpoint, which So, this this is the file where we don't have any authenticated MCP access, right? Now, the way the workshop works is if you at each lesson we have a checkpoint that will provide the code essentially for the end of that lesson. So, what we can do now is we can go and look for the checkpoint for exercise two, which is the one that we're at. So, if we go to checkpoints, we go to authenticated, and then we find the same file.

Where are we here? Okay. So, we want to find this MCP tool set. This is passing in the keycard zone token off from the keycard SDK. And this is what is going to actually make that token request every time the agent actually wants to make a tool call with the MCP server. So, for every individual tool call, we're going to get a token that's scoped to that specific tool call. Now, this call is the this is Okay. So, now if we go to the keycard file, we look for this mint function. This is what actually goes ahead and makes that request to mint a token for the exchange.

So, if we don't have a subject token, that is if we don't have a logged in user, then the agent is actually going to authenticate as itself using a client credentials grant. So, it's going to use that uh client ID and client secret to identify itself to Keycloak. And this means it's acting on its own behalf. Okay, so now if we go to the terminal tab, we basically want to make sure that we activate this updated code. So, we're going to uh run that so it switches over to checkpoint two. And then we have to restart the services. Stop. There it goes. Okay, now if we go and look at the expense desk, we have the screen that will let us continue with an authenticated expense desk agent.

So, if we go ahead and click continue as the agent, then we're going to ask it uh which I can just copy this. Which account is being used right now? And what is my approval limit? And the approval limit is essentially the amount of money that you as a user are allowed to approve for other people's expenses. So, it says I'm signed in as the expense desk agent K Mida, which was in Keycloak that was the uh specific application that we are acting as that we provided the client ID and secret for. And the approval limit for the agent is about $50. So next I am going to tell the agent file a $75 expense or let's just say for some swag for the workshop on my behalf.

So the agent submitted that expense. The memo says it was it filed on behalf of Kim but the actual activity details indicate that this was filed by this specific agent. So the on behalf of Kim is just something I told it. It doesn't necessarily actually have any to verify that because I as a user have not signed in it created this expense under its own identity. Okay. So now if we go into our applications actually I probably could have left that up. Oh, it's right there. We can see the activity in Keycloak.

This is essentially the audit log that Keycloak generates of all the activities that have been happening against this application. And I am going to look for specifically because there's uh several different resources here. To look for the expense MCP resource. Anything that was calling the expense MCP resource, I'm going to limit this to the last hour. And we can see that there was a credential issued using the client credentials grant, which means that the actor was the application itself, and it was accessing the MCP resource.

So now I'm going to go and look for what we have already registered for you, which is the expense MCP actor. So the expense MCP server has multiple registrations as well because it can receive tokens uh from the agent application, and then it can exchange that token on for a downstream token to access the ledger API. So if we click on this one, now we have uh because everybody is using the same resource for the MCP server, we're going to go ahead and just filter it down to it calling the ledger API. And I could technically use any of these, but oops, this is looking for people.

So if we look at one of the calls that the expense MCP server used executed in order to exchange the token that it received from the agent application to access the ledger API. We can see that the delegation chain shows that the application that made the request was the expense MCP server and that it was acting on behalf of this application. And the resource that is accessing is the ledger API. So, this is essentially the the record of the delegation chain for the agent acting as itself. I'm going to go ahead and check that. Okay. So, now we have an agent that can identify itself and act as itself, but we maybe want it to act on our behalf but without impersonating us. So, we don't want it to just say Kim did this.

We want it to say this agent did this thing on behalf of Kim with Kim's permissions. So, the next thing we're going to do is we're going to add human user identity into this process. So, now we have user delegation. At this point, we need to load the expense desk in a new tab because the iframe that's embedded into the sandbox doesn't handle external authentication. Um so, we're going to go ahead and open that in external window. Put this down here. And now we have this sign in button up on the right. I'm going to click that. If it's spinning.

Okay. So, because I was already logged in, it went ahead and logged me in automatically. Um but now what I can do is I can say we can see that my email is here. If I ask the agent which account am I using now, before it said you're use you're acting as this agent, now it knows that I have an identity because I authenticated as a human user. All right. And my approval limit, which is associated with my identity metadata, is $100. So, I haven't filed any expenses under my human user yet. So, if I change this filter to to show all expenses, you can see that these expenses from earlier still um say that they were submitted by the API key. Like, adding all of this doesn't go back and change anything that's happened in the past.

And we can see, you know, examples of the Expense Desk agent acting on behalf of other people as well. So, we have the ability now to act as ourself as a human user, but we have still retained the ability for the agent to act as itself as well. And the way that kind of plays out in this workshop is by the agent having the capability approve expenses that are under a certain amount of money. So, if I I'm just going to do this one. If I submit an expense for, say, $30 for office supplies for the team, it's going to verify who I am, it's going to file the expense. >> And then because the agent actually has the ability to approve expenses under $49.99 on its own, it filed the expense and then the agent approved it as itself.

So if I click on the activity to examine what happened here, it says that this expense was created by me under my email. And then it was approved by the expense desk agent. So it shows, you know, that this is the identifier for the agent that acted in the approval and it gives a reason says eligible office supplies under $50 with a clear business purpose. Uh so now maybe I want to Let's do this $75 one as well. So if I submit an expense that is over that approval limit that the agent can act on on its own, then the status for that one is pending. A human needs to review it.

If I go and say, "Hey uh I think this should be approved. So I'm going to send the message to the agent that says, 'Hey, approve expense 123.'" It says, "I can't approve or reject my own expense. So I have to ask another reviewer to do that." And it's able to do that because we have identities now. We know who all of the users are, who are logged in, and we also know the identity of the agent itself. Uh so before we didn't demonstrate this, but before I could have approved [music] my own expenses or anybody else's because it didn't know who anybody was. So if I go here, let's see. Let's find somebody else's pending expense.

I'm going to say, uh, I want to reject this one. The agent goes through that song and dance. And now if we look at the recorded activity for this expense, it says that it was created by the expense desk agent, uh, because it was from kind of the previous cycle. And it was rejected by my identity. So, now we have a kind of a clear record of who filed what, who approved or rejected what. So, if we go into the KeyCloak console at this point, go to applications, we are going to find the Kitematic application.

I'm going to clear any previous filters. I think they're already gone. I'm going to say actor, and I'm going to look for my email. Change this to hour. And if I click on one of these issued credentials, I can see that the application that, uh, made the token exchange request was my application agent app Kitematic, and it was acting on behalf of me, the human user Kitematic KeyCloak or Kitematic KeyCloak, and it was accessing the expense MCP resource. So, that was the first hop, right? So, now if we go over to the applications again and we want to find the expense MCP actor. Actually, I don't think we need to filter anything. It's all here.

And we go to a credential that was issued for access to the ledger API. We can now see that on the second hop, the expense MCP server made the request on behalf of me and it was accessing the ledger API. So, on the first hop, what happened is the agent took my sign-in token that it acquired when I logged in as my human user and it exchanged the it made a token exchange request to keycard offering that sign-in token in exchange for a token for the next thing in in the uh chain, right? Which is the MCP server. So, it exchanged the subject token for an MCP server token. It then took that MCP server token and called the MCP server to make tool call with the appropriate tool scoped scopes. And then the MCP server took that token that it had received from the agent and it made a token exchange request to keycard asking to give that token up for a token to access the ledger API, which was the next resource in the stream.

And you can trace these hops. You can trace these things across hops because they have a session associated with them. So, we're not going to spend a whole lot of time like digging into the the session numbers and copying them, but you can follow any specific sequence of events uh by the session number will be consistent on the first hop and also the second hop. I can close all of that. And I will now give it up to Mason. >> Awesome. Thank you so much, Kim. I have my little plushy tardigrade Ziggy goes with me everywhere. We have plenty of them. So, make sure you grab one before you leave. Take one home for your kids or for yourself. No shame. Whatever you'd like to do. So, we won't just walk through how to build all of this security and stuff into our agent and stuff. But now, how do we deploy these agents to ensure that they execute to their completion, that they don't fail or when they do fail, we know how to handle that, and we know how to reliably bring them back up so we don't lose all of our process and waste a whole bunch of time and compute. And that's where Temporal comes into play.

>> [clears throat] >> So, Temporal is a durable execution framework. So, essentially what it does is we take your code, we execute it, we maintain the state of the application. If the application were to fail, run into an error, things of that nature, it will reconstruct the state or rebuild the state to the point of failure and continue executing onwards. Certain things such as function calls and things if say a distributed system were to go down, there are built-in retries and all sorts of failure handling modes. So, one of the things that we've seen a lot of with the customers that I work with and with our users is that we need a way of making your agents durable. And in reality, the agent loop is or the agent itself is just a loop. It's a it's a loop of ever calling tool calls to external services. Agents are just distributed systems at the end of the day with a little bit of added spice on top with some thinking. But the problems that we've solved for distributed systems apply to agent loops, and this is where Temporal comes in. It works out really well to keep your application running and going so you don't lose data, so you don't have to re- call LLMs, uh tokens and all of that nature. So, you So, your users don't experience any sort of outage. All sorts of really awesome things with that. So, what we're going to do a little bit is just kind of talk about a little bit of how you would structure uh Temporal application code. I'll leave this here for you to read later. I'm not going to dive super deep into the code itself, but essentially Temporal is a a system where you can you build a workflow and you define it by things what are called activities. Think parent function and child function calls. Easiest way to think about it. We implement our loop uh

as a this parent function call and this becomes a workflow. And every single uh instance of or execution of this uh workflow, so like every single time we would have keycard go through and uh authenticate for user, that would be a workflow. And every single tool call is essentially an activity. These are things that can fail, that we expect to fail, and when they do fail, we either retry them or we rebuild the state and see what happens with it. So, we have our workflow calls here. I'm also I do the other direction for Mac Mac mouse. So, if I go the wrong direction, it's cuz I'm not not using my laptop. Um I remap everything on my computer. Uh and then down here you actually see like this async def SS. This is actually uh what an activity would look like. So, you it looks like normal Python code. It looks like the typical code you would write. Temporal is a code-first system, so you can build all of your things and hook it into anything that you would want. Um so we have our settlement code workflow, which I'm not going to read through too much. I'll leave that to you for you. But again, it's a Temporal workflow that uh essentially allows us to set how often we want to retry, and then we execute the activities. We have our um our debit ledger right here and our mark settled. So, we're able to, you know, add something to the ledger and then mark it as settled. And we have this timer here in the middle for us to kind of simulate some like uh human-in-the-loop time that it would wait take for someone to do this.

Uh and then one of our activities uh for this, as you can see here, is uh this section right here, which is just the typical code. But I want to draw your line your eye to the to the first decorator on top, which is grant uh resource. So, Temporal exposes a lot of native uh primitives within its code base, uh one of which is what's called an interceptor. So, it allows you at certain points within the code to allow for essentially think of them as hooks. So, this is actually something that the Keycard team uh implemented, which is where whenever this um this activity is being executed, it actually goes and gets permission and gets an identity from Keycard, and then comes back, and essentially now we are able to authenticate this through uh our Temporal workflow and our activities through this, and we can maintain this identity separation that Kim was talking about, where you can have each individual worker have its own one. You can have a fleet of them have their own, but you'd be able to identify and limit scope, access, all of those wonderful things uh through there. Um so, the next thing we're going to do is we're basically going to find uh our work I think we already did the workers' identity in Keycard. Yeah. Okay, so we're going to start our worker. So, we actually already set up our Temporal application and set up all of these uh environment variable configurations. So, let's go ahead and start uh our worker.

So, Temporal is a worker-server model a version of execution. You There's a Temporal service that maintains the execution, the application state, all of that. And then there's a fleet of workers. You can think Kubernetes pods, processes within a pod that your work as it's being uh as it gets as the request comes in. So, from here we'll go to start worker, and we'll just start our first worker right here, and so, we have worker PID 572 running. So, now we have a uh automatic uh a Temporal worker that's running the agent that was just done in the previous thing in the previous exercise, which will automatically uh accept our uh expenses. So, we have our worker running right here, and we see our PID is 572. And then we can start a settlement. So, what we're going to do is we're going to click on this client tab here, and we're going to invoke a settlement uh that's going to happen, and we'll see our workflow ID settlement came up. So, if we go back to our worker tab here, we'll actually see the output of our Temporal application as it's logging out what it is doing, and it's currently doing it as you can see here, we debited a token um and minted it for this particular call, and then we're running it. If we go to our Temporal Web UI, and we refresh this here, we can actually see our application running. I have to do this a little quickly, and then I can continue talking.

Oh, no, it's not control Is it There it is. Okay, command-control-C you Macs is not fun for me. So, we had our application running. We said we had our debit ledger. I killed the application cold. So, as you can see here, the running state of this application is gone. We no longer have it. In a traditional system, this would be potentially problematic. We would have to you know, either have saved that state to a database and then re- uh rehydrate it when it comes back online. But, uh with Temporal, that's not really a problem because to the state of our application is stored right here inside of it. So, the very first uh the debit ledger uh function stored the output right here. So, we have a retry policy uh activity task completed. We debited 4,200. So, we have that done. There's nothing running right now, so it's going to start complaining and telling me, "Hey, you have nothing running. Our workflow task has timed out." We have a dead application. Uh and then we go back to our worker mid-settlement. Whenever we're ready, we can just restart the worker. And the application picks up as if the if the failure never happened. It rebuilt the entire state of our agent from all of the previous responses. No LLM calls were redone. No previous activities or functions were re-executed. It has picked up the state, replayed the actions, and then pushed it through, and we got the execution to work.

So, why does durability matter for agents? Well, uh your agents are loops. Um there's a lot of things inside of your agents right now that cost a lot of money. Making LLM calls on on a massive scale is not cheap. There's their tokens you have to deal with. There's a lot of failure states. There's a lot of advanced states. Say you get through a really long loop and you know, you're 12 steps in and you lose the entire state of that application. Now you either have you've either had to built basically built a version of Temporal yourself where you are maintaining the state of your application in a database somewhere, and then you have to build all the reconstruction logic and all the retry logic, or uh you can use something like a durable execution system like Temporal, and it will do all of this for you. So, this allows for you to have your agents go offline, fail, your LLM provider can go down, you can lose an entire region in AWS. Doesn't matter when they when those when that compute comes back when those resources come back online, the agents pick up right where they left off at the exact last spot and continue execution. And if it happens quickly enough, if it's a quick enough failure, your user will never even be aware that they that they lost service of this. Um if you had been on the receiving end of this worker call that I had just done where we waited a minute and you were knew you were going to wait a minute for the approval and that failure happened and it happened and I restarted the worker within, you know, 5 seconds of the failure. As a user on the client side, you probably never would even notice that there was this kind of failure and that's what Temporal does. Is it brings it all the way out uh for you and just keeps the execution going forward regardless.

Now, the interesting thing is um the secrets for all of this were actually not inside of our Temporal event history. So, if we go back to our client over here and we run this command, oop, I hit a button. Oh, I have to actually put in the correct workflow ID. If I don't, it doesn't work. If I go back here and collect this workflow ID. Uh it's on the client tab. Yeah. And we click the up arrow. Now, we're going to hope I know how to paste on Mac. Aha, we got it. Good. So, we can actually see we actually have a little little Python script here that says that no JWT token uh material uh was basically sent. We didn't like our worker through the entire event history here um you can see the the starting process and the result. So, if we click in, I can see the debit ledger here with the result down here. So, we have our uh each individual function is stored there. So, our second function here, mark settled, was also uh settled as well. So, this was the result of it, but nowhere in our workflow if I go to the actual event history, which is this is actually what the event history looks like under the hood, you will not find it anywhere uh because we didn't store it. Um so, Temporal gives you basically two different ways uh to keep the sensitive data out. So, we have the interceptor, which is what Keycard provided for us.

So, they built an interceptor that basically injects it at uh uh keeps the data and the credentials out of it entirely and then injects it back in when it's time to execute. Think of it like almost like just-in-time credentials uh and does that. Or the other thing for Temporal that's really awesome is there's what's called a payload uh encoder. So, what you can do is you saw in our Temporal Web UI, your data was There we go. Wrong button. Your data is just sitting here in nice wonderful plain text. You have the ability to encrypt this yourself. So, you can basically spin up There's like a There's an entire pattern and library in Temporal. You spin up a sidecar, you create an encryption algorithm, you set what the encryption policy is, and all data that comes to Temporal, whether it's self-hosted or in the cloud, is essentially encrypted. And we can't read it. We don't control your keys. We honestly don't quite care about what you're doing inside of it. That's not the part that we care about. We care about putting all these fancy things in order. So, that's a quick overview of like Temporal and building out it building out agents with it. There's a lot of integrations with Temporal you can try out. There's an Open AI, Pydantic AI.

I know there's LangChain and a whole bunch of other ones too as well. Um And I think we even now have like an agentic harness now that we've been building out that's going out too. So, you can come by and see us at our booth if you want to learn more about that. Um And yeah, so the expense desk, you know, basically we I'll bring him back up to kind of wrap it up and we can take some questions if you want. Building out the agent identity, building out all of the agent worker and all of that stuff, and then Temporal becomes the execution layer to make sure that your agent runs properly and doesn't fail in the case of, you know, any sort of cataclysmic failure, API outage, AWS region just decides it wants to disappear for a day, which does happen. So, yeah, that's all I have.