1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS

Hello everyone. I think we Hello everyone. I think we can start. So can start. So , a few months ago , a few months ago , a few months ago I did something a bit I did something a bit I did something a bit absurd. I took a absurd. I took a programming agent, programming agent, pointed it at pointed it at my my team's cloud sandbox environment, and said, " team's cloud sandbox environment, and said, " Try to break this." Try to break this." Try to break this." I left it I left it I left it running overnight, and this running overnight, and this running overnight, and this conversation is about what we conversation is about what we conversation is about what we discovered. My name is discovered. My name is discovered. My name is Kevin. I work on the Kevin. I work on the Kevin. I work on the Amazon Bedrock agent Amazon Bedrock agent Amazon Bedrock agent core team. This is a core team. This is a sandbox platform and we sandbox platform and we do much do much do much more. We provide more. We provide more. We provide agents, agents, agents, identity, identity, identity, memory, memory, memory, runtimes, sandboxes, runtimes, sandboxes, runtimes, sandboxes, gateways—everything you gateways—everything you gateways—everything you need to need to need to build and build and build and deploy your deploy your deploy your agents in the cloud. It all agents in the cloud. It all agents in the cloud. It all started when I took a close started when I took a close started when I took a close look at look at look at how customers how customers how customers use our use our use our platform. And increasingly, platform. And increasingly, platform. And increasingly, many of many of many of us in the industry us in the industry us in the industry are seeing that many of the are seeing that many of the are seeing that many of the components of platforms components of platforms components of platforms on the internet are on the internet are on the internet are simply agents. And simply agents. And simply agents. And increasingly, they are also increasingly, they are also increasingly, they are also placed in placed in placed in sandboxes. I think sandboxes. I think sandboxes. I think today's talks today's talks today's talks will help you will help you will help you understand the understand the understand the architectures that architectures that architectures that can exist in can exist in can exist in cloud sandboxes.

cloud sandboxes. This got me This got me thinking: how can I thinking: how can I thinking: how can I simulate simulate simulate customer actions and test the customer actions and test the customer actions and test the service in the same way that service in the same way that service in the same way that agents agents agents use our use our use our platform. So the first thing platform. So the first thing platform. So the first thing we decided to we decided to we decided to do was create an do was create an do was create an interface that interface that interface that would allow would allow would allow programming agents and programming agents and programming agents and agent applications agent applications agent applications to interact with our to interact with our to interact with our service in the most service in the most service in the most natural way for them natural way for them natural way for them . Today we . Today we . Today we know that advanced know that advanced know that advanced models models models understand the MCP protocol quite well understand the MCP protocol quite well . So we spent a . So we spent a . So we spent a day or two day or two day or two setting up an MCP server setting up an MCP server setting up an MCP server to access our to access our to access our environment so you environment so you environment so you can get can get can get started quickly. started quickly. started quickly. The second stage was The second stage was The second stage was to improve this to improve this to improve this solution. We spent solution. We spent solution. We spent time setting up the time setting up the time setting up the available available available tools, tools, tools, running smoke running smoke running smoke tests, and tests, and tests, and things like that. We ran some tests things like that. We ran some tests things like that. We ran some tests manually: manually: manually: for example, writing for example, writing for example, writing code in Python and executing code in Python and executing code in Python and executing it, or taking a it, or taking a it, or taking a Python file and converting Python file and converting Python file and converting it to Java. Then we it to Java. Then we it to Java. Then we realized that we wanted to realized that we wanted to realized that we wanted to expand the expand the expand the testing scope for testing scope for programming agents. So we programming agents. So we simply asked them simply asked them simply asked them to write 1000 tests in to write 1000 tests in to write 1000 tests in 10 different domains.

10 different domains. 10 different domains. The number 1000 was quite The number 1000 was quite The number 1000 was quite arbitrary. This was arbitrary. This was arbitrary. This was done to cover the done to cover the done to cover the widest possible field and widest possible field and widest possible field and look at the look at the look at the results. This results. This results. This needed needed needed refinement because we refinement because we refinement because we understood that 1000 understood that 1000 understood that 1000 tests would take tests would take tests would take several hours and several hours and several hours and consume a lot of consume a lot of consume a lot of tokens. So we tokens. So we tokens. So we took on the task of managing took on the task of managing took on the task of managing credentials, credentials, credentials, implementing them, and implementing them, and implementing them, and handling cases handling cases handling cases when the test when the test when the test fails. Is this a fails. Is this a fails. Is this a service problem? Is this an service problem? Is this an service problem? Is this an MCP level issue? Um, these are the MCP level issue? Um, these are the MCP level issue? Um, these are the things we things we things we let the agent let the agent let the agent figure out on their own. figure out on their own. So the architecture is So the architecture is actually very actually very actually very simple, but at the same time simple, but at the same time simple, but at the same time very powerful, and I very powerful, and I very powerful, and I think many of think many of think many of us have probably seen this us have probably seen this us have probably seen this in our daily in our daily in our daily work. So we have the work. So we have the work. So we have the encoding agent in the encoding agent in the encoding agent in the top left top left top left corner. We have an MCP server corner. We have an MCP server , which essentially , which essentially , which essentially encompasses our cloud encompasses our cloud sandbox environment. And all sandbox environment. And all the results the results the results are sent back to us are sent back to us are sent back to us . We store . We store . We store them in an SQLite database them in an SQLite database them in an SQLite database simply because, as simply because, as simply because, as some of you may have some of you may have some of you may have noticed, agents noticed, agents noticed, agents tend to write tend to write tend to write files in Markdown format, files in Markdown format, files in Markdown format, but that doesn't but that doesn't but that doesn't scale when scale when scale when you have 1000 tests to you have 1000 tests to you have 1000 tests to run. run. run. This allowed us to This allowed us to This allowed us to leverage leverage leverage concurrency, where concurrency, where concurrency, where you can have multiple sub- you can have multiple sub- agents running agents running agents running locally, executing locally, executing locally, executing their tests, and their tests, and their tests, and storing those storing those storing those results locally.

results locally. Now, 1000 tests—that's Now, 1000 tests—that's probably easy to say out probably easy to say out probably easy to say out loud, but in reality, loud, but in reality, loud, but in reality, each dot on each dot on each dot on the screen represents a the screen represents a the screen represents a worker pool that worker pool that worker pool that the client would actually the client would actually the client would actually run. So we run. So we run. So we did things like did things like sandbox lifecycle testing. sandbox lifecycle testing. How How How reliable is it when you reliable is it when you reliable is it when you put it under put it under put it under load? Um, load? Um, load? Um, things like things like file file system I/O operations. What system I/O operations. What system I/O operations. What happens when you happens when you happens when you upload a file to the upload a file to the sandbox and then sandbox and then sandbox and then upload it upload it upload it from there? Um, from there? Um, from there? Um, they were testing they were testing they were testing performance. We performance. We performance. We tested some tested some software development lifecycles software development lifecycles , , , such as creating such as creating such as creating Python packages, Java Python packages, Java packages, etc. So, packages, etc. So, packages, etc. So, to give you an idea to give you an idea to give you an idea of ​​what of ​​what of ​​what one of one of one of these dots could represent: it could these dots could represent: it could these dots could represent: it could be an end-to-end be an end-to-end machine learning training program. machine learning training program. This could be This could be This could be creating a PDF report. Um creating a PDF report. Um , it could also , it could also , it could also just be opening a just be opening a just be opening a web page. And we web page. And we web page. And we quickly realized that quickly realized that quickly realized that our platform, which our platform, which our platform, which at the time at the time at the time hosted two hosted two hosted two main applications— main applications— one was a one was a code sandbox, the other a code sandbox, the other a cloud browser— cloud browser— needed different needed different needed different tests that had tests that had tests that had different meanings for the different meanings for the sandboxes sandboxes sandboxes they were hosted in.

they were hosted in. they were hosted in. For example, here in For example, here in For example, here in the browser, we could the browser, we could the browser, we could ask an agent ask an agent ask an agent to create a bunch of to create a bunch of to create a bunch of tabs and try to tabs and try to tabs and try to scan them all, scan them all, scan them all, summarize, maybe summarize, maybe summarize, maybe take some take some take some screenshots, screenshots, screenshots, upload them, upload them, upload them, maybe fill out a maybe fill out a maybe fill out a form, form, form, automate the automate the automate the login workflow. login workflow. login workflow. But the point is that these But the point is that these sandboxed applications sandboxed applications usually go usually go usually go through different through different through different workflows that workflows that workflows that place different place different place different demands on the sandbox demands on the sandbox . So, we've . So, we've . So, we've looked at the looked at the looked at the test results, and test results, and test results, and I want to talk I want to talk I want to talk about just three failures that about just three failures that I think I think I think demonstrate some of the demonstrate some of the demonstrate some of the patterns that we've patterns that we've patterns that we've seen in these seen in these sandboxed applications. So, it sandboxed applications. So, it was simple: we was simple: we was simple: we uploaded uploaded uploaded an image, an image, an image, uploaded it, and uploaded it, and uploaded it, and wanted to check if wanted to check if wanted to check if we got back the we got back the we got back the same thing we uploaded. same thing we uploaded. same thing we uploaded. And, strangely enough, that was And, strangely enough, that was And, strangely enough, that was not the case at all. We not the case at all. We not the case at all. We dug deeper into the dug deeper into the dug deeper into the issue and found that the issue and found that the issue and found that the coding level at the coding level at the coding level at the MCP level was MCP level was MCP level was incorrect. The second incorrect. The second incorrect. The second case was also case was also case was also quite interesting, quite interesting, quite interesting, related to the related to the related to the zero byte. He zero byte. He zero byte. He just just just confused our tests. confused our tests. confused our tests. Um, a zero byte Um, a zero byte Um, a zero byte can be zero. It can be zero. It can be zero. It can also be a can also be a can also be a marker in some marker in some marker in some programming languages programming languages programming languages that simply means that simply means that simply means that a line is complete. An that a line is complete. An interesting case, interesting case, especially at the especially at the sandbox level, occurred sandbox level, occurred sandbox level, occurred when we wrote when we wrote when we wrote files to a specific files to a specific files to a specific directory in the directory in the sandbox, and when sandbox, and when sandbox, and when we tried to we tried to we tried to restore them, they restore them, they restore them, they simply weren't simply weren't simply weren't available. So now available. So now available. So now we've started to separate we've started to separate we've started to separate patterns and errors into patterns and errors into patterns and errors into two levels. One was at the two levels. One was at the two levels. One was at the interface level, and the interface level, and the interface level, and the other was at the other was at the sandbox level. And I sandbox level. And I sandbox level. And I think it's important to

think it's important to think it's important to draw a line when draw a line when draw a line when we start we start we start putting putting putting applications in " applications in " sandboxes." They sandboxes." They sandboxes." They deviate a bit deviate a bit deviate a bit from the typical from the typical from the typical API service pattern, where there is a API service pattern, where there is a API service pattern, where there is a long-running service long-running service that exposes an API, and you that exposes an API, and you that exposes an API, and you send a request, send a request, send a request, get a result of get a result of get a result of some form, and know some form, and know what your backend is doing what your backend is doing . But obviously, in . But obviously, in . But obviously, in today's world, we don't have that many today's world, we don't have that many long-term long-term applications anymore. applications anymore. applications anymore. Instead, we have Instead, we have Instead, we have specialized specialized specialized agents who operate agents who operate agents who operate in a "sandbox" in a "sandbox" in a "sandbox" environment. We environment. We environment. We provide access to them provide access to them provide access to them through a gateway. And the demands through a gateway. And the demands through a gateway. And the demands they place on they place on they place on their their their computing computing computing environment will environment will environment will vary greatly. We vary greatly. We vary greatly. We know, and it's an experience know, and it's an experience know, and it's an experience that I think that I think that I think many of many of many of us are familiar with: you're developing us are familiar with: you're developing us are familiar with: you're developing something locally and something locally and something locally and you start to see you start to see you start to see your agent—whether it's Kiro, your agent—whether it's Kiro, your agent—whether it's Kiro, Cloud Codex, or something else— Cloud Codex, or something else— behaving strangely. behaving strangely. behaving strangely. For example, it might For example, it might For example, it might download a package download a package download a package you weren't expecting you weren't expecting , or convert a CSV , or convert a CSV file to JSON or PDF for file to JSON or PDF for file to JSON or PDF for some reason. And all of some reason. And all of some reason. And all of these workflows these workflows these workflows that that that your your your agents want to perform will require agents want to perform will require agents want to perform will require different resources from different resources from different resources from your environment.

your environment. your environment. For example, how much For example, how much For example, how much memory or memory or memory or CPU time CPU time CPU time your sandbox has. The your sandbox has. The your sandbox has. The access rights that access rights that access rights that your processes and your processes and your processes and file systems have. file systems have. file systems have. I think a lot of I think a lot of I think a lot of the talks we'll the talks we'll the talks we'll see today see today see today will talk about how will talk about how sandboxes can sandboxes can sandboxes can have many different have many different have many different architectures. And the broader architectures. And the broader architectures. And the broader point here, to me, is that point here, to me, is that point here, to me, is that all of all of all of these limitations have these limitations have these limitations have pretty serious pretty serious pretty serious implications for implications for implications for your your your agent's ability to solve these agent's ability to solve these agent's ability to solve these tasks. To tasks. To tasks. To visualize this, here are visualize this, here are visualize this, here are some errors we some errors we some errors we saw when comparing saw when comparing saw when comparing two different applications two different applications two different applications hosted in " hosted in " sandboxes". So, sandboxes". So, sandboxes". So, at the top we have a at the top we have a code sandbox, and at code sandbox, and at code sandbox, and at the bottom we have a cloud the bottom we have a cloud the bottom we have a cloud browser. In the browser. In the browser. In the lower right corner are the lower right corner are the failure modes, failure modes, failure modes, highlighted in red, highlighted in red, highlighted in red, which show which show which show how many times the how many times the sandbox actually sandbox actually sandbox actually stopped working. In stopped working. In stopped working. In this case, this case, this case, since cloud since cloud since cloud browsers, as we browsers, as we browsers, as we know, know, know, consume consume consume a lot of memory when opening tabs, a lot of memory when opening tabs, a lot of memory when opening tabs, many of our tests many of our tests many of our tests failed precisely failed precisely failed precisely because of because of because of memory limitations. And this was not a memory limitations. And this was not a memory limitations. And this was not a bug that we bug that we bug that we observed in the observed in the code interpreter environment. code interpreter environment. And when we And when we And when we map them map them map them a little differently, we a little differently, we a little differently, we start to see that start to see that start to see that failure modes failure modes failure modes are centered are centered are centered around the around the sandbox primitives. We know sandbox primitives. We know that some of the components of that some of the components of that some of the components of file systems are file systems are file systems are permissions, permissions, permissions, mounted file mounted file mounted file systems, permissions to systems, permissions to systems, permissions to read, write, read, write, read, write, execute files, execute files, execute files, and whether the file and whether the file and whether the file system is persistent system is persistent system is persistent or ephemeral, and or ephemeral, and or ephemeral, and the tests have been quite the tests have been quite the tests have been quite rigorous. For example, rigorous. For example, rigorous. For example, in the lower right in the lower right in the lower right corner we see how corner we see how corner we see how browser tests browser tests browser tests exhaust exhaust exhaust session memory in the sandboxes,

session memory in the sandboxes, session memory in the sandboxes, signaling a signaling a signaling a complete failure. And I complete failure. And I complete failure. And I think the key thing think the key thing think the key thing here is that when we here is that when we here is that when we put these put these put these applications in applications in sandboxes and sandboxes and sandboxes and expect customers expect customers expect customers to be able to create to be able to create to be able to create arbitrary arbitrary arbitrary workloads, we workloads, we workloads, we have to be very have to be very have to be very careful because the careful because the careful because the sandbox restrictions sandbox restrictions have a huge have a huge have a huge impact on the impact on the impact on the agents' ability to perform agents' ability to perform agents' ability to perform these tasks. Now, these tasks. Now, these tasks. Now, if we look if we look if we look more closely—this slide is more closely—this slide is more closely—this slide is quite complex—in the quite complex—in the quite complex—in the second column we second column we second column we see many of the see many of the see many of the primitives that primitives that primitives that make up the make up the make up the sandbox container sandbox container . We can . We can . We can delve into delve into delve into kernel capabilities, such as kernel capabilities, such as kernel interceptors, or so-called kernel interceptors, or so-called extended BPF programs extended BPF programs , which allow you , which allow you , which allow you to run to run to run kernel modules and kernel modules and kernel modules and intercept intercept intercept programs at the Python level programs at the Python level . They can . They can . They can control exactly how control exactly how control exactly how your agent makes your agent makes your agent makes outbound calls, outbound calls, outbound calls, perhaps making perhaps making perhaps making internal API requests internal API requests internal API requests to your to your to your company's services. If company's services. If company's services. If we visualize this in a diagram we visualize this in a diagram , it's a general , it's a general , it's a general overview of how all overview of how all overview of how all the components of the the components of the sandbox can sandbox can sandbox can affect the affect the affect the ability of your ability of your ability of your workflows workflows workflows to do their to do their to do their job. For job. For job. For example, earlier in example, earlier in example, earlier in our conversations we our conversations we our conversations we mentioned mentioned mentioned secure computing, secure computing, secure computing, which controls which controls which controls system programs that system programs that your applications may need. So, your applications may need. So, let's say I have a let's say I have a let's say I have a machine machine machine learning package that learning package that learning package that needs to create a needs to create a needs to create a separate process. Does separate process. Does separate process. Does my process my process my process or container have

or container have or container have the ability to run the ability to run the ability to run any processes at all? any processes at all? Maybe I have a Maybe I have a task for a task for a task for a code interpreter code interpreter code interpreter that requires that requires that requires loading a loading a loading a Python package into a very Python package into a very Python package into a very specific module specific module specific module so that the Python core can so that the Python core can so that the Python core can recognize it. Or recognize it. Or recognize it. Or are the are the are the Rust binaries in Rust binaries in Rust binaries in the right place the right place the right place when the encoding agent when the encoding agent when the encoding agent needs to run or needs to run or needs to run or execute the program? execute the program? execute the program? All of these very small All of these very small All of these very small details ultimately details ultimately details ultimately impact impact impact your your your agent’s ability to solve agent’s ability to solve agent’s ability to solve business problems. business problems. business problems. Now, Now, Now, taking a step taking a step taking a step back, I find the back, I find the back, I find the idea of idea of idea of using a large using a large using a large language model to language model to language model to create unit create unit create unit tests quite interesting. I think we've all tests quite interesting. I think we've all tests quite interesting. I think we've all done it. I don't even done it. I don't even done it. I don't even know if anyone know if anyone know if anyone writes unit tests writes unit tests writes unit tests by hand these days. And similarly, by hand these days. And similarly, by hand these days. And similarly, if we move to a if we move to a if we move to a higher level—to higher level—to higher level—to integration tests integration tests —we can expect —we can expect agents to agents to agents to understand the understand the understand the problem domain quite well and problem domain quite well and problem domain quite well and write tests for it. write tests for it. But as we But as we move to move to move to hosting more and more hosting more and more applications that are applications that are simply agents in simply agents in sandboxes, we sandboxes, we sandboxes, we begin to see begin to see begin to see the flaws in the model: these the flaws in the model: these the flaws in the model: these low-level low-level low-level primitives and primitives and primitives and controls controls controls can prevent the can prevent the can prevent the agent from achieving its agent from achieving its agent from achieving its intended goal. We intended goal. We intended goal. We hope to hope to hope to show a reliable show a reliable show a reliable way to detect way to detect way to detect such failure modes, such failure modes, such failure modes, which seem which seem which seem quite innocent to us: quite innocent to us: quite innocent to us: this is fuzzing - this is fuzzing - sandbox testing of your sandbox testing of your agent, where you agent, where you agent, where you ask it ask it ask it to execute, for example, to execute, for example, to execute, for example, a thousand a thousand a thousand workflows and look workflows and look workflows and look at the

at the at the results. To results. To results. To make this make this make this clearer: many clearer: many clearer: many of us of us of us developers are used to developers are used to developers are used to running running running encoding agents locally. encoding agents locally. encoding agents locally. You can have your, You can have your, You can have your, you know, your cloud you know, your cloud you know, your cloud code running code running code running locally. And as we locally. And as we locally. And as we expand and expand and expand and create more create more create more workspaces, it workspaces, it workspaces, it quickly becomes quickly becomes quickly becomes a limitation for a limitation for a limitation for your laptop. And your laptop. And your laptop. And now your teams now your teams now your teams want to move these want to move these want to move these coding agents to coding agents to coding agents to remote “sandboxes remote “sandboxes ” so you can ” so you can ” so you can scale scale scale the number of GitHub PRs the number of GitHub PRs the number of GitHub PRs you work on, and you work on, and you work on, and so on. But as you've so on. But as you've so on. But as you've probably noticed, the probably noticed, the probably noticed, the tools on your tools on your tools on your local local local workstation, your MCP workstation, your MCP tools, your tools, your tools, your local applications, local applications, local applications, may not match may not match may not match what's in your what's in your what's in your remote remote sandbox. A quick sandbox. A quick sandbox. A quick hello to agent core. This is hello to agent core. This is hello to agent core. This is actually one of the actually one of the actually one of the issues we are issues we are issues we are actively working on actively working on actively working on as we understand as we understand as we understand that customers want that customers want that customers want to scale their to scale their to scale their coding agents and coding agents and coding agents and how they generally how they generally how they generally manage the software manage the software manage the software development lifecycle development lifecycle . So, . So, one of the recent one of the recent one of the recent releases was releases was releases was the ability for the ability for the ability for clients clients clients to connect via to connect via to connect via SSH to SSH to SSH to their "sandbox" their "sandbox" their "sandbox" to see what's to see what's to see what's going on there. Of course, going on there. Of course, now your "sandbox" is now your "sandbox" is not connected to a person, but to an agent- not connected to a person, but to an agent- coder with whom coder with whom coder with whom you develop you develop you develop the project, but the idea itself the project, but the idea itself the project, but the idea itself remains.

remains. remains. Connecting to the Connecting to the sandbox and sandbox and sandbox and checking what checking what checking what problems problems problems prevent your prevent your prevent your agent or application from agent or application from agent or application from working stably working stably is a very is a very is a very relevant business relevant business task for us. And that's what we're task for us. And that's what we're task for us. And that's what we're trying to solve trying to solve trying to solve for our customers. And for our customers. And for our customers. And if we look if we look if we look a little bit towards a little bit towards a little bit towards academic academic academic research, an research, an research, an interesting example that interesting example that probably worries many of probably worries many of probably worries many of us is the emergence of us is the emergence of us is the emergence of new advanced new advanced new advanced models, where the first thing models, where the first thing models, where the first thing we look at is the we look at is the we look at is the test report: test report: test report: they say, this model they say, this model they say, this model scored so-and-so scored so-and-so scored so-and-so percent in this percent in this percent in this test compared to test compared to test compared to another one. Recently, another one. Recently, another one. Recently, researchers at the researchers at the University of California, University of California, Berkeley, found that with Berkeley, found that with Berkeley, found that with proper proper proper environmental control and environmental control and environmental control and the use of an the use of an exploit agent, they exploit agent, they exploit agent, they were able to surpass were able to surpass were able to surpass test results, test results, test results, reaching 100%. reaching 100%. reaching 100%. Conversely, there was another Conversely, there was another Conversely, there was another study where study where researchers researchers tried to give tried to give tried to give a series of business tasks to an a series of business tasks to an a series of business tasks to an agent with faulty agent with faulty agent with faulty tools, and tools, and tools, and of course, the agent's performance of course, the agent's performance dropped significantly. So what I dropped significantly. So what I want to summarize want to summarize want to summarize today is this: today is this: today is this: as the applications as the applications as the applications we build we build we build become increasingly “ become increasingly “ agent-like” and are not agent-like” and are not agent-like” and are not limited to REST limited to REST limited to REST APIs for clients, but rather APIs for clients, but rather APIs for clients, but rather operate as arbitrary operate as arbitrary operate as arbitrary agents in a “sandbox,” it agents in a “sandbox,” it agents in a “sandbox,” it becomes extremely becomes extremely becomes extremely difficult to detect all the difficult to detect all the difficult to detect all the nuances where the nuances where the nuances where the sandbox constraints sandbox constraints reduce the reduce the reduce the agent’s ability agent’s ability agent’s ability to effectively perform to effectively perform to effectively perform its task.

its task. So my call to you So my call to you today is: if you today is: if you today is: if you plan to sandbox plan to sandbox plan to sandbox your agents your agents , of course, , of course, , of course, test them locally, test them locally, test them locally, but also but also but also fuzz test them fuzz test them fuzz test them with a with a with a broad set of broad set of broad set of domain tests domain tests domain tests remotely. Yes, thank you.