Agents in Orbit: Workflows, Not Autopilot — Eduardo Simon & Javier Casal, Flywire

Hello everyone and Hello everyone and welcome. My name is welcome. My name is welcome. My name is Edu, and together with Hobbie, Edu, and together with Hobbie, Edu, and together with Hobbie, I represent the I represent the I represent the AI ​​platform team at AI ​​platform team at AI ​​platform team at Flywire. Our main Flywire. Our main Flywire. Our main mission is to create a mission is to create a mission is to create a practical agent practical agent practical agent infrastructure that infrastructure that infrastructure that empowers empowers empowers our engineers while our engineers while our engineers while adhering to adhering to adhering to established established established standards and standards and standards and development practices. development practices. development practices. Today we Today we Today we will tell you how will tell you how will tell you how structured structured workflow automation workflow automation provides real provides real provides real business value by business value by business value by ensuring control, ensuring control, ensuring control, security, and security, and security, and reliability of our reliability of our reliability of our development. Before we development. Before we development. Before we delve into the delve into the delve into the technical details of technical details of technical details of our AI platform, our AI platform, our AI platform, let me provide let me provide let me provide some important some important some important context about Flywire. context about Flywire. Flywire is built to Flywire is built to process process the world's most complex payments in the world's most complex payments in industries like industries like industries like education, education, education, healthcare, tourism, and healthcare, tourism, and healthcare, tourism, and business services. business services. Today, we Today, we serve over serve over serve over 5,000 clients in more 5,000 clients in more 5,000 clients in more than 240 countries around the world. than 240 countries around the world. than 240 countries around the world. Because we Because we Because we process process high- high- value international transactions, value international transactions, value international transactions, regulatory compliance, regulatory compliance, regulatory compliance, privacy, and privacy, and privacy, and security are not security are not security are not optional. These are optional. These are optional. These are key requirements of key requirements of key requirements of our business. We our business. We our business. We operate in accordance operate in accordance operate in accordance with strict with strict with strict regulatory regulatory regulatory standards such as standards such as standards such as PCI DSS and SOC 2. This means PCI DSS and SOC 2. This means that customer data, that customer data, that customer data, personal personal personal identification and identification and identification and payment details payment details payment details must remain must remain must remain completely secure. We completely secure. We completely secure. We cannot simply cannot simply cannot simply transfer transfer transfer sensitive sensitive sensitive information to third-party information to third-party information to third-party services without services without services without strict governance strict governance strict governance and architectural and architectural and architectural constraints. To give constraints. To give constraints. To give you an idea of ​​our you an idea of ​​our you an idea of ​​our engineering engineering engineering organization, we have organization, we have organization, we have about 400 about 400 about 400 developers developers developers distributed around the

distributed around the distributed around the world. world. world. We We We support support support a variety of a variety of a variety of technology stacks across our teams, technology stacks across our teams, technology stacks across our teams, including Java, Ruby, Go, including Java, Ruby, Go, including Java, Ruby, Go, Python, .NET, and PHP. Almost Python, .NET, and PHP. Almost Python, .NET, and PHP. Almost everything we build and everything we build and everything we build and run is run is run is deployed on deployed on deployed on Victoria—our Victoria—our Victoria—our internal Platform internal Platform internal Platform as a Service (PaaS) as a Service (PaaS) as a Service (PaaS) running on running on running on GitLab GitLab GitLab and AWS infrastructure. Victoria provides and AWS infrastructure. Victoria provides and AWS infrastructure. Victoria provides standardized standardized standardized build pipelines and build pipelines and build pipelines and consistent consistent consistent deployment practices. So, how deployment practices. So, how deployment practices. So, how did our journey in did our journey in did our journey in artificial artificial artificial intelligence begin? About 6 intelligence begin? About 6 intelligence begin? About 6 months ago, in months ago, in months ago, in April, we officially April, we officially April, we officially created a created a created a specialized specialized specialized team to work with the AI team to work with the AI platform. Even platform. Even platform. Even earlier, our earlier, our earlier, our enthusiasts at Flywire were already enthusiasts at Flywire were already enthusiasts at Flywire were already experimenting with experimenting with AI tools, AI tools, building skills, building skills, building skills, MCP servers, and an early MCP servers, and an early code review bot. This organic code review bot. This organic activity has proven that activity has proven that activity has proven that our engineers are our engineers are our engineers are passionate about passionate about passionate about automation automation automation through AI. It through AI. It through AI. It also showed that also showed that also showed that we needed a separate we needed a separate we needed a separate team to team to team to provide the provide the provide the infrastructure for infrastructure for infrastructure for secure and secure and secure and reliable agent reliable agent reliable agent solutions. Since then, we've solutions. Since then, we've solutions. Since then, we've moved quickly, moved quickly, moved quickly, making nearly 1,000 making nearly 1,000 making nearly 1,000 commits to our commits to our commits to our internal internal internal tool. But tool. But tool. But the industry does not stand still the industry does not stand still the industry does not stand still . In the same 6 . In the same 6 . In the same 6 months, Cloud Code alone months, Cloud Code alone months, Cloud Code alone produced over 160 produced over 160 produced over 160 releases. Almost releases. Almost releases. Almost one every day. And one every day. And one every day. And leading AI leading AI labs labs labs have released more than 20 have released more than 20 have released more than 20 new models. Every new models. Every new models. Every week brings a week brings a week brings a new model, a new new model, a new new model, a new tool, or a new tool, or a new tool, or a new way of working. Our way of working. Our way of working. Our challenge is not challenge is not challenge is not only to only to only to build quickly, but also to

build quickly, but also to build quickly, but also to keep up with keep up with keep up with it. it. it. This brings us to This brings us to This brings us to Orbit, our “ Orbit, our “ Orbit, our “ routine killer.” Orbit is an routine killer.” Orbit is an routine killer.” Orbit is an internal platform internal platform internal platform that eliminates daily that eliminates daily that eliminates daily repetitive repetitive repetitive engineering tasks engineering tasks engineering tasks by integrating LLM by integrating LLM by integrating LLM directly into directly into directly into structured structured structured workflows. Instead of workflows. Instead of allowing AI allowing AI agents to run in agents to run in agents to run in infinite loops, infinite loops, infinite loops, Orbit combines Orbit combines Orbit combines deterministic code deterministic code deterministic code with small with small with small LLM steps. We LLM steps. We LLM steps. We turn turn turn repetitive logic repetitive logic repetitive logic into code and into code and into code and only refer to the model when only refer to the model when reasoning is truly needed. When we were choosing between When we were choosing between development and development and development and purchase, creating our purchase, creating our purchase, creating our own solution own solution own solution turned out to be quite turned out to be quite turned out to be quite justified for justified for justified for our environment. our environment. our environment. Advanced models have Advanced models have Advanced models have significantly accelerated significantly accelerated significantly accelerated feature development, which feature development, which feature development, which significantly reduces significantly reduces significantly reduces the cost of your own the cost of your own the cost of your own platform. More platform. More platform. More importantly, as a importantly, as a importantly, as a financial company financial company financial company working with PCI, PII and PHI working with PCI, PII and PHI working with PCI, PII and PHI data, we keep data, we keep data, we keep execution within execution within execution within our our our Victoria infrastructure, ensuring that Victoria infrastructure, ensuring that Victoria infrastructure, ensuring that sensitive data sensitive data sensitive data does not leave our does not leave our does not leave our secure perimeter. We secure perimeter. We secure perimeter. We also avoid also avoid also avoid lengthy lengthy lengthy security checks from security checks from security checks from suppliers and suppliers and suppliers and unnecessary compliance bureaucracy unnecessary compliance bureaucracy . So, what does the workflow actually look like? Each workflow actually look like? Each agent in Orbit agent in Orbit agent in Orbit is described in a single is described in a single is described in a single YAML file, which YAML file, which YAML file, which is stored in is stored in is stored in our repository our repository our repository and undergoes the same and undergoes the same and undergoes the same merge request validation as merge request validation as merge request validation as any other code.

any other code. any other code. Let's take Let's take Let's take our code our code our code validation agent as an example validation agent as an example validation agent as an example . It all starts . It all starts . It all starts with a trigger—a with a trigger—a with a trigger—a GitLab webhook that fires GitLab webhook that fires GitLab webhook that fires when an engineer mentions a when an engineer mentions a when an engineer mentions a bot for review in a bot for review in a bot for review in a merge request. merge request. Next, a sequence of Next, a sequence of deterministic actions deterministic actions deterministic actions retrieves retrieves retrieves the query metadata, classifies the the query metadata, classifies the the query metadata, classifies the risk level, risk level, risk level, loads previous loads previous loads previous results, and clones results, and clones results, and clones the repository. Only the repository. Only the repository. Only after that do we after that do we after that do we call the model at the call the model at the call the model at the NLM stage, which can NLM stage, which can NLM stage, which can read the code but has no read the code but has no read the code but has no rights to change it and rights to change it and rights to change it and must comply must comply must comply with a strict scheme. with a strict scheme. Finally, another Finally, another deterministic step deterministic step deterministic step publishes the results publishes the results publishes the results in a merge query. This is what we in a merge query. This is what we in a merge query. This is what we call call call workflows, workflows, workflows, not autopilot. The code not autopilot. The code not autopilot. The code does the does the does the predictable predictable predictable parts, and the model is parts, and the model is parts, and the model is only involved where only involved where reasoning is truly needed. We reasoning is truly needed. We intentionally chose to intentionally chose to intentionally chose to work with YAML files work with YAML files work with YAML files and the and the and the merge request review process. YAML merge request review process. YAML is a format that our is a format that our is a format that our engineers already know, engineers already know, engineers already know, and validation works the way and validation works the way and validation works the way they work they work they work every day. This also every day. This also every day. This also makes the codebase makes the codebase makes the codebase AI-friendly.

AI-friendly. AI-friendly. Coding agents can Coding agents can Coding agents can read, write, and read, write, and read, write, and verify verify verify process definitions process definitions process definitions as easily as as easily as as easily as humans. However, not every humans. However, not every use case use case comes from a comes from a comes from a team of engineers. team of engineers. For enterprise For enterprise tasks, we tasks, we tasks, we are considering a are considering a are considering a creation tool creation tool creation tool in the Orbit interface, where in the Orbit interface, where in the Orbit interface, where you can create a you can create a you can create a workflow workflow workflow by by by dragging and dropping dragging and dropping predefined actions without predefined actions without writing YAML. A writing YAML. A writing YAML. A workflow can be workflow can be workflow can be started in three started in three started in three ways. Engineers in ways. Engineers in ways. Engineers in other systems other systems other systems can call can call can call Orbit's internal API Orbit's internal API Orbit's internal API directly or directly or directly or through our internal through our internal through our internal CLI. Webhooks from GitLab, Jira, CLI. Webhooks from GitLab, Jira, CLI. Webhooks from GitLab, Jira, Sentry, and AWS trigger a Sentry, and AWS trigger a Sentry, and AWS trigger a process when an process when an process when an event occurs, event occurs, event occurs, such as the creation of a such as the creation of a such as the creation of a new new new merge request or a new merge request or a new merge request or a new bug in production. bug in production. bug in production. Scheduled tasks Scheduled tasks Scheduled tasks run run run scheduled workflows, scheduled workflows, scheduled workflows, such as daily such as daily translation synchronization, translation synchronization, feature flag clearing, and feature flag clearing, and feature flag clearing, and knowledge base updates. knowledge base updates. knowledge base updates. In addition to these entry points In addition to these entry points , we are currently building an , we are currently building an , we are currently building an MCP server for Orbit so that MCP server for Orbit so that MCP server for Orbit so that other AI services and other AI services and other AI services and tools we tools we tools we use at Flywire use at Flywire can directly can directly can directly run run run workflows. For example, workflows. For example, workflows. For example, an engineer will be able an engineer will be able an engineer will be able to ask Cloud in Slack to ask Cloud in Slack to ask Cloud in Slack via a Cloud Tag to launch an via a Cloud Tag to launch an via a Cloud Tag to launch an Orbit workflow without Orbit workflow without Orbit workflow without leaving the chat. We'll leaving the chat. We'll leaving the chat. We'll start small, start small, start small, with clear and most with clear and most with clear and most effective effective effective use cases.

use cases. First, the daily First, the daily First, the daily engineering work: engineering work: engineering work: automated automated automated code reviews in code reviews in code reviews in merge requests and merge requests and merge requests and our Victoria update agent our Victoria update agent our Victoria update agent , which , which , which automatically automatically automatically fixes fixes fixes Victoria linter results. Victoria linter results. Secondly, translations. Secondly, translations. Secondly, translations. One workflow One workflow One workflow now covers the full now covers the full now covers the full cycle. It receives cycle. It receives cycle. It receives new lines, new lines, new lines, translates them, translates them, translates them, updates files, and updates files, and updates files, and notifies teams with notifies teams with notifies teams with updated updated updated documents. Third documents. Third , errors in the , errors in the , errors in the production production production environment. The environment. The environment. The Sentry webhook launches the Sentry webhook launches the Sentry webhook launches the Sentry investigator. He Sentry investigator. He Sentry investigator. He assesses the issue, assesses the issue, assesses the issue, posts the assessment to Slack, posts the assessment to Slack, posts the assessment to Slack, and shares the and shares the and shares the merge request with the merge request with the merge request with the fix in the fix in the fix in the same place. On the same place. On the same place. On the support side, support side, support side, ticket sorting ticket sorting ticket sorting categorizes new categorizes new categorizes new requests and requests and requests and routes them to the routes them to the routes them to the appropriate engineering appropriate engineering appropriate engineering department. The department. The unit readiness score is an unit readiness score is an indicator of indicator of indicator of how how how ready a repository is ready a repository is ready a repository is for for for AI development. And AI development. And AI development. And the corrector acts based on the corrector acts based on the results obtained. Finally, the the results obtained. Finally, the Jira implementer Jira implementer Jira implementer takes the ticket and takes the ticket and takes the ticket and converts it into a converts it into a converts it into a merge request. It merge request. It merge request. It works like a small works like a small works like a small software factory software factory software factory . One of the . One of the . One of the biggest benefits biggest benefits biggest benefits we've found is we've found is we've found is that we that we that we can make changes in can make changes in can make changes in bulk across multiple bulk across multiple bulk across multiple repositories. This repositories. This repositories. This allows us allows us allows us to engage in to engage in software development on a software development on a large scale. This is large scale. This is large scale. This is very convenient for very convenient for very convenient for migrations, migrations, migrations, library updates, and library updates, and library updates, and legacy versions. We are legacy versions. We are legacy versions. We are also using also using also using this to launch Cloud Doctor this to launch Cloud Doctor this to launch Cloud Doctor and to create our and to create our and to create our own own own AI readiness map.

AI readiness map. AI readiness map. Even with all this Even with all this Even with all this automation, the automation, the automation, the merge request merge request merge request remains our remains our remains our unit of work. And this is where unit of work. And this is where unit of work. And this is where a person a person a person gets stuck in the gets stuck in the gets stuck in the decision-making cycle. We decision-making cycle. We decision-making cycle. We operate in a operate in a operate in a regulated regulated regulated environment, so people are environment, so people are environment, so people are still reviewing and still reviewing and still reviewing and approving these approving these approving these merger requests. merger requests. Let's take a deeper Let's take a deeper look at look at look at Orbit's technical architecture. Orbit's technical architecture. We initially started with a We initially started with a simple simple container setup on AWS ECS container setup on AWS ECS Fargate, where the Fargate, where the Fargate, where the orchestration layer, Harness, and orchestration layer, Harness, and orchestration layer, Harness, and runtime runtime runtime were all in a single were all in a single were all in a single container. As container. As container. As our needs grew, we our needs grew, we our needs grew, we evolved towards evolved towards evolved towards separating the separating the separating the orchestration layer, orchestration layer, agent runtime, and session state agent runtime, and session state into separate components. into separate components. into separate components. This improves isolation, This improves isolation, serviceability, and serviceability, and resource allocation. This resource allocation. This resource allocation. This also makes the system also makes the system also makes the system more robust, as more robust, as more robust, as an error in effort does not an error in effort does not an error in effort does not cause the cause the cause the sandbox to stop. sandbox to stop. sandbox to stop. And this allows us And this allows us And this allows us to scale each to scale each to scale each part independently and part independently and part independently and adapt to adapt to adapt to changing requirements. Fargate changing requirements. Fargate changing requirements. Fargate allows us allows us allows us to scale to scale to scale tasks elastically, tasks elastically, tasks elastically, so we can so we can so we can run a very run a very run a very large number of large number of large number of agents in parallel.

agents in parallel. agents in parallel. Deploying Orbit was Deploying Orbit was Deploying Orbit was also very easy also very easy also very easy because we already had because we already had because we already had Victoria, our platform, Victoria, our platform, Victoria, our platform, which scales which scales which scales applications applications applications effortlessly. But this effortlessly. But this effortlessly. But this scale requires scale requires scale requires safeguards to prevent the safeguards to prevent the safeguards to prevent the platform from being misused platform from being misused . First, time-outs. First, time-outs. First, time-outs. Each agent Each agent Each agent stops after a stops after a stops after a configurable configurable configurable time interval. time interval. Secondly, Secondly, Secondly, speed limits, which speed limits, which speed limits, which are are are quite strict by default. A quite strict by default. A quite strict by default. A broken API integration broken API integration broken API integration could start could start could start creating thousands of creating thousands of creating thousands of agents and significantly agents and significantly agents and significantly increase the size of increase the size of increase the size of our build. These our build. These our build. These limits allow you limits allow you limits allow you to keep the situation under to keep the situation under to keep the situation under control. Third, control. Third, concurrency control. Our concurrency control. Our control panel control panel control panel uses uses uses locking to locking to locking to prevent the same task from prevent the same task from prevent the same task from being performed again being performed again being performed again . . . For example, if a For example, if a For example, if a code review is already code review is already code review is already running, we running, we running, we reject the reject the reject the duplicate request. duplicate request. Running agents in Victoria Running agents in Victoria has container has container has container limitations. Victoria limitations. Victoria limitations. Victoria uses uses uses standard standard Fargate task definitions, so containers are Fargate task definitions, so containers are lightweight and strictly lightweight and strictly lightweight and strictly constrained. Our base constrained. Our base constrained. Our base agent container agent container agent container includes common includes common language runtimes, but intentionally does not language runtimes, but intentionally does not include include include package managers such as Bundler package managers such as Bundler , Maven , or Composer. Therefore, , Maven , or Composer. Therefore, , Maven , or Composer. Therefore, heavy agents like heavy agents like heavy agents like our Jira integrator our Jira integrator our Jira integrator cannot easily cannot easily cannot easily run native run native run native builds or unit builds or unit builds or unit tests inside a tests inside a tests inside a container. When we container. When we container. When we started working on started working on started working on Orbit in April, Orbit in April, AWS Lambda micro-virtual machines AWS Lambda micro-virtual machines AWS Lambda micro-virtual machines didn’t exist yet. AWS didn’t exist yet. AWS didn’t exist yet. AWS released them in June released them in June , so our first , so our first , so our first version version version only worked on Fargate. These are only worked on Fargate. These are only worked on Fargate. These are managed micro-VMs that managed micro-VMs that managed micro-VMs that run as run as run as Lambda functions, meaning they are Lambda functions, meaning they are Lambda functions, meaning they are resumable resumable resumable processes: they can processes: they can processes: they can stop and stop and stop and continue from continue from continue from where they left off.

where they left off. where they left off. This fits very well with This fits very well with This fits very well with the nature of the nature of the nature of agent execution, agent execution, agent execution, so we so we so we use them use them use them together with an together with an together with an execution environment that execution environment that execution environment that is chosen depending is chosen depending is chosen depending on the type of project. on the type of project. on the type of project. Each technology Each technology Each technology stack quickly gets stack quickly gets stack quickly gets its own set of its own set of its own set of tools in an tools in an tools in an isolated isolated isolated environment. This is not a environment. This is not a environment. This is not a complete sandbox yet, but complete sandbox yet, but complete sandbox yet, but we are actively moving we are actively moving we are actively moving in this direction. in this direction. in this direction. The goal is to support The goal is to support The goal is to support setups like setups like setups like Docker in Docker or multiple Docker in Docker or multiple Docker in Docker or multiple micro-VMs so that agents micro-VMs so that agents micro-VMs so that agents can run can run can run tests or more complex tests or more complex tests or more complex applications in any applications in any applications in any repository. repository. repository. We did not We did not We did not create our own create our own create our own toolkit for each step of the LLM. toolkit for each step of the LLM. toolkit for each step of the LLM. Instead, we Instead, we Instead, we use the Cloud use the Cloud use the Cloud Agent SDK—the same Agent SDK—the same Agent SDK—the same toolkit that toolkit that toolkit that powers Cloud powers Cloud powers Cloud Code. This gives us a mature, Code. This gives us a mature, Code. This gives us a mature, well-tested well-tested well-tested toolkit with out- toolkit with out- of-the-box support for tools, subagents, and skills, so we of-the-box support for tools, subagents, and skills, so we can focus can focus can focus on the workflow, on the workflow, on the workflow, not the technical not the technical not the technical details. We add details. We add details. We add control from above. We control from above. We control from above. We limit four limit four limit four different things for different things for different things for each step. The network an each step. The network an agent has access to, the tools agent has access to, the tools they can they can they can invoke, the skills invoke, the skills invoke, the skills they can they can they can download, and the download, and the download, and the MCP servers they MCP servers they MCP servers they can use can use —down to the specific —down to the specific —down to the specific tools tools tools allowed on allowed on allowed on each server. All of each server. All of each server. All of this is declared this is declared this is declared directly in the YAML directly in the YAML file. For example, a file. For example, a file. For example, a code review step code review step code review step can read and search can read and search can read and search code, but cannot code, but cannot code, but cannot edit files or edit files or edit files or commit changes. And commit changes. And commit changes. And our Sentry investigator can our Sentry investigator can our Sentry investigator can request request request traces through our traces through our traces through our MCP server, but only MCP server, but only MCP server, but only with a limited with a limited with a limited set of set of set of reading tools. In reading tools. In reading tools. In practice, each practice, each practice, each LLM step is executed with only LLM step is executed with only LLM step is executed with only the access rights the access rights the access rights it needs, and it needs, and it needs, and these rights are visible these rights are visible these rights are visible and available for and available for and available for inspection like any inspection like any other code.

other code. other code. The skills themselves come from the The skills themselves come from the The skills themselves come from the same source. At Flywire, same source. At Flywire, same source. At Flywire, we maintain a we maintain a we maintain a central central central skills repository skills repository skills repository that that that is published internally as a is published internally as a is published internally as a plugin marketplace plugin marketplace plugin marketplace for Cloud Code. Orbit for Cloud Code. Orbit for Cloud Code. Orbit connects to connects to connects to this marketplace this marketplace this marketplace during assembly. This during assembly. This during assembly. This means that the same means that the same means that the same skills skills skills work in two work in two work in two places. An engineer can places. An engineer can places. An engineer can run code in the cloud run code in the cloud run code in the cloud on a laptop, and Orbit can run it as on a laptop, and Orbit can run it as on a laptop, and Orbit can run it as part of an part of an part of an automated automated automated workflow. This workflow. This workflow. This common foundation also common foundation also common foundation also shapes our shapes our shapes our recommendations for recommendations for recommendations for teams. Start with teams. Start with teams. Start with one skill. one skill. one skill. Skills are easy Skills are easy Skills are easy to create, simple to create, simple to create, simple to improve, and to improve, and to improve, and useful from useful from useful from day one. When day one. When day one. When the team is confident that the team is confident that the team is confident that a task is worth a task is worth a task is worth automating, we automating, we automating, we transfer it to Orbit transfer it to Orbit transfer it to Orbit and turn it into a and turn it into a and turn it into a workflow. The workflow. The workflow. The predictable predictable predictable parts become parts become parts become deterministic code deterministic code . The model is . The model is . The model is only involved where it is only involved where it is only involved where it is useful, and each useful, and each useful, and each run becomes more run becomes more run becomes more reliable and reliable and reliable and cheaper than an open cheaper than an open cheaper than an open session. Workflows session. Workflows session. Workflows also run in the also run in the also run in the background, so background, so background, so engineers don't engineers don't engineers don't need to sit with an need to sit with an need to sit with an open session and open session and open session and manage every manage every manage every step. This means step. This means step. This means less less less context switching and more context switching and more context switching and more time for work that time for work that time for work that really requires really requires really requires their participation. In addition to their participation. In addition to their participation. In addition to execution isolation, execution isolation, execution isolation, we rely on we rely on we rely on structured inference structured inference structured inference to guarantee to guarantee to guarantee system reliability system reliability . Each LLM step in the . Each LLM step in the . Each LLM step in the Orbit workflow Orbit workflow Orbit workflow ensures strict ensures strict ensures strict adherence to JSON schemas adherence to JSON schemas adherence to JSON schemas for for for model responses. By limiting model responses. By limiting model responses. By limiting output formats, we output formats, we output formats, we ensure that ensure that ensure that subsequent branches of subsequent branches of subsequent branches of execution, execution, execution, error handling, and error handling, and error handling, and state transitions are driven by state transitions are driven by state transitions are driven by deterministic deterministic deterministic flow, rather than flow, rather than flow, rather than unpredictable

unpredictable unpredictable string parsing. string parsing. If the LLM response If the LLM response fails validation fails validation fails validation against the expected JSON against the expected JSON schema, Orbit schema, Orbit schema, Orbit automatically triggers a automatically triggers a automatically triggers a targeted targeted targeted retry or retry or retry or fallback. Another fallback. Another fallback. Another important pattern important pattern important pattern we rely on we rely on is isolated is isolated is isolated component evaluation. Evaluating component evaluation. Evaluating component evaluation. Evaluating complex end-to-end complex end-to-end complex end-to-end agent agent agent workflows in workflows in workflows in synthetic synthetic synthetic environments is often environments is often environments is often noisy, expensive, noisy, expensive, noisy, expensive, and difficult to and difficult to and difficult to reproduce. reproduce. reproduce. Instead of evaluating Instead of evaluating Instead of evaluating workflows from workflows from workflows from start to finish, start to finish, start to finish, our our our evaluation system isolates evaluation system isolates evaluation system isolates individual steps of the LLM. individual steps of the LLM. individual steps of the LLM. By injecting accurate By injecting accurate By injecting accurate historical state and historical state and historical state and context context context directly into directly into directly into workflow nodes workflow nodes workflow nodes , we can , we can , we can quickly and inexpensively quickly and inexpensively quickly and inexpensively check the quality of check the quality of check the quality of model responses, model responses, model responses, adjust prompts, adjust prompts, adjust prompts, and detect and detect and detect performance regressions. So, performance regressions. So, performance regressions. So, how do you know if how do you know if how do you know if an agent is of high an agent is of high an agent is of high quality? Today, a quality? Today, a quality? Today, a new agent usually new agent usually new agent usually enters production enters production enters production after a "subjective after a "subjective after a "subjective check." The team check." The team check." The team tests it on tests it on tests it on several real-world several real-world several real-world cases and cases and cases and confirms that it confirms that it confirms that it works as works as works as expected. Once expected. Once expected. Once launched in production, launched in production, launched in production, we track every we track every we track every execution from execution from execution from start to finish start to finish start to finish using OpenTelemetry, using OpenTelemetry, using OpenTelemetry, including every action, including every action, including every action, model, and model, and model, and tool call. We then tool call. We then tool call. We then extract traces extract traces extract traces from real executions and from real executions and from real executions and add them to the " add them to the " golden" golden" golden" dataset. Currently, each dataset. Currently, each dataset. Currently, each such dataset such dataset such dataset is versioned in a is versioned in a is versioned in a repository alongside the repository alongside the repository alongside the code, and we code, and we code, and we run it as run it as run it as needed. Based on needed. Based on needed. Based on these results, we these results, we these results, we improve the prompt improve the prompt improve the prompt or change the model, or change the model, or change the model, then then then run the run the run the dataset again for comparison.

dataset again for comparison. dataset again for comparison. We don't have online We don't have online assessment yet, but assessment yet, but assessment yet, but we are working on it we are working on it we are working on it internally. Our goal is internally. Our goal is to run the entire to run the entire to run the entire evaluation cycle within evaluation cycle within evaluation cycle within Orbit via the interface Orbit via the interface Orbit via the interface or through the MCP server. or through the MCP server. or through the MCP server. First, you First, you First, you deploy your deploy your deploy your agent. Then you agent. Then you agent. Then you monitor and monitor and monitor and evaluate its evaluate its evaluate its performance in the work performance in the work performance in the work environment. Finally, environment. Finally, environment. Finally, you improve you improve you improve the agent based on these the agent based on these the agent based on these evaluations without the need for evaluations without the need for redeployment. This is our redeployment. This is our Roma Roma Roma evaluation platform. A single evaluation platform. A single evaluation platform. A single place to launch, place to launch, place to launch, measure, and measure, and measure, and improve improve improve every every every workflow. The operational workflow. The operational workflow. The operational impact of this approach impact of this approach impact of this approach has been significant for has been significant for has been significant for our engineering our engineering our engineering organization. In the organization. In the organization. In the last 30 days alone, our last 30 days alone, our last 30 days alone, our automated automated automated code review agent code review agent code review agent has completed over 9,700 has completed over 9,700 has completed over 9,700 runs, runs, runs, averaging less than averaging less than averaging less than 3 minutes per 3 minutes per 3 minutes per merge request. In addition to merge request. In addition to merge request. In addition to code reviews, our code reviews, our audit fix tool audit fix tool created 159 created 159 created 159 merge requests to fix merge requests to fix merge requests to fix vulnerable vulnerable vulnerable dependencies, and our dependencies, and our AI readiness tool AI readiness tool improved about 80 improved about 80 improved about 80 repositories. The repositories. The repositories. The Victoria Linter fixer Victoria Linter fixer Victoria Linter fixer processed 413 projects processed 413 projects , and the , and the , and the merge conflict resolver merge conflict resolver merge conflict resolver worked 921 times.

worked 921 times. worked 921 times. In total, In total, In total, we have completed we have completed we have completed over 11,000 runs across these workflows with a over 11,000 runs across these workflows with a over 11,000 runs across these workflows with a 99% success rate. 99% success rate. Looking back on our Looking back on our journey, adopting a journey, adopting a self-service model self-service model was one of the was one of the was one of the best best best decisions we made. decisions we made. decisions we made. Standardized Standardized Standardized constraints allow constraints allow constraints allow SRE, security, SRE, security, SRE, security, IT, and support engineers to easily IT, and support engineers to easily IT, and support engineers to easily and securely and securely and securely create their own create their own create their own workflow steps workflow steps workflow steps . We also . We also . We also realized that realized that realized that cost control cost control cost control starts with starts with starts with model selection. Not every step model selection. Not every step model selection. Not every step requires the requires the requires the most powerful model most powerful model , so each LLM step in the , so each LLM step in the , so each LLM step in the workflow workflow workflow defines its own defines its own defines its own model. For example, the model. For example, the model. For example, the planning step and the planning step and the planning step and the implementation step implementation step implementation step may may may use use use different models. different models. different models. Model-to-task matching Model-to-task matching keeps costs under keeps costs under control, which is why control, which is why control, which is why our our our development plan includes a development plan includes a cost-conscious model router . On the other hand, . On the other hand, data partition management data partition management remains an remains an remains an ongoing responsibility. ongoing responsibility. ongoing responsibility. We maintain We maintain We maintain strict isolation between strict isolation between strict isolation between engineering engineering engineering development cycle data, development cycle data, development cycle data, corporate corporate corporate data, and data, and data, and sensitive sensitive sensitive payment data. payment data. payment data. Additionally, as Additionally, as Additionally, as the demand for the demand for the demand for automation is automation is automation is high, it is important high, it is important high, it is important to stay focused and to stay focused and to stay focused and prioritize prioritize prioritize high- high- high- impact work. Looking impact work. Looking impact work. Looking ahead, our development plan ahead, our development plan is focused on is focused on expanding the expanding the expanding the platform's features and platform's features and platform's features and increasing overall increasing overall increasing overall architectural architectural architectural flexibility. We are actively flexibility. We are actively flexibility. We are actively moving towards moving towards moving towards full full sandboxing so that sandboxing so that sandboxing so that agents can agents can agents can seamlessly seamlessly seamlessly handle complex handle complex handle complex builds, local builds, local builds, local environments, and environments, and environments, and test suites for all types of test suites for all types of test suites for all types of projects. Additionally, projects. Additionally, projects. Additionally, we implement we implement recoverable sessions to recoverable sessions to handle complex handle complex handle complex multi-step multi-step multi-step tasks. We are also

tasks. We are also tasks. We are also building more building more assessment tools assessment tools directly into Orbit directly into Orbit directly into Orbit so that teams can so that teams can so that teams can measure the quality of measure the quality of measure the quality of their workflows their workflows directly on the directly on the platform. platform. platform. In parallel, we are In parallel, we are In parallel, we are redesigning our redesigning our redesigning our orchestration layer orchestration layer orchestration layer to become to become model agnostic, model agnostic, providing flexible providing flexible providing flexible routing between routing between routing between different different model providers, including an model providers, including an automated automated automated router that router that router that selects selects selects the most cost-effective the most cost-effective the most cost-effective model for each model for each model for each task. This will help task. This will help task. This will help us control us control us control costs as we costs as we costs as we scale the scale the scale the platform. We also platform. We also platform. We also want to make want to make want to make the platform independent the platform independent the platform independent of of of build tools (harness) so as not to build tools (harness) so as not to build tools (harness) so as not to depend on a single depend on a single depend on a single solution. In solution. In solution. In conclusion, here are three conclusion, here are three conclusion, here are three things we learned. things we learned. things we learned. First, start First, start First, start small. Choose a small. Choose a small. Choose a repeatable, well- repeatable, well- repeatable, well- defined task and defined task and defined task and prove it prove it prove it works before works before works before scaling it up. scaling it up. Second, Second, Second, use use use the model only where the model only where the model only where it is it is it is beneficial. Everything beneficial. Everything beneficial. Everything predictable predictable predictable remains code, remains code, remains code, ensuring ensuring ensuring the reliability of the reliability of the reliability of workflows. And third, workflows. And third, workflows. And third, ensure ensure ensure people's control and participation. people's control and participation. people's control and participation. Owners, limits, and Owners, limits, and mail verification requests. This is how we mail verification requests. This is how we move from move from move from AI experiments to the AI experiments to the AI experiments to the daily daily daily workflows of our workflows of our workflows of our engineers.

engineers. engineers. Thank you all very much for Thank you all very much for Thank you all very much for your time and attention your time and attention your time and attention today. Please today. Please today. Please contact us contact us contact us if you have any if you have any if you have any additional questions additional questions additional questions or comments.