From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot

Hello everyone, thank you for Hello everyone, thank you for coming. My name is coming. My name is coming. My name is José Palafox. I've been on GitHub for José Palafox. I've been on GitHub for José Palafox. I've been on GitHub for 6 and a half 6 and a half 6 and a half years. I joined the years. I joined the years. I joined the team right team right team right after GitHub was acquired after GitHub was acquired after GitHub was acquired by Microsoft, and by Microsoft, and by Microsoft, and later GitHub acquired a later GitHub acquired a later GitHub acquired a tool tool tool called Semmle. We called Semmle. We called Semmle. We acquired a acquired a security company. security company. So for the last five or So for the last five or six years, my role six years, my role six years, my role at GitHub has been to at GitHub has been to at GitHub has been to grow our grow our grow our security focus. security focus. security focus. I've I've I've been focused on been focused on been focused on artificial intelligence for the last year, artificial intelligence for the last year, artificial intelligence for the last year, but my job is to but my job is to but my job is to engage with engage with engage with customers: how they customers: how they customers: how they implement our implement our implement our technologies and how to technologies and how to technologies and how to scale them scale them scale them within their within their within their organizations. organizations. Today we're Today we're going to talk about introducing going to talk about introducing going to talk about introducing agents on the agents on the agents on the platform, and there are two platform, and there are two platform, and there are two different modes in which I different modes in which I different modes in which I look at agents. There are look at agents. There are look at agents. There are autonomous agents that autonomous agents that autonomous agents that work in a pipeline, work in a pipeline, work in a pipeline, and there are local agents and there are local agents and there are local agents that I use that I use that I use to perform various to perform various to perform various tasks. I'll tell tasks. I'll tell tasks. I'll tell you how to build them, you how to build them, you how to build them, how to think about them, how to how to think about them, how to how to think about them, how to scale them, and scale them, and scale them, and what workflows are what workflows are what workflows are possible using possible using agents. This report agents. This report will be quite will be quite will be quite general. So, if general. So, if general. So, if you have any you have any you have any additional questions additional questions additional questions or would like to or would like to or would like to chat, don't chat, don't chat, don't hesitate to reach out to hesitate to reach out to hesitate to reach out to me. After the performance, I me. After the performance, I me. After the performance, I will be in the back of the will be in the back of the will be in the back of the hall. You can feel hall. You can feel hall. You can feel free to come and free to come and free to come and chat with me chat with me chat with me after it's finished. We after it's finished. We after it's finished. We can can can discuss your discuss your usage scenarios in more detail. This is a usage scenarios in more detail. This is a brief overview of what is brief overview of what is brief overview of what is available on the available on the available on the platform. Let's start at a platform. Let's start at a platform. Let's start at a basic level: I basic level: I basic level: I always like to always like to always like to start with a start with a start with a general flow of general flow of general flow of data about how data about how data about how Copilot works on the Copilot works on the Copilot works on the platform, as platform, as platform, as many people are not many people are not many people are not familiar with how it familiar with how it familiar with how it works "under the hood" works "under the hood" works "under the hood" and may not be aware and may not be aware and may not be aware of all of our of all of our of all of our interfaces. So, to interfaces. So, to interfaces. So, to give you a quick

give you a quick give you a quick overview of the overview of the overview of the product's capabilities, you have product's capabilities, you have product's capabilities, you have a few different a few different a few different interfaces to interfaces to interfaces to interact with the interact with the interact with the encoding agent. You have an IDE, you encoding agent. You have an IDE, you encoding agent. You have an IDE, you have the GitHub CLI, which is a kind of have the GitHub CLI, which is a kind of have the GitHub CLI, which is a kind of lightweight lightweight lightweight interface for interface for interface for your your your desktop. You can desktop. You can desktop. You can take any session take any session take any session from the CLI and broadcast it from the CLI and broadcast it . So, if you want to . So, if you want to . So, if you want to close your laptop or close your laptop or close your laptop or step away from it for step away from it for step away from it for a while, you can a while, you can a while, you can either delegate the either delegate the either delegate the agent to github.com to agent to github.com to agent to github.com to run in the run in the run in the background, or background, or background, or broadcast it broadcast it broadcast it using the slash using the slash /remote command, which will allow you /remote command, which will allow you /remote command, which will allow you to log in via your to log in via your to log in via your mobile device and mobile device and mobile device and control the agent control the agent control the agent directly from there. You directly from there. You directly from there. You also have the SDK. So, if also have the SDK. So, if also have the SDK. So, if you want to create you want to create you want to create agents that work agents that work agents that work in Slack or Teams, providing in Slack or Teams, providing in Slack or Teams, providing individual features to individual features to individual features to team members who team members who team members who may not have a may not have a may not have a GitHub license, you GitHub license, you GitHub license, you can can can do that quite easily with the do that quite easily with the do that quite easily with the SDK. So SDK. So SDK. So you can you can you can build applications build applications build applications using the CLI or SDK. using the CLI or SDK. using the CLI or SDK. No matter No matter No matter which door which door which door you go through at the top, you you go through at the top, you you go through at the top, you end up in this blue end up in this blue end up in this blue box that we box that we box that we call Cappy or the call Cappy or the call Cappy or the Copilot API service, and it's Copilot API service, and it's Copilot API service, and it's also called a " also called a " harness," harness," harness," right? This is the right? This is the right? This is the component that component that component that does all the heavy does all the heavy does all the heavy engineering work: engineering work: engineering work: it takes your input it takes your input it takes your input data, breaks it into data, breaks it into data, breaks it into pieces, and sends it pieces, and sends it pieces, and sends it to the models, ensures to the models, ensures to the models, ensures there is no there is no there is no duplicate duplicate duplicate content from content from open source providers, handles authentication, authentication, metrics logging, metrics logging, metrics logging, auditing, and all other auditing, and all other auditing, and all other technical tasks of the technical tasks of the platform infrastructure. And of course, platform infrastructure. And of course, when you send when you send when you send a message, we a message, we a message, we take it and take it and take it and pass it on to the pass it on to the model providers. So, model providers. So, no matter no matter no matter which door you

which door you which door you use, you use, you use, you get into the " get into the " harness" and we harness" and we harness" and we take your data and take your data and take your data and send it to the send it to the model providers. Once we receive model providers. Once we receive the response, we the response, we the response, we filter it and filter it and filter it and display it to you. This is display it to you. This is display it to you. This is how the product works " how the product works " under the hood" so that you under the hood" so that you under the hood" so that you have a general have a general have a general understanding of the understanding of the understanding of the ways you can ways you can ways you can interact with it. interact with it. Now, if you're Now, if you're starting out, starting out, starting out, for example, through the CLI, for example, through the CLI, for example, through the CLI, you have several you have several you have several built-in agents, and built-in agents, and built-in agents, and I find it useful I find it useful I find it useful to take a look at them to to take a look at them to to take a look at them to understand what exactly the understand what exactly the understand what exactly the agents do. After all, agents do. After all, agents do. After all, there are many different there are many different there are many different ways to implement it ways to implement it —whether it's an agent or a —whether it's an agent or a —whether it's an agent or a skill, ultimately it all skill, ultimately it all skill, ultimately it all comes down to comes down to comes down to markdown markdown markdown at the end of the day. So it's at the end of the day. So it's at the end of the day. So it's useful to see useful to see useful to see how we implement how we implement how we implement agents on the platform agents on the platform agents on the platform in a way that you in a way that you in a way that you can can can use. use. use. So, I just type / So, I just type / agents and it brings up the agents and it brings up the agents and it brings up the standard agents standard agents standard agents that are part of the that are part of the that are part of the CLI binding. You CLI binding. You CLI binding. You see I have see I have see I have several of them, right? So several of them, right? So several of them, right? So if I want to explore a if I want to explore a if I want to explore a codebase, codebase, codebase, for example, I don't for example, I don't for example, I don't necessarily have necessarily have necessarily have to use the to use the to use the largest possible largest possible largest possible model to do that. model to do that.

model to do that. So we have a So we have a So we have a specialized specialized specialized research agent, research agent, research agent, our research agent, our research agent, our research agent, which you can which you can which you can invoke via a invoke via a invoke via a keyword or, keyword or, keyword or, actually, via the actually, via the actually, via the {slash} command in the CLI, and {slash} command in the CLI, and {slash} command in the CLI, and it will run it will run it will run a little a little a little Haiku-based model to Haiku-based model to Haiku-based model to explore the explore the explore the codebase, okay? If codebase, okay? If codebase, okay? If you, say, have a you, say, have a you, say, have a ready-made plan, ready-made plan, ready-made plan, right? And you wrote a right? And you wrote a planning document, and you want another another another model developer to check that model developer to check that model developer to check that plan because they plan because they plan because they have different have different have different training data and different training data and different training data and different weights, weights, weights, then fine, you can then fine, you can then fine, you can do that. Rubber Duck is a do that. Rubber Duck is a do that. Rubber Duck is a built-in agent built-in agent built-in agent that checks a plan that checks a plan that checks a plan that was written, that was written, that was written, for example, by Opus, for example, by Opus, for example, by Opus, by running it by running it by running it through GPT or something through GPT or something through GPT or something similar. You can similar. You can similar. You can see that in all of see that in all of see that in all of these models or in these models or in these models or in all of these agents, all of these agents, all of these agents, the model is specified, and that's the model is specified, and that's the model is specified, and that's one of the options you one of the options you one of the options you have when creating your have when creating your have when creating your own agents. You own agents. You own agents. You can always can always can always control the model you control the model you control the model you use, as well use, as well use, as well as the instructions as the instructions as the instructions you give it. So, you give it. So, you give it. So, if these models or these if these models or these if these models or these agents don't do what agents don't do what agents don't do what you need, you need, you need, right? If you have a right? If you have a right? If you have a specific task specific task specific task or audit, or you or audit, or you or audit, or you have have have compliance or compliance or compliance or regulatory requirements to regulatory requirements to meet, and meet, and perhaps there is a tool perhaps there is a tool you want you want you want to run, or a set of to run, or a set of to run, or a set of specific specific specific audits that audits that audits that you need.

you need. you need. You may want to You may want to You may want to write your write your write your agent. And if you want, agent. And if you want, agent. And if you want, you can just you can just you can just use the use the use the agent command and it will agent command and it will agent command and it will guide you through the guide you through the guide you through the workflow for workflow for workflow for adding the agent to adding the agent to adding the agent to your CLI environment. your CLI environment. Here I can choose: do Here I can choose: do I want it at the I want it at the I want it at the project level, like in the Git project level, like in the Git project I project I project I 'm working on, or do I want 'm working on, or do I want 'm working on, or do I want it to be an agent it to be an agent it to be an agent available independently available independently available independently of the project of the project of the project I'm working on, at the I'm working on, at the I'm working on, at the user level, right? So user level, right? So user level, right? So maybe I'll say I'm maybe I'll say I'm maybe I'll say I'm adding this at the adding this at the adding this at the organization level or at organization level or at organization level or at my individual my individual my individual user level. And then user level. And then user level. And then I create it I create it I create it using Copilot. Copilot using Copilot. Copilot using Copilot. Copilot will do all the will do all the will do all the prep work for prep work for prep work for me. So if I me. So if I me. So if I want, I can want, I can want, I can create an agent that create an agent that , say, looks at the , say, looks at the , say, looks at the GitHub changelog and GitHub changelog and GitHub changelog and summarizes all the latest summarizes all the latest summarizes all the latest features or changes that features or changes that features or changes that Copilot has made. So, I Copilot has made. So, I Copilot has made. So, I did this did this did this earlier today and just earlier today and just earlier today and just generated an example generated an example generated an example agent agent agent using this using this using this agent builder agent builder agent builder built into the built into the built into the platform, you know? platform, you know? platform, you know? And it's available in all And it's available in all And it's available in all interfaces. If interfaces. If interfaces. If you have an IDE, you you have an IDE, you you have an IDE, you also have an also have an also have an agent interface there that will agent interface there that will agent interface there that will help you go through the help you go through the help you go through the same same same development process. But the nice thing is that development process. But the nice thing is that development process. But the nice thing is that it it it automatically creates the automatically creates the automatically creates the structure. So from a structure. So from a structure. So from a simple simple simple one-line query one-line query that says, "Hi, I that says, "Hi, I that says, "Hi, I want to go to the want to go to the want to go to the changelog and changelog and changelog and summarize all summarize all summarize all the changes." He will write a the changes." He will write a the changes." He will write a description for me. It will description for me. It will description for me. It will format all format all format all input data. It input data. It input data. It structures the criteria for structures the criteria for structures the criteria for success. He will try success. He will try success. He will try to structure to structure to structure the methodology. He seems to the methodology. He seems to the methodology. He seems to know how to write these know how to write these know how to write these files. So you can files. So you can files. So you can be a little lazy, be a little lazy, be a little lazy, right? You just right? You just right? You just write a general write a general write a general description of what you want description of what you want description of what you want to do. We will fill in to do. We will fill in to do. We will fill in all the details for you. You all the details for you. You all the details for you. You can of course can of course can of course go here and

go here and go here and edit all of this, edit all of this, edit all of this, right? So if you right? So if you right? So if you don't like the don't like the don't like the order in which it order in which it order in which it selects the sources in selects the sources in selects the sources in this case, we this case, we this case, we can change that, can change that, can change that, right? For example, this one right? For example, this one right? For example, this one wants to start wants to start wants to start using the using the using the GitHub roadmap, GitHub roadmap, GitHub roadmap, but maybe you but maybe you but maybe you want it to want it to want it to jump straight to the jump straight to the jump straight to the changelog so it doesn't changelog so it doesn't changelog so it doesn't get points from the get points from the get points from the roadmap. You roadmap. You roadmap. You can just can just can just edit it here, edit it here, edit it here, you know? There is you know? There is you know? There is nothing nothing nothing complicated about this. But it complicated about this. But it complicated about this. But it does all the work of does all the work of does all the work of creating the structure creating the structure creating the structure for you. What's good now is that for you. What's good now is that you've you've you've created an agent that created an agent that created an agent that runs on the system. runs on the system. runs on the system. So, if you have So, if you have So, if you have recurring tasks recurring tasks , you can call , you can call , you can call this agent to this agent to this agent to do them for you, right do them for you, right ? This is a kind of first ? This is a kind of first ? This is a kind of first step. But after that, step. But after that, step. But after that, you wonder, "What should you wonder, "What should you wonder, "What should I do next?" And I I do next?" And I I do next?" And I think the next think the next think the next question is how do we question is how do we question is how do we share this share this share this agent, right? How agent, right? How agent, right? How can other people can other people can other people use it? use it? use it? The easiest way The easiest way The easiest way to do this in Copilot is to do this in Copilot is to create a marketplace to create a marketplace . So a marketplace . So a marketplace is just a is just a is just a repository with a repository with a repository with a little little little extra JSON in extra JSON in extra JSON in one of the directories one of the directories one of the directories that says, "Hello, this is a that says, "Hello, this is a that says, "Hello, this is a marketplace." It marketplace." It marketplace." It provides you with some provides you with some provides you with some basic basic basic package management capabilities package management capabilities package management capabilities within within within the repository. So if the repository. So if the repository. So if I subscribe to the I subscribe to the I subscribe to the marketplace, I can marketplace, I can marketplace, I can update all update all update all agents at the same time. I agents at the same time. I agents at the same time. I can manage their can manage their can manage their versions, etc. I can versions, etc. I can versions, etc. I can inform people inform people inform people about about about security vulnerabilities.

security vulnerabilities. security vulnerabilities. You know, there are built-in You know, there are built-in You know, there are built-in features that features that features that allow me allow me allow me to communicate with to communicate with to communicate with users further down the users further down the chain. So chain. So creating creating creating a marketplace is a marketplace is a marketplace is often one of the first often one of the first often one of the first issues I issues I issues I work on with companies work on with companies work on with companies when discussing when discussing when discussing scaling scaling scaling Copilot use within Copilot use within Copilot use within their organization. There their organization. There their organization. There is often no is often no is often no central central central repository where repository where repository where teams can teams can teams can add agents or add agents or add agents or skills. So, if skills. So, if skills. So, if you don't have a you don't have a you don't have a central repository central repository , you should , you should , you should create one, right? create one, right? create one, right? There is an option There is an option There is an option to synchronize them with a to synchronize them with a to synchronize them with a local local local client client client device and then device and then device and then deploy different deploy different deploy different agents. You can also agents. You can also use a use a specific directory in specific directory in specific directory in your organization your organization your organization or a separate or a separate or a separate repository. If you're repository. If you're repository. If you're on GitHub, I'm at the on GitHub, I'm at the on GitHub, I'm at the organization level now, I organization level now, I organization level now, I went into the repository, went into the repository, went into the repository, and it should be called and it should be called and it should be called GitHub private, right? This is the GitHub private, right? This is the GitHub private, right? This is the naming convention naming convention naming convention we we we use. But use. But use. But if you have a if you have a if you have a private private private GitHub repository, you GitHub repository, you GitHub repository, you can make the can make the can make the agent files public. In agent files public. In agent files public. In this case, I this case, I this case, I have three agents, and you have three agents, and you have three agents, and you see the cloud see the cloud see the cloud agent, I can agent, I can agent, I can call that call that call that particular agent particular agent particular agent on the platform. As I on the platform. As I on the platform. As I showed earlier in showed earlier in showed earlier in the diagram, you can the diagram, you can the diagram, you can interact with all of interact with all of interact with all of them from different them from different them from different surfaces. This is cloud surfaces. This is cloud surfaces. This is cloud interaction, right?

interaction, right? interaction, right? If I want to If I want to If I want to delegate a task to a delegate a task to a delegate a task to a cloud agent cloud agent cloud agent hosted on the hosted on the hosted on the platform, I can platform, I can platform, I can do it here. But the do it here. But the do it here. But the good thing is, this matching bot good thing is, this matching bot good thing is, this matching bot that I'm that I'm that I'm showing in the showing in the showing in the dropdown dropdown dropdown is the same bot that's is the same bot that's is the same bot that's in the repository, right in the repository, right ? So this is a way of ? So this is a way of ? So this is a way of distributing distributing distributing agents among the agents among the agents among the team, right? team, right? team, right? If I write a If I write a If I write a handy agent that handy agent that handy agent that checks the checks the checks the changelog, I can changelog, I can changelog, I can publish it publish it publish it here and share it here and share it here and share it with others, right? This with others, right? This with others, right? This will send it to will send it to each user's IDE or CLI environment each user's IDE or CLI environment and they can and they can and they can call the call the call the same agent that same agent that same agent that I am working on. Now I I am working on. Now I I am working on. Now I have the option to have the option to reuse it, right? reuse it, right? I can publish I can publish I can publish them on the marketplace them on the marketplace them on the marketplace or directly as or directly as or directly as assets within the assets within the assets within the business that business that business that anyone can anyone can anyone can refer to, right? And that's what refer to, right? And that's what refer to, right? And that's what allows me allows me allows me to ensure to ensure to ensure standardization, standardization, standardization, right? If right? If right? If you look at the you look at the you look at the cost model for all of these cost model for all of these cost model for all of these products, products, products, the cheapest way to the cheapest way to the cheapest way to use them is to use them is to use them is to get cash hits, get cash hits, get cash hits, right? You need right? You need right? You need exact matches between exact matches between exact matches between queries on shared queries on shared queries on shared data. Having data. Having data. Having reusable reusable reusable components components components will increase your will increase your will increase your caching level, right? caching level, right? caching level, right? So, if you're So, if you're So, if you're concerned about costs, concerned about costs, concerned about costs, creating creating creating reusable reusable reusable agents and agents and agents and ensuring that ensuring that ensuring that everyone solves everyone solves everyone solves problems the same way, problems the same way, problems the same way, rather than rather than rather than reinventing the wheel every time reinventing the wheel every time reinventing the wheel every time , is a , is a , is a great way to great way to great way to reduce your overall reduce your overall reduce your overall bill and increase bill and increase bill and increase efficiency efficiency efficiency .

. . So, again, So, again, So, again, this private this private this private .github directory is a .github directory is a .github directory is a way to share the way to share the way to share the agent with everyone in agent with everyone in agent with everyone in your your your organization, organization, organization, right? You can right? You can right? You can create a marketplace create a marketplace that works that works that works across different across different across different organizations and even, organizations and even, organizations and even, perhaps, externally. We perhaps, externally. We perhaps, externally. We have a repository have a repository have a repository called awesome co-pilot that called awesome co-pilot that called awesome co-pilot that has a bunch of has a bunch of has a bunch of example agents and example agents and example agents and other tools. other tools. other tools. You can You can You can register it register it register it as a marketplace. This is a as a marketplace. This is a as a marketplace. This is a purely internal purely internal purely internal tool, right? And tool, right? And tool, right? And the administrator can the administrator can the administrator can easily easily easily deploy it. And the last thing deploy it. And the last thing you sometimes want to you sometimes want to you sometimes want to do when do when do when working with an agent is to be working with an agent is to be working with an agent is to be able to able to able to use use use agents in different agents in different agents in different repositories. repositories. repositories. The biggest problem with The biggest problem with The biggest problem with such such such compliance agents, compliance agents, compliance agents, for example, is for example, is for example, is that they always that they always that they always run in the run in the run in the repository where they are repository where they are repository where they are hosted. So if hosted. So if hosted. So if my agents live in my agents live in my agents live in this repository, this repository, this repository, they might not they might not they might not be able to perform be able to perform be able to perform actions in others, right? I actions in others, right? I actions in others, right? I need to run need to run need to run the agent in another the agent in another the agent in another repository so that it repository so that it repository so that it has the appropriate has the appropriate has the appropriate permissions there. In the permissions there. In the permissions there. In the co-pilot application, there is a command co-pilot application, there is a command co-pilot application, there is a command called orchestrate. Orchestrate called orchestrate. Orchestrate is our way of is our way of scaling horizontally across scaling horizontally across repositories. It repositories. It repositories. It allows you to set allows you to set allows you to set a number of tasks in GitHub a number of tasks in GitHub a number of tasks in GitHub Issues as targets for an Issues as targets for an Issues as targets for an agent, and then agent, and then agent, and then launch sub- launch sub- agents to agents to agents to complete all of those complete all of those complete all of those tasks. But tasks. But tasks. But the best thing is that the best thing is that the best thing is that they run they run they run in exactly the in exactly the in exactly the repositories they are supposed to repositories they are supposed to repositories they are supposed to work in. So work in. So , if you want to , if you want to , if you want to perform a task in perform a task in perform a task in a microservice where you a microservice where you a microservice where you are changing the API and because of are changing the API and because of are changing the API and because of that you need to commit that you need to commit that you need to commit the changes elsewhere, the changes elsewhere, the changes elsewhere, Orchestrate will allow you Orchestrate will allow you Orchestrate will allow you to set up multiple to set up multiple to set up multiple agents that agents that will commit changes to will commit changes to different repositories at the same time.

different repositories at the same time. different repositories at the same time. So I can So I can So I can use use use ready-made agents, ready-made agents, ready-made agents, right? If they are right? If they are right? If they are available to me on available to me on available to me on other marketplaces other marketplaces other marketplaces or provided by or provided by or provided by my my my organization's administrator, I can " organization's administrator, I can " call" them to call" them to call" them to perform work in perform work in perform work in individual repositories individual repositories individual repositories and control them using the and control them using the Orchestrate tool. These are actually three levels of These are actually three levels of agent usage agent usage , right? Local , right? Local , right? Local use of agents use of agents use of agents has great practical has great practical has great practical value, right? But value, right? But value, right? But sometimes I don't want sometimes I don't want sometimes I don't want to run the agent to run the agent to run the agent locally. I actually locally. I actually locally. I actually want the agent to be want the agent to be want the agent to be part of the repository part of the repository , as certain , as certain , as certain agents can agents can agents can perform maintenance tasks perform maintenance tasks directly in directly in it. So, I want to it. So, I want to it. So, I want to run them in the run them in the run them in the repository on repository on repository on that that that repository's budget, not repository's budget, not repository's budget, not my own, right? my own, right? my own, right? Or I want to run Or I want to run Or I want to run this using this using the repository's identity instead of mine, the repository's identity instead of mine, right? Not every agent right? Not every agent right? Not every agent should act on my should act on my should act on my behalf. You know, behalf. You know, behalf. You know, maybe this is a maybe this is a maybe this is a maintenance task that maintenance task that maintenance task that needs to be performed on needs to be performed on needs to be performed on behalf of the project. When behalf of the project. When behalf of the project. When we work with this we work with this we work with this model, we have a model, we have a model, we have a framework that you framework that you framework that you can can can use— use— agent agent agent workflows. And agent workflows. And agent workflows. And agent workflows are a workflows are a workflows are a pretty simple pretty simple pretty simple concept. For example, concept. For example, concept. For example, Copilot CLI can Copilot CLI can Copilot CLI can be used in be used in be used in headless mode.

headless mode. headless mode. So I can call up the So I can call up the So I can call up the Copilot CLI, use Copilot CLI, use Copilot CLI, use the -p command, and essentially the -p command, and essentially the -p command, and essentially pass it a single pass it a single pass it a single query to execute. query to execute. query to execute. And if I take this and And if I take this and And if I take this and run it inside GitHub run it inside GitHub run it inside GitHub Actions runner or some Actions runner or some Actions runner or some CI system, I get an CI system, I get an CI system, I get an agent that runs in the agent that runs in the agent that runs in the pipeline for me, pipeline for me, pipeline for me, right? So, I can right? So, I can right? So, I can trigger it trigger it trigger it based on events based on events based on events happening in the happening in the happening in the repository. repository. repository. For example, during a For example, during a For example, during a release I want to update release I want to update release I want to update the documentation. Or the documentation. Or the documentation. Or every week I want to every week I want to every week I want to update my update my update my context files. Or context files. Or , you know, in this , you know, in this , you know, in this model, I have different model, I have different maintenance tasks that I maintenance tasks that I want to do. want to do. want to do. For example, one of the For example, one of the For example, one of the biggest problems biggest problems biggest problems today in today in today in repositories where repositories where repositories where active active active agent development is taking place agent development is taking place is the addition of a large is the addition of a large is the addition of a large amount of “spaghetti amount of “spaghetti code” and code” and code” and duplicate code. duplicate code. duplicate code. Agents are lazy about Agents are lazy about Agents are lazy about implementing implementing implementing features and very features and very features and very lazy about lazy about lazy about namespaces. So namespaces. So namespaces. So they just keep they just keep they just keep creating duplicates creating duplicates creating duplicates within the within the within the codebase. You may codebase. You may codebase. You may need an agent to need an agent to need an agent to review your review your review your codebase, identify codebase, identify codebase, identify opportunities to opportunities to opportunities to remove duplicates, remove duplicates, remove duplicates, and create a and create a and create a series of tasks for you. And if series of tasks for you. And if series of tasks for you. And if you're using you're using you're using this this this agent agent agent workflow framework, what's handy is workflow framework, what's handy is workflow framework, what's handy is that we'll build in that we'll build in that we'll build in triggers with a human in the triggers with a human in the triggers with a human in the loop so you can loop so you can loop so you can review and review and review and decide when decide when decide when to spend money on the to spend money on the to spend money on the next stage of the next stage of the next stage of the process. So, in this process. So, in this process. So, in this thread you can see that thread you can see that thread you can see that my first agent can my first agent can my first agent can run on a run on a run on a cron schedule. He cron schedule. He cron schedule. He is browsing is browsing is browsing the repository. It the repository. It the repository. It identifies, you know, three identifies, you know, three identifies, you know, three points where there is an points where there is an points where there is an opportunity to improve the opportunity to improve the opportunity to improve the code base. Okay, code base. Okay, code base. Okay, great. The developer great. The developer great. The developer reads this, decides whether it's reads this, decides whether it's reads this, decides whether it's worth fixing, worth fixing, worth fixing, and expresses their and expresses their and expresses their decision via a slash

decision via a slash command in a comment command in a comment command in a comment to the task, right? to the task, right? to the task, right? So they go So they go So they go to the task and say, to the task and say, to the task and say, "Okay, move this "Okay, move this "Okay, move this task to the task to the task to the next stage of the next stage of the next stage of the process, okay?" For us process, okay?" For us process, okay?" For us , this is probably the , this is probably the , this is probably the planning stage, right? So planning stage, right? So planning stage, right? So maybe I'll call maybe I'll call maybe I'll call another another another conveyor agent, right? It's the conveyor agent, right? It's the product manager agent who product manager agent who creates the PRD, right? It creates the PRD, right? It creates the PRD, right? It will define the scope of will define the scope of will define the scope of the function for me. He the function for me. He the function for me. He generates a plan. He generates a plan. He generates a plan. He passes it to the Co-pilot passes it to the Co-pilot passes it to the Co-pilot on the platform for on the platform for on the platform for implementation. And then implementation. And then implementation. And then the developer the developer the developer comes back into the loop, comes back into the loop, comes back into the loop, maybe at the maybe at the maybe at the acceptance stage, right? acceptance stage, right? acceptance stage, right? Let's take a look Let's take a look Let's take a look at the code and evaluate whether at the code and evaluate whether at the code and evaluate whether we really want to we really want to we really want to include it in the project. include it in the project. include it in the project. Now I have Now I have Now I have conveyor agents. And what's really conveyor agents. And what's really conveyor agents. And what's really cool about pipeline agents is how cool about pipeline agents is how they they scale. You scale. You scale. You know, the problem with know, the problem with know, the problem with working locally is that working locally is that working locally is that you're you're you're in the same in the same in the same session or on your session or on your session or on your device. If your device. If your device. If your device turns off device turns off device turns off or you lose or you lose or you lose network, you know, your network, you know, your network, you know, your workflow workflow workflow is interrupted. They is interrupted. They is interrupted. They run on a run on a run on a platform, so they platform, so they platform, so they work independently work independently work independently of you. If you of you. If you of you. If you want to see a want to see a want to see a maximalist maximalist maximalist approach to how this approach to how this approach to how this works, the GitHub AW project, works, the GitHub AW project, works, the GitHub AW project, the one that is the framework the one that is the framework the one that is the framework for building all for building all for building all these agents, is these agents, is these agents, is built entirely by agents built entirely by agents . And this team has an . And this team has an . And this team has an internal goal that internal goal that internal goal that they will they will they will only develop on only develop on only develop on their mobile their mobile their mobile devices, because devices, because devices, because they they they interact with this interact with this interact with this application entirely through slash application entirely through slash commands, right? So, commands, right? So, commands, right? So, to give you an idea to give you an idea to give you an idea of ​​what a of ​​what a of ​​what a fully fully fully agent-driven project looks like, agent-driven project looks like, agent-driven project looks like, this is our framework, right?

this is our framework, right? And it has, I don't know, 200 And it has, I don't know, 200 agents that agents that agents that do different do different do different parts of the pipeline, right parts of the pipeline, right ? And they all follow a ? And they all follow a ? And they all follow a similar similar similar workflow that we workflow that we workflow that we just showed on just showed on just showed on that slide, right? This that slide, right? This that slide, right? This architecture is architecture is architecture is reusable. So reusable. So reusable. So let me let me let me try to pick try to pick try to pick one of them. I'm picking a bit at one of them. I'm picking a bit at one of them. I'm picking a bit at random, but random, but random, but let's see let's see let's see what he does, shall we? what he does, shall we? what he does, shall we? So, this is a review of the So, this is a review of the other agents' daily reports, right? This is a other agents' daily reports, right? This is a meta-agent that meta-agent that meta-agent that oversees things. oversees things. oversees things. It runs on a It runs on a It runs on a daily schedule and daily schedule and daily schedule and has its own agent.md file has its own agent.md file has its own agent.md file that runs here, right? that runs here, right? that runs here, right? Every time it Every time it Every time it runs, it runs, it runs, it creates a creates a creates a markdown report or markdown report or markdown report or some action or some action or some action or input for another input for another input for another agent, and we just agent, and we just agent, and we just keep keep keep firing it. You are firing it. You are firing it. You are free to view free to view free to view this. There are hundreds of people here this. There are hundreds of people here this. There are hundreds of people here who are creating this who are creating this who are creating this project. GitHub also has a project. GitHub also has a project. GitHub also has a bunch of those that we've bunch of those that we've bunch of those that we've merged together. So, merged together. So, merged together. So, within our within our within our GitHub Next organization, there is a GitHub Next organization, there is a GitHub Next organization, there is a project called project called project called Agentics. Agentics has Agentics. Agentics has Agentics. Agentics has about 30 different about 30 different about 30 different pipeline agents that pipeline agents that pipeline agents that you may want to you may want to you may want to just grab just grab just grab and run and run and run in your project.

in your project. in your project. Some of them Some of them Some of them will perform the will perform the will perform the code deduplication code deduplication code deduplication we talked about. we talked about. we talked about. Some of them are different Some of them are different Some of them are different types of reviewers, types of reviewers, types of reviewers, so if you so if you so if you need a QA reviewer need a QA reviewer need a QA reviewer or, let's say, you or, let's say, you or, let's say, you want one that's a want one that's a want one that's a joker, there are all kinds of joker, there are all kinds of joker, there are all kinds of different different example agents that you can example agents that you can example agents that you can download and download and download and add to your pipeline. But add to your pipeline. But add to your pipeline. But the nice thing is that it the nice thing is that it the nice thing is that it takes the takes the takes the workload off workload off workload off your computer. And I your computer. And I your computer. And I think when we think when we think when we go back to go back to go back to thinking about costs, thinking about costs, thinking about costs, taking the taking the taking the load off your load off your load off your machine is key to machine is key to machine is key to optimizing costs, optimizing costs, optimizing costs, right? If I right? If I right? If I ask everyone in my ask everyone in my ask everyone in my organization organization organization to work with to work with to work with AI and tell them AI and tell them AI and tell them to use it to use it , but I don't see how they , but I don't see how they , but I don't see how they actually work with the actually work with the actually work with the tool. I tool. I tool. I don't have any room for don't have any room for don't have any room for optimization, do I optimization, do I optimization, do I ? I can't ? I can't ? I can't really see really see really see if the agents they if the agents they if the agents they use are use are use are too verbose too verbose too verbose or if they even or if they even or if they even get the job done. I get the job done. I get the job done. I can't tell when the can't tell when the tool calls are happening, whether tool calls are happening, whether they are successful or not. I they are successful or not. I they are successful or not. I can't tell if can't tell if can't tell if they're using they're using they're using agent logic for agent logic for agent logic for everything, or if everything, or if everything, or if they're, you know, they're, you know, they're, you know, creating creating creating little bash scripts little bash scripts little bash scripts to actually to actually to actually do the different do the different do the different search functions and so on. There search functions and so on. There search functions and so on. There are a lot of are a lot of are a lot of opportunities for opportunities for opportunities for optimization here, but if optimization here, but if optimization here, but if I don't see it, I can't I don't see it, I can't optimize anything. So, optimize anything. So, by moving the by moving the by moving the workload from workload from workload from the laptop to the CI, I can see the laptop to the CI, I can see the laptop to the CI, I can see what's happening, what's happening, what's happening, right? I have every right? I have every right? I have every single agent up single agent up single agent up and running and and running and and running and equipped with equipped with equipped with Otel tools. So Otel tools. So Otel tools. So I can go into SIEM or I can go into SIEM or I can go into SIEM or some some some logging platform and logging platform and logging platform and see: how is see: how is see: how is my my my code deduplication agent performing across code deduplication agent performing across code deduplication agent performing across different repositories?

different repositories? different repositories? How much is this How much is this How much is this costing me, right? I can costing me, right? I can costing me, right? I can analyze the average analyze the average analyze the average token usage token usage . I can also do . I can also do . I can also do things like, "Hey, things like, "Hey, things like, "Hey, every time an agent every time an agent every time an agent exceeds their P90 on exceeds their P90 on exceeds their P90 on expected expected expected work cost, investigate why work cost, investigate why work cost, investigate why they exceeded the threshold," they exceeded the threshold," they exceeded the threshold," right? I want it to right? I want it to right? I want it to tell me what tell me what tell me what debugging debugging debugging information I can information I can information I can use for use for use for further evaluation or further evaluation or further evaluation or improvement, right? improvement, right? improvement, right? Or if I see that the Or if I see that the Or if I see that the tool call is tool call is tool call is failing all the time, failing all the time, failing all the time, maybe I want maybe I want maybe I want to AB to AB test different test different test different inputs or inputs or inputs or different models to different models to different models to see if see if see if they give the right they give the right they give the right result so that I can later result so that I can later result so that I can later choose a cheaper choose a cheaper choose a cheaper model, right? All of this model, right? All of this model, right? All of this becomes possible once becomes possible once becomes possible once I add I add I add agents to the pipeline. agents to the pipeline. agents to the pipeline. Anyway, that Anyway, that Anyway, that was a quick and concise was a quick and concise was a quick and concise overview of the agents on the overview of the agents on the overview of the agents on the platform, right? We platform, right? We platform, right? We discussed creating an discussed creating an discussed creating an agent locally, agent locally, agent locally, exchanging agents across the exchanging agents across the exchanging agents across the platform, and then platform, and then platform, and then moving those moving those moving those agents into the pipeline. agents into the pipeline. agents into the pipeline. If you want If you want If you want to discuss this to discuss this to discuss this in more detail, I'll be in more detail, I'll be in more detail, I'll be behind you, come up to behind you, come up to behind you, come up to me or add me me or add me me or add me on LinkedIn and write there. on LinkedIn and write there. on LinkedIn and write there. I would be happy I would be happy I would be happy to continue the conversation. I to continue the conversation. I to continue the conversation. I work at GitHub in a work at GitHub in a work at GitHub in a remote position. So remote position. So remote position. So if you know your if you know your if you know your GitHub AE and want to GitHub AE and want to GitHub AE and want to chat with chat with chat with me, feel free to me, feel free to me, feel free to reach out to them.

reach out to them. reach out to them. We will be able to get in touch We will be able to get in touch We will be able to get in touch and also quickly and also quickly and also quickly discuss everything. discuss everything. discuss everything. Thank you very much for your time Thank you very much for your time Thank you very much for your time today.