Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI

Congratulations. Please Congratulations. Please take your seats if take your seats if take your seats if you haven't already you haven't already you haven't already . My name is . My name is . My name is Viem. With us are Jangal, Viem. With us are Jangal, Viem. With us are Jangal, Drew and Dax. Also Drew and Dax. Also Drew and Dax. Also , a huge thank you to , a huge thank you to , a huge thank you to Sean and the AI ​​Engineers World Fair Sean and the AI ​​Engineers World Fair Sean and the AI ​​Engineers World Fair for hosting us for hosting us . There's a . There's a . There's a great AI community here, great AI community here, great AI community here, and we're very, very excited and we're very, very excited and we're very, very excited to be with you. If you have to be with you. If you have to be with you. If you have n't managed to get everything n't managed to get everything n't managed to get everything set up yet, don't set up yet, don't set up yet, don't worry. We'll have worry. We'll have worry. We'll have plenty of time plenty of time plenty of time to do that to do that to do that later. Jangal will later. Jangal will later. Jangal will briefly tell briefly tell briefly tell us about the basics of Auto us about the basics of Auto us about the basics of Auto Research. Research. OK. If any of OK. If any of you are not yet familiar with you are not yet familiar with you are not yet familiar with Auto Research, this is just a Auto Research, this is just a Auto Research, this is just a quick overview. So, quick overview. So, quick overview. So, Auto Research is essentially a Auto Research is essentially a Auto Research is essentially a cycle of trial and error. cycle of trial and error. You provide the language You provide the language model with your code and a model with your code and a model with your code and a script for evaluation, script for evaluation, script for evaluation, which can be repeated. which can be repeated. She continues to She continues to try ideas and try ideas and try ideas and test them test them test them through evaluation.

through evaluation. through evaluation. She keeps ideas She keeps ideas She keeps ideas that improve the that improve the that improve the outcome and discards outcome and discards outcome and discards everything else. Examples of everything else. Examples of everything else. Examples of metrics are accuracy, metrics are accuracy, metrics are accuracy, speed, and cost. A speed, and cost. A similar cycle has existed in similar cycle has existed in academic circles academic circles academic circles for about two for about two for about two years. But years. But years. But it really went viral it really went viral it really went viral this year when this year when this year when Kapati wrote about Kapati wrote about Kapati wrote about it in his it in his model training scripts. And model training scripts. And it turned out that it turned out that it turned out that it worked very it worked very it worked very well for him. In just a few well for him. In just a few well for him. In just a few weeks, weeks, weeks, Auto Research's scope Auto Research's scope Auto Research's scope expanded from expanded from expanded from training to everything from training to everything from training to everything from backend engineering to backend engineering to backend engineering to prompt engineering to prompt engineering to prompt engineering to agent engineering to agent engineering to model inference acceleration, and it model inference acceleration, and it quickly made its way into quickly made its way into quickly made its way into production as well. production as well. For example, it For example, it reduced the latency of Shopify's CI/CD reduced the latency of Shopify's CI/CD reduced the latency of Shopify's CI/CD pipeline, and also pipeline, and also pipeline, and also improved their improved their improved their browser agent browser agent browser agent . In April, . In April, . In April, our our our experimental experimental experimental research agent research agent research agent Aiden even won an Aiden even won an Aiden even won an OpenAI competition OpenAI competition OpenAI competition called "Prompt Golf," called "Prompt Golf," called "Prompt Golf," which we'll be which we'll be which we'll be covering in more detail soon covering in more detail soon , and I'll also have , and I'll also have , and I'll also have a talk about that on a talk about that on a talk about that on Wednesday. So, Auto Research

Wednesday. So, Auto Research Wednesday. So, Auto Research can be very can be very can be very powerful, but it can be powerful, but it can be powerful, but it can be difficult to master. With a difficult to master. With a difficult to master. With a regular regular regular encoding agent, you encoding agent, you encoding agent, you actually control the actually control the actually control the agent every agent every agent every step of the way. But for Auto Research, the step of the way. But for Auto Research, the step of the way. But for Auto Research, the agent works in a loop on its agent works in a loop on its agent works in a loop on its own. Therefore, it own. Therefore, it own. Therefore, it is a much more is a much more is a much more powerful multiplier powerful multiplier powerful multiplier for your prompt and for your prompt and for your prompt and settings. settings. Set it to the Set it to the right valuation and right valuation and right valuation and you let it you let it you let it do all the hard do all the hard do all the hard work for you, otherwise work for you, otherwise work for you, otherwise you'll just burn all you'll just burn all you'll just burn all your tokens. your tokens. We hope this We hope this workshop will give you a workshop will give you a workshop will give you a good starting good starting good starting point to learn point to learn point to learn about Auto Research and about Auto Research and about Auto Research and apply it to apply it to apply it to your own work. Thank you your own work. Thank you , we're going back to , we're going back to , we're going back to Viem. Viem. Thank you, Jangal. So, Thank you, Jangal. So, this is how we'll spend the this is how we'll spend the this is how we'll spend the next two hours. next two hours. next two hours. Dex will take you Dex will take you Dex will take you inside Parameter Golf, a inside Parameter Golf, a inside Parameter Golf, a challenge challenge challenge our agent participated in our agent participated in , tell you what the , tell you what the , tell you what the competition is all about, and competition is all about, and competition is all about, and how our agent became the how our agent became the how our agent became the most influential most influential most influential participant. Then it's participant. Then it's participant. Then it's your turn. We'll your turn. We'll your turn. We'll help you help you help you set everything up and set everything up and set everything up and run run run auto-research on auto-research on auto-research on your laptop. While your laptop. While your laptop. While auto-research auto-research auto-research is running in is running in is running in the background, we the background, we the background, we will examine the basics of the will examine the basics of the auto-research cycle and the auto-research cycle and the elements that elements that elements that determine the determine the determine the agent's actions. We agent's actions. We agent's actions. We will delve into the topic and will delve into the topic and will delve into the topic and provide a provide a provide a demo where you demo where you demo where you can launch, can launch, can launch, direct, and direct, and direct, and verify your verify your verify your agent's actions in real

agent's actions in real agent's actions in real time. At the end, the time. At the end, the most interesting thing awaits us—a live most interesting thing awaits us—a live competition. You competition. You competition. You will use everything you will use everything you will use everything you learned today, learned today, learned today, direct direct direct your auto-research to a your auto-research to a your auto-research to a familiar game, and familiar game, and familiar game, and compete to see whose compete to see whose compete to see whose agent will score agent will score agent will score the most points, and the the most points, and the the most points, and the winner will receive a winner will receive a winner will receive a cool prize. A little cool prize. A little cool prize. A little more about who we are more about who we are more about who we are . We have . We have . We have been working on been working on been working on auto research for quite some time. auto research for quite some time. auto research for quite some time. Before that, in April 2024 Before that, in April 2024 Before that, in April 2024 , we released AID — an autonomous , we released AID — an autonomous machine learning engineering agent. The machine learning engineering agent. The basic idea basic idea basic idea was to was to was to give it a give it a give it a machine learning task, machine learning task, machine learning task, let it write code, let it write code, let it write code, run run run experiments, experiments, experiments, read the results, read the results, read the results, and constantly and constantly and constantly improve. improve. improve. When OpenAI created the When OpenAI created the When OpenAI created the MLE-bench, AID was one of the MLE-bench, AID was one of the MLE-bench, AID was one of the first agents first agents first agents they they they independently evaluated, and in independently evaluated, and in independently evaluated, and in the results, it the results, it the results, it scored four times scored four times scored four times higher than the next higher than the next higher than the next best agent. best agent. Later, as we Later, as we mentioned, we mentioned, we mentioned, we directed our directed our directed our agent to Parameter Golf, and agent to Parameter Golf, and agent to Parameter Golf, and among over 1,000 among over 1,000 among over 1,000 researchers, he became the researchers, he became the researchers, he became the number one participant.

number one participant. number one participant. I'll hand over to Dex I'll hand over to Dex , who will tell you more , who will tell you more , who will tell you more about this competition. about this competition. Greetings to everyone. My Greetings to everyone. My name is Dex. I name is Dex. I name is Dex. I will talk about Parameter Golf. will talk about Parameter Golf. Parameter Golf is Parameter Golf is OpenAI's OpenAI's OpenAI's model building challenge. The goal is model building challenge. The goal is to train the best to train the best to train the best language model under two language model under two language model under two hard constraints. hard constraints. First: all artifacts, First: all artifacts, including including including model weights and model weights and model weights and training code, must training code, must training code, must fit within 16 fit within 16 fit within 16 megabytes. And megabytes. And megabytes. And the training should the training should the training should be completed within be completed within be completed within 10 minutes. Works 10 minutes. Works 10 minutes. Works are evaluated by are evaluated by are evaluated by bits per bits per bits per byte (BPB) on the byte (BPB) on the FineWeb validation set. And the less, the FineWeb validation set. And the less, the better. OpenAI provided a better. OpenAI provided a better. OpenAI provided a baseline dataset and baseline dataset and baseline dataset and evaluation script evaluation script evaluation script so that participants could so that participants could so that participants could fork the repository fork the repository , improve the model, and , improve the model, and , improve the model, and submit results submit results submit results via GitHub. So everything is via GitHub. So everything is via GitHub. So everything is open source. They open source. They open source. They also allocated a also allocated a also allocated a million dollars in million dollars in million dollars in computing computing computing credits and credits and credits and vetted every vetted every vetted every request submitted. Over the 44 request submitted. Over the 44 request submitted. Over the 44 days of the competition, 1,016 days of the competition, 1,016 days of the competition, 1,016 researchers submitted researchers submitted researchers submitted 2,048 pool requests, 2,048 pool requests, 2,048 pool requests, using using using approximately approximately approximately $250,000 in $250,000 in $250,000 in GPU computing. The GPU computing. The final final leaderboard is here.

leaderboard is here. leaderboard is here. Only 47 Only 47 Only 47 merge requests (PRs) were merge requests (PRs) were merge requests (PRs) were accepted as entries accepted as entries accepted as entries from 31 participants. For from 31 participants. For from 31 participants. For this competition, we this competition, we this competition, we created Aiden—a created Aiden—a system of many system of many self-improving agents self-improving agents based on cloud based on cloud based on cloud technologies and technologies and technologies and a family of GPT models a family of GPT models a family of GPT models that work that work that work continuously to continuously to continuously to optimize a single optimize a single optimize a single metric. In " metric. In " parametric parametric parametric golf," Aiden golf," Aiden golf," Aiden automated the full automated the full automated the full cycle of research. It cycle of research. It cycle of research. It downloads public downloads public downloads public articles from the archive, articles from the archive, articles from the archive, effective implementations effective implementations effective implementations from GitHub, and its own from GitHub, and its own from GitHub, and its own past journals, then past journals, then past journals, then suggests new suggests new suggests new ideas, tests them, ideas, tests them, ideas, tests them, checks for checks for checks for correctness, leaves correctness, leaves correctness, leaves successful ones, writes code, and successful ones, writes code, and successful ones, writes code, and conducts conducts conducts experiments. When experiments. When experiments. When the results pass the results pass the results pass quality review, he quality review, he quality review, he creates and submits a creates and submits a creates and submits a request (PR) for review. And request (PR) for review. And request (PR) for review. And the results are simply the results are simply the results are simply impressive. Aiden impressive. Aiden impressive. Aiden achieved seven achieved seven achieved seven accepted entries on the accepted entries on the accepted entries on the leaderboard. That's leaderboard. That's leaderboard. That's twice as much as the twice as much as the twice as much as the next next next best best human participant. He also had human participant. He also had human participant. He also had the highest OPR H-index in the the highest OPR H-index in the the highest OPR H-index in the entire competition, entire competition, entire competition, meaning Aiden wasn't meaning Aiden wasn't meaning Aiden wasn't just setting just setting just setting accepted records. He accepted records. He accepted records. He published published published results that other results that other results that other participants could participants could participants could use in use in use in further work. 435 further work. 435 further work. 435 subsequent queries subsequent queries subsequent queries were based on were based on were based on Aiden's work. And he achieved Aiden's work. And he achieved Aiden's work. And he achieved this with a meager this with a meager this with a meager budget.

budget. budget. Using only 4 Using only 4 % of the community's total % of the community's total % of the community's total computing computing computing power power power , it , it , it provided 15% of all provided 15% of all provided 15% of all accepted entries. This is accepted entries. This is accepted entries. This is 3.7 times more efficient 3.7 times more efficient 3.7 times more efficient than the than the than the industry average. This industry average. This industry average. This concludes the story concludes the story concludes the story of how Aiden became a of how Aiden became a of how Aiden became a major participant in the major participant in the major participant in the Open AI competition. Thank you. Open AI competition. Thank you. Steeply. So, we're Steeply. So, we're going to run going to run going to run an auto-study on a an auto-study on a an auto-study on a version of the same " version of the same " parametric parametric parametric golf" that golf" that golf" that Dex just told you about. Dex just told you about. Dex just told you about. For further For further For further use, we use, we use, we will call this mini PG, will call this mini PG, will call this mini PG, which stands for "mini- which stands for "mini- parametric golf". parametric golf". parametric golf". As Dex mentioned, the As Dex mentioned, the As Dex mentioned, the actual actual actual parametric parametric parametric golf competition was very golf competition was very golf competition was very challenging. Each challenging. Each challenging. Each experiment experiment experiment involved training a involved training a involved training a language model on a language model on a language model on a huge huge huge dataset, which is quite dataset, which is quite dataset, which is quite appropriate when appropriate when appropriate when the competition lasts 4 the competition lasts 4 the competition lasts 4 weeks, but at weeks, but at weeks, but at today's today's today's workshop you would workshop you would workshop you would never see the never see the never see the result of your result of your result of your auto-research. So auto-research. So auto-research. So think of mini PG as a think of mini PG as a think of mini PG as a scaled-down version of that scaled-down version of that scaled-down version of that challenge. You challenge. You challenge. You will still train a will still train a will still train a real language real language real language model, rate it model, rate it model, rate it on the same fixed on the same fixed on the same fixed scale, and try to scale, and try to scale, and try to lower the same lower the same lower the same metric—val BBP, which metric—val BBP, which metric—val BBP, which essentially measures essentially measures essentially measures how well the how well the how well the model predicts model predicts model predicts text it has not yet text it has not yet text it has not yet seen. But we have significantly seen. But we have significantly seen. But we have significantly reduced the scale. We reduced the scale. We reduced the scale. We use a use a use a tiny tiny tiny dataset instead of FineWeb, and dataset instead of FineWeb, and dataset instead of FineWeb, and training and evaluation training and evaluation training and evaluation only take a few only take a few only take a few minutes. So you minutes. So you minutes. So you can run the can run the can run the loop and see loop and see loop and see some real some real some real steps steps steps live. So, before we

live. So, before we live. So, before we move on to move on to move on to the instructions on the the instructions on the the instructions on the next slide, next slide, next slide, a little orientation. a little orientation. a little orientation. You will independently You will independently You will independently run run run auto-research on auto-research on auto-research on mini-parametric mini-parametric mini-parametric golf. You will need to golf. You will need to golf. You will need to register register register on both platforms on both platforms on both platforms if you haven't already if you haven't already if you haven't already . The first is . The first is . The first is Weasel, our Weasel, our Weasel, our automated automated automated research agent that research agent that research agent that performs the search, and performs the search, and performs the search, and the second is Modal, the the second is Modal, the the second is Modal, the computing computing computing power to power to power to train and train and train and evaluate your evaluate your evaluate your experiments. experiments. experiments. Free credits, Free credits, Free credits, thanks Modal. So, you thanks Modal. So, you thanks Modal. So, you clone our clone our clone our parameter golf repository parameter golf repository parameter golf repository and push it to and push it to and push it to your your your programming agent. Give programming agent. Give programming agent. Give it the command it the command it the command to start running. It will to start running. It will to start running. It will take over take over take over the settings and the settings and the settings and start the cycle. Once start the cycle. Once start the cycle. Once everything is working in the everything is working in the everything is working in the background, you background, you background, you can can can activate it, and we'll be activate it, and we'll be activate it, and we'll be back in 15-20 back in 15-20 back in 15-20 minutes to sort out the minutes to sort out the minutes to sort out the details. Instructions details. Instructions details. Instructions here. Sign up for here. Sign up for here. Sign up for Weasel at dashboard.weasel.ai. And Weasel at dashboard.weasel.ai. And Weasel at dashboard.weasel.ai. And for Modal—at modal.com/sign-up. There for Modal—at modal.com/sign-up. There for Modal—at modal.com/sign-up. There 's a code there to 's a code there to 's a code there to get your get your get your Modal credits. If Modal credits. If Modal credits. If you already have a you already have a you already have a Modal account, you can Modal account, you can Modal account, you can go directly to go directly to go directly to modal.com/credits and enter modal.com/credits and enter modal.com/credits and enter your code there.

your code there. After completing these two After completing these two steps, clone steps, clone steps, clone the repository the repository the repository shown here, shown here, shown here, tell the agent tell the agent tell the agent to configure the mini PG from the to configure the mini PG from the to configure the mini PG from the read me file, and read me file, and read me file, and start the process. start the process. If you have any If you have any problems or problems or problems or questions, questions, questions, scan the QR code and scan the QR code and scan the QR code and write there, we write there, we write there, we will follow up on it will follow up on it , or just , or just , or just raise your hand and we will raise your hand and we will raise your hand and we will come over. come over. come over. We'll be back in We'll be back in We'll be back in about 10-15 about 10-15 about 10-15 minutes. minutes. minutes. Thank you, friends. Thank you, friends. Thank you, friends. OK. Sorry for the OK. Sorry for the internet problems. I think they internet problems. I think they are working on this, but for are working on this, but for are working on this, but for now it would be now it would be now it would be nice to understand the nice to understand the nice to understand the anatomy of the anatomy of the anatomy of the automated automated automated research cycle. We will research cycle. We will research cycle. We will do this do this do this using Weasel as an using Weasel as an auto-research system and mini PG auto-research system and mini PG as a task. as a task. In fact, the In fact, the approach we have approach we have approach we have here is quite here is quite here is quite universal. universal. universal. Most Most Most auto-research systems auto-research systems auto-research systems force you to ask force you to ask force you to ask similar questions and similar questions and similar questions and think about the same think about the same think about the same things. Okay. So, the things. Okay. So, the things. Okay. So, the first thing is the source first thing is the source first thing is the source file. This is what the agent file. This is what the agent file. This is what the agent has to actually has to actually has to actually change. It change. It change. It defines the defines the defines the search space and places where search space and places where search space and places where changes can be made.

changes can be made. changes can be made. This can be a single This can be a single This can be a single file. Or it could be file. Or it could be file. Or it could be multiple files. In multiple files. In multiple files. In the case of MiniPG, it is a the case of MiniPG, it is a the case of MiniPG, it is a single file called single file called single file called solution.py. Solution.py essentially solution.py. Solution.py essentially solution.py. Solution.py essentially defines the defines the defines the model configuration model configuration model configuration and the learning dynamics and the learning dynamics and the learning dynamics that will occur that will occur that will occur during during during model training. And the second thing, which, in model training. And the second thing, which, in model training. And the second thing, which, in my opinion, is perhaps the my opinion, is perhaps the my opinion, is perhaps the most important most important most important part of the part of the part of the auto-research system, is auto-research system, is auto-research system, is evaluation. So, you evaluation. So, you evaluation. So, you need a very need a very need a very simple way simple way simple way to let your to let your auto-research system auto-research system evaluate every evaluate every evaluate every solution it solution it solution it generates. Um, and you generates. Um, and you generates. Um, and you want it to want it to want it to output a single output a single output a single scalar metric. And scalar metric. And scalar metric. And you want the you want the you want the evaluation to be evaluation to be evaluation to be quick. You want it to quick. You want it to quick. You want it to be cheap be cheap . You want it to . You want it to . You want it to be low-noise be low-noise be low-noise because that essentially because that essentially because that essentially allows you to do allows you to do allows you to do more steps if more steps if more steps if you're on a budget you're on a budget you're on a budget or time constraint. Um, and if there's or time constraint. Um, and if there's or time constraint. Um, and if there's little noise, both you and the little noise, both you and the little noise, both you and the agent will feel more agent will feel more agent will feel more confident confident confident rising to the rising to the rising to the top in whatever top in whatever top in whatever task you're task you're task you're working on. So, one thing to working on. So, one thing to working on. So, one thing to remember: remember: remember: agents love to agents love to agents love to hack the hack the hack the reward system. They reward system. They reward system. They love to find love to find love to find holes in your holes in your holes in your assessment protocol assessment protocol , holes in your , holes in your , holes in your benchmark. So you benchmark. So you benchmark. So you really want to really want to really want to make your make your make your grading system as grading system as grading system as reliable as possible. The next reliable as possible. The next reliable as possible. The next two points are the two points are the two points are the metric and the goal. So metric and the goal. So metric and the goal. So you should ask you should ask you should ask yourself two questions.

yourself two questions. yourself two questions. Um, what exactly do I want Um, what exactly do I want Um, what exactly do I want to improve? And is it better to improve? And is it better to improve? And is it better when the indicator is higher when the indicator is higher when the indicator is higher or lower? Um, in or lower? Um, in or lower? Um, in our case for our case for our case for MiniPG, you want to MiniPG, you want to MiniPG, you want to minimize the minimize the minimize the validation BPP. validation BPP. That is, you actually That is, you actually want to find a model want to find a model want to find a model that better predicts the that better predicts the that better predicts the unknown text. The unknown text. The next point is the next point is the number of steps number of steps number of steps during which you during which you during which you run the run the run the auto-research system. This is auto-research system. This is auto-research system. This is a bit tricky because a bit tricky because a bit tricky because it's hard it's hard it's hard to know in advance how many steps to know in advance how many steps to know in advance how many steps your system should allocate your system should allocate your system should allocate . We chose 20 in . We chose 20 in . We chose 20 in this case, simply this case, simply this case, simply because we want to give because we want to give because we want to give agents enough agents enough agents enough time to explore different time to explore different time to explore different ideas so that you can ideas so that you can ideas so that you can see exactly what the agent see exactly what the agent see exactly what the agent is exploring, what works, is exploring, what works, is exploring, what works, and what doesn't. Um, but and what doesn't. Um, but and what doesn't. Um, but in reality, the more in reality, the more in reality, the more ideas are ideas are ideas are tested, the tested, the tested, the more breakthroughs it more breakthroughs it more breakthroughs it usually brings. usually brings. So this brings us back So this brings us back to what I to what I to what I was saying earlier: if was saying earlier: if was saying earlier: if you have a quick or cheap you have a quick or cheap you have a quick or cheap estimate, then given estimate, then given estimate, then given limited time or limited time or limited time or money, you can money, you can money, you can do more do more do more steps. Um, and that's why steps. Um, and that's why steps. Um, and that's why evaluation is evaluation is evaluation is always something that's extremely always something that's extremely always something that's extremely important to consider.

important to consider. OK. You also OK. You also want to limit the time want to limit the time want to limit the time your your your assessment takes, because a lot assessment takes, because a lot assessment takes, because a lot can go wrong. can go wrong. Loops can Loops can hang, hang, hang, equipment can equipment can equipment can fail, sometimes an fail, sometimes an fail, sometimes an agent creates very agent creates very agent creates very bad code that causes the bad code that causes the bad code that causes the entire entire entire system to freeze. Therefore, you system to freeze. Therefore, you system to freeze. Therefore, you want to set want to set want to set a limit for the a limit for the a limit for the evaluation process itself. evaluation process itself. evaluation process itself. Um, in our case Um, in our case Um, in our case we set it to we set it to we set it to 210 seconds. And the 210 seconds. And the 210 seconds. And the reason for this is as reason for this is as reason for this is as follows: we follows: we follows: we allocate 60 seconds to allocate 60 seconds to allocate 60 seconds to train the model, 60 train the model, 60 train the model, 60 seconds to seconds to seconds to test it on test it on test it on unknown text, and unknown text, and unknown text, and another 90 seconds as a buffer another 90 seconds as a buffer another 90 seconds as a buffer so that the model has time to so that the model has time to so that the model has time to connect the GPU for connect the GPU for connect the GPU for training, training, training, set up for set up for set up for testing, and testing, and testing, and take into account other take into account other take into account other technical technical technical time costs. And the last thing is the time costs. And the last thing is the time costs. And the last thing is the instructions you instructions you instructions you give to the agent at the very give to the agent at the very give to the agent at the very beginning of beginning of beginning of optimization. So, that's how optimization. So, that's how optimization. So, that's how you manage the you manage the you manage the agent from the very agent from the very agent from the very beginning. You should beginning. You should beginning. You should think of this as think of this as think of this as experience: you've probably experience: you've probably experience: you've probably done done done experiments before, experiments before, experiments before, have a lot of have a lot of have a lot of professional knowledge and professional knowledge and professional knowledge and human experience, human experience, human experience, so you may have so you may have so you may have ideas about what ideas about what ideas about what works, what's worth works, what's worth works, what's worth trying, and what trying, and what trying, and what might not might not might not work. So this is where work. So this is where work. So this is where you can provide you can provide you can provide such guidance. In such guidance. In such guidance. In the case of WeCo, you the case of WeCo, you the case of WeCo, you can simply can simply can simply specify a specify a specify a markdown file that markdown file that markdown file that contains all your

contains all your contains all your information and information and information and context. The agent will be able to context. The agent will be able to context. The agent will be able to take advantage of this, and take advantage of this, and take advantage of this, and it will give him the it will give him the it will give him the best chance of best chance of best chance of success. In the case of MiniPG, success. In the case of MiniPG, success. In the case of MiniPG, these could be certain these could be certain these could be certain hard constraints hard constraints hard constraints or ideas that or ideas that or ideas that work well, such as a focus work well, such as a focus work well, such as a focus on compression, different on compression, different on compression, different quantization methods, quantization methods, quantization methods, or even or even or even sharing weights across sharing weights across sharing weights across different blocks of the model. different blocks of the model. We will demonstrate We will demonstrate how to get started with how to get started with how to get started with auto-research, auto-research, auto-research, taking into account taking into account taking into account internet problems. But internet problems. But internet problems. But before that, a little before that, a little before that, a little more about what more about what more about what we think makes we think makes we think makes auto-research auto-research auto-research truly industrial- truly industrial- truly industrial- grade. First, it is the grade. First, it is the grade. First, it is the structure. We structure. We structure. We use a use a use a tree-like search tree-like search , where the agent shows , where the agent shows , where the agent shows each attempt it has each attempt it has each attempt it has made, so that ideas, as made, so that ideas, as made, so that ideas, as Drew mentioned, can Drew mentioned, can Drew mentioned, can accumulate, accumulate, accumulate, instead of instead of instead of starting from scratch.

starting from scratch. The second is The second is observability. observability. observability. You can open You can open You can open any attempt, any attempt, any attempt, read the exact read the exact read the exact code changes code changes code changes behind the behind the behind the result, and result, and result, and understand why the agent understand why the agent understand why the agent is doing what it is doing. And is doing what it is doing. And is doing what it is doing. And the third is the third is the third is manageability. You manageability. You manageability. You feel in control, feel in control, feel in control, you can add you can add you can add your own ideas and knowledge your own ideas and knowledge your own ideas and knowledge as the loop runs, as the loop runs, as the loop runs, so the agent tries so the agent tries so the agent tries new directions if new directions if new directions if it gets stuck. A little it gets stuck. A little it gets stuck. A little more about more about more about tree-like search tree-like search . We'll . We'll . We'll demonstrate this demonstrate this demonstrate this in our in our in our dashboard soon, but dashboard soon, but dashboard soon, but in general, in general, in general, tree-based tree-based tree-based auto-exploration auto-exploration auto-exploration looks at a problem looks at a problem looks at a problem as a structured as a structured as a structured process for optimizing process for optimizing process for optimizing code. It all starts code. It all starts code. It all starts with the root of the tree. As with the root of the tree. As with the root of the tree. As you can see, the score you can see, the score you can see, the score was 0.5, which is was 0.5, which is was 0.5, which is simply a baseline simply a baseline simply a baseline program and a program and a program and a measured measured measured outcome. Each outcome. Each outcome. Each node after this node after this node after this root is a root is a root is a modified version of modified version of modified version of the code, run for the code, run for the code, run for evaluation with an added evaluation with an added evaluation with an added score. You can score. You can score. You can think of this tree think of this tree think of this tree as a graph of as a graph of as a graph of experiments. As experiments. As experiments. As you can see, the search you can see, the search you can see, the search is based on what is based on what is based on what works, works, works, improving successful improving successful improving successful options and options and options and cutting out unsuccessful ones.

cutting out unsuccessful ones. cutting out unsuccessful ones. So, as a branch So, as a branch So, as a branch improves the outcome, improves the outcome, improves the outcome, it develops these ideas it develops these ideas it develops these ideas and moves and moves and moves deeper. This means deeper. This means deeper. This means you can check the you can check the you can check the exact changes exact changes exact changes your your your auto-research system is making and auto-research system is making and auto-research system is making and build on them. So the build on them. So the build on them. So the next idea next idea next idea we talked about is we talked about is we talked about is observability. observability. observability. Because you can see Because you can see Because you can see each of the nodes in each of the nodes in each of the nodes in this this this tree-like tree-like tree-like search, you can search, you can search, you can open any of open any of open any of them and see exactly what has them and see exactly what has them and see exactly what has changed. You changed. You changed. You can watch can watch can watch your your your metric change in real metric change in real metric change in real time, linking movement time, linking movement time, linking movement to cause. You to cause. You to cause. You can can can hover over a node, hover over a node, hover over a node, get a summary of get a summary of get a summary of the changes, and also the changes, and also the changes, and also open it and open it and open it and read the exact diff. read the exact diff. read the exact diff. So when a metric So when a metric So when a metric jumps, you can jumps, you can jumps, you can read the change that read the change that read the change that caused it. And caused it. And caused it. And usually such usually such usually such runs last for runs last for runs last for hours over hours over hours over hundreds of experiments hundreds of experiments . So, when you see . So, when you see . So, when you see what she's doing, the what she's doing, the what she's doing, the next logical next logical next logical step is to be able step is to be able step is to be able to change her actions. This is to change her actions. This is to change her actions. This is essentially the moment when you essentially the moment when you essentially the moment when you stop stop stop observing and observing and observing and start acting yourself.

start acting yourself. start acting yourself. If I have an idea If I have an idea If I have an idea that I want that I want that I want to test in simple to test in simple to test in simple language, I implement language, I implement language, I implement it, a it, a it, a new branch opens, and you new branch opens, and you new branch opens, and you see it see it see it testing that idea, testing that idea, testing that idea, creating new nodes creating new nodes creating new nodes and a new curve. I and a new curve. I and a new curve. I can also set a can also set a can also set a budget for each budget for each budget for each node so that the system node so that the system node so that the system spends spends spends computing computing computing resources where I want resources where I want . And that's the part that . And that's the part that . And that's the part that made the system made the system made the system suitable for suitable for suitable for real-world problems, real-world problems, real-world problems, because you can because you can because you can have context and have context and have context and intuition that an agent doesn't have intuition that an agent doesn't have intuition that an agent doesn't have . At least for . At least for . At least for now, we think it's now, we think it's now, we think it's important to find important to find important to find ways to integrate ways to integrate ways to integrate human judgment into the human judgment into the human judgment into the system when system when system when necessary. I want necessary. I want necessary. I want to back up this to back up this to back up this point, controllability, with point, controllability, with point, controllability, with something real, something real, something real, because in parametric because in parametric because in parametric golf, controllability golf, controllability golf, controllability has been a big part of has been a big part of has been a big part of our our our success. This is a real success. This is a real success. This is a real breakthrough achieved breakthrough achieved breakthrough achieved during the competition. It all during the competition. It all during the competition. It all started with our started with our started with our agent, who is agent, who is agent, who is marked with a pink marked with a pink marked with a pink line. For 5 line. For 5 line. For 5 days, from April 15 to 19, days, from April 15 to 19, days, from April 15 to 19, he continued he continued he continued to try new things— to try new things— new focus, compression new focus, compression , a richer , a richer , a richer stack—but the results stack—but the results stack—but the results essentially leveled off. essentially leveled off.

essentially leveled off. There was a person in the There was a person in the There was a person in the community who community who community who released a clever released a clever released a clever new tokenizer, which is new tokenizer, which is new tokenizer, which is marked with a blue marked with a blue marked with a blue line. Um, and within line. Um, and within line. Um, and within 8 hours of 8 hours of 8 hours of this change appearing, our this change appearing, our this change appearing, our system picked it up, system picked it up, system picked it up, combined it with the combined it with the combined it with the elements it was elements it was elements it was already working on, and already working on, and already working on, and produced the biggest produced the biggest produced the biggest jump in jump in jump in weeks. Running weeks. Running weeks. Running auto-research for auto-research for auto-research for such complex such complex such complex tasks is not a tasks is not a tasks is not a case of case of case of setting it and forgetting it. setting it and forgetting it. setting it and forgetting it. It's a collaboration where It's a collaboration where It's a collaboration where we guide the we guide the we guide the agent, we pull agent, we pull agent, we pull ideas from the people ideas from the people ideas from the people around us, and the around us, and the around us, and the controllability is what controllability is what controllability is what makes this cycle makes this cycle makes this cycle possible. This is how possible. This is how real breakthroughs actually happen. Um, real breakthroughs actually happen. Um, I'll turn it over to Dex I'll turn it over to Dex I'll turn it over to Dex and Zhang Gao to and Zhang Gao to and Zhang Gao to talk a little talk a little talk a little more about how to more about how to more about how to set up set up set up auto-research. We will auto-research. We will auto-research. We will essentially recreate essentially recreate essentially recreate the process that you the process that you the process that you tried to perform tried to perform tried to perform on your laptops. on your laptops. on your laptops. OK. So we'll just OK. So we'll just OK. So we'll just show you the show you the show you the general concept: general concept: general concept: how does it all work? We'll how does it all work? We'll how does it all work? We'll just use just use just use our mobile our mobile our mobile hotspot. So hotspot. So hotspot. So we'll we'll we'll improvise a little.

improvise a little. I hope you don't I hope you don't mind. Well, so, yes. mind. Well, so, yes. You can just...Yes. Okay, maybe we'll Okay, maybe we'll also go through the also go through the also go through the entire entire entire data submission process for data submission process for data submission process for Claude. So, Claude. So, Claude. So, Yes. So, essentially, you Yes. So, essentially, you Yes. So, essentially, you can go into the can go into the can go into the repository and, repository and, repository and, with simple with simple with simple instructions, instructions, instructions, for example, run the for example, run the for example, run the execution as instructed execution as instructed execution as instructed in the Readme. And in the Readme. And in the Readme. And if the internet is if the internet is if the internet is stable, I think the stable, I think the standard standard execution process should start. You execution process should start. You execution process should start. You can also learn can also learn can also learn how the how the how the launch command works. launch command works. Yes. In this case, Yes. In this case, you have the source file. you have the source file. you have the source file. This solution—essentially a This solution—essentially a This solution—essentially a mini PG training scenario mini PG training scenario mini PG training scenario , I suppose. The , I suppose. The eval command eval command , as you seem to , as you seem to , as you seem to have mentioned, is an have mentioned, is an have mentioned, is an additional instruction additional instruction additional instruction or hint that or hint that or hint that will guide the entire will guide the entire will guide the entire search process. And search process. And search process. And the metric is the the metric is the the metric is the validation BPB. It's validation BPB. It's validation BPB. It's like losses on a like losses on a like losses on a delayed sample.

delayed sample. Type of loss only on the Type of loss only on the test set. test set. test set. The goal is to minimize The goal is to minimize The goal is to minimize them. And the other points are them. And the other points are them. And the other points are just configurations just configurations just configurations that we have already mentioned. that we have already mentioned. Oh, sorry. The eval command is actually the bash is actually the bash command you command you command you will run to will run to will run to perform the evaluation. And perform the evaluation. And perform the evaluation. And in the end, she will give a in the end, she will give a in the end, she will give a score. It outputs score. It outputs score. It outputs the score to the terminal, which is the score to the terminal, which is the score to the terminal, which is then analyzed by then analyzed by then analyzed by the agent. And this is the the agent. And this is the the agent. And this is the key signal that key signal that key signal that will drive all will drive all will drive all the cycles of trial and the cycles of trial and the cycles of trial and error. error. Yes, and in the meantime, I Yes, and in the meantime, I still have to show the dashboard. still have to show the dashboard. So if you go So if you go to the dashboard, and I to the dashboard, and I to the dashboard, and I think I was think I was think I was looking at the history there, looking at the history there, looking at the history there, you'll see you'll see you'll see a lot of things there. And, uh, a lot of things there. And, uh, a lot of things there. And, uh, after the after the after the commands are commands are commands are entered by the agents, entered by the agents, entered by the agents, I think you'll see the I think you'll see the I think you'll see the process running and you'll process running and you'll process running and you'll be able to try out be able to try out be able to try out all of our features.

all of our features. Yes. Do we have a Yes. Do we have a process in place to get process in place to get process in place to get started? started? Yes, I think it's Yes, I think it's starting up. starting up. starting up. Okay, it still works. He says Okay, it still works. He says , uh, yes, the cloud , uh, yes, the cloud , uh, yes, the cloud works. Can I works. Can I works. Can I switch to a switch to a switch to a cloud account? cloud account? cloud account? Uh, questions from Uh, questions from Uh, questions from the audience while you're at it the audience while you're at it the audience while you're at it . . . Can I increase Can I increase Can I increase the brightness in the the brightness in the the brightness in the control panel or control panel or control panel or enable the light enable the light enable the light theme? Sorry, once again. theme? Sorry, once again. theme? Sorry, once again. I'm here, right here. I'm here, right here. I'm here, right here. Can I turn on Can I turn on Can I turn on light mode on the light mode on the light mode on the panel? It's a little hard panel? It's a little hard panel? It's a little hard to see. to see. to see. I understand. Light I understand. Light I understand. Light mode. Where is he? Is that him?

mode. Where is he? Is that him? The second question is: The second question is: can autoresearch be applied can autoresearch be applied can autoresearch be applied to to to reinforcement learning reinforcement learning reinforcement learning (RL)? What (RL)? What (RL)? What scalar metric scalar metric scalar metric is used is used is used when when when multiple multiple multiple things need to be optimized? things need to be optimized? What is it? This Is it What is it? This Is it possible to apply possible to apply possible to apply auto-research to RL? auto-research to RL? auto-research to RL? What scalar metric What scalar metric What scalar metric is used is used is used when when when multiple multiple multiple things need to be optimized? things need to be optimized? things need to be optimized? Got it. Yes, this is Got it. Yes, this is Got it. Yes, this is entirely entirely entirely applicable to RL, as well as applicable to RL, as well as applicable to RL, as well as to RL training scripts to RL training scripts to RL training scripts . Uh, I think there's a project . Uh, I think there's a project . Uh, I think there's a project called Sokoban speedrunner called Sokoban speedrunner that does exactly that that does exactly that that does exactly that : : : retraining a model retraining a model retraining a model for the game. And yes, in that for the game. And yes, in that for the game. And yes, in that project you can project you can project you can apply apply apply auto-learning to auto-learning to auto-learning to RL training scripts.

RL training scripts. And regarding the second And regarding the second question: do you have question: do you have question: do you have multiple goals for multiple goals for multiple goals for optimization? The idea is to optimization? The idea is to optimization? The idea is to eventually eventually eventually combine them into a single combine them into a single combine them into a single metric: either metric: either metric: either apply different apply different apply different weights to these metrics, weights to these metrics, weights to these metrics, or consider some or consider some or consider some of them as constraints and of them as constraints and of them as constraints and optimize one optimize one optimize one specific one. For example, specific one. For example, specific one. For example, when detecting when detecting when detecting fraud, you can fraud, you can fraud, you can record accuracy record accuracy record accuracy and try and try and try to optimize to optimize to optimize precision. precision. And for parameterization, And for parameterization, you fix the you fix the you fix the model size as a constraint model size as a constraint model size as a constraint and optimize and optimize and optimize the accuracy of the system. Yes the accuracy of the system. Yes , we see that the agent , we see that the agent , we see that the agent has already created one node. has already created one node.

So, the zero node is essentially the is essentially the is essentially the root node, the root node, the root node, the base solution, and 3.93 base solution, and 3.93 base solution, and 3.93 means the validation means the validation means the validation BPP tied to BPP tied to BPP tied to that solution. You that solution. You that solution. You can see that can see that can see that auto-exploration auto-exploration auto-exploration has already created one has already created one has already created one node here. There is also a node here. There is also a node here. There is also a brief overview of the ideas brief overview of the ideas brief overview of the ideas it it it tested. It seems there tested. It seems there tested. It seems there is already an improvement. Oh is already an improvement. Oh is already an improvement. Oh yeah, you can also yeah, you can also yeah, you can also open the code and open the code and open the code and see exactly what see exactly what see exactly what was changed. was changed. Yes. Do we have Yes. Do we have stability? stability? So, for So, for stability, you stability, you stability, you can simply can simply can simply tell the agent, tell the agent, tell the agent, for example, to investigate the for example, to investigate the for example, to investigate the quantization method. Is quantization method. Is he, is he connected? Yes. The Yes. The quantization method, don't quantization method, don't quantization method, don't just change the just change the just change the model configuration, model configuration, model configuration, because we don't just want because we don't just want because we don't just want to tune to tune to tune hyperparameters, we hyperparameters, we hyperparameters, we want to try want to try want to try more aggressive more aggressive more aggressive methods. It methods. It doesn't seem to be doesn't seem to be connected.

connected. connected. O. automation O. automation It seems that Clockwork did not It seems that Clockwork did not follow the instruction. follow the instruction. Yes, in fact, we also Yes, in fact, we also support support support migrating your migrating your migrating your Clockwork code agent Clockwork code agent Clockwork code agent to our to our to our control panel. We control panel. We control panel. We call this a call this a call this a merged session. merged session. Um, so basically yes, you can see the yes, you can see the dashboard, the dashboard, the dashboard, the tree tree tree structure, the metrics, and structure, the metrics, and structure, the metrics, and Clockwork all at the same time. And Clockwork all at the same time. And Clockwork all at the same time. And then you can then you can then you can say to Clockwork, "This is the say to Clockwork, "This is the say to Clockwork, "This is the direction I direction I direction I want to take the whole want to take the whole want to take the whole search process." search process." I checked to see how you were I checked to see how you were doing. I think the doing. I think the doing. I think the internet could be internet could be internet could be a little better. Some of a little better. Some of a little better. Some of you are using a you are using a you are using a hotspot. I know there hotspot. I know there hotspot. I know there are some issues with are some issues with are some issues with payment in modal. We payment in modal. We payment in modal. We move on to the move on to the move on to the final part, final part, final part, namely the competitive namely the competitive namely the competitive aspect. You don't aspect. You don't aspect. You don't need a modal for need a modal for need a modal for this. This was originally this. This was originally this. This was originally supposed to be a 30-minute supposed to be a 30-minute supposed to be a 30-minute competition, but we'll give competition, but we'll give competition, but we'll give you until the end of the day.

you until the end of the day. A little more detail about A little more detail about what it is. You've all what it is. You've all what it is. You've all played Flappy Bird. We played Flappy Bird. We played Flappy Bird. We will be holding the Flappy WeGo Cup. will be holding the Flappy WeGo Cup. So, I hope So, I hope each of you has played Flappy each of you has played Flappy each of you has played Flappy Bird. You press to Bird. You press to Bird. You press to flap your wings. You flap your wings. You flap your wings. You dodge the pipes. dodge the pipes. dodge the pipes. And fly as far as possible And fly as far as possible . Your final score . Your final score . Your final score depends on how depends on how depends on how far you flew. far you flew. We turned this into an automated automated research task. So research task. So research task. So instead of instead of instead of playing it yourself, there's a playing it yourself, there's a playing it yourself, there's a "should flop" controller "should flop" controller "should flop" controller that decides each that decides each that decides each frame whether the bird should frame whether the bird should frame whether the bird should flap its wings. flap its wings. Your task is to optimize this to optimize this to optimize this controller. The metric controller. The metric is an average score, and you is an average score, and you is an average score, and you want to get want to get want to get the highest score possible the highest score possible , because the one with the , because the one with the , because the one with the highest total highest total highest total score wins. As with all score wins. As with all score wins. As with all games, there is a nuance here. We're games, there is a nuance here. We're games, there is a nuance here. We're not just launching this not just launching this not just launching this for Flappy Bird. We're for Flappy Bird. We're for Flappy Bird. We're launching this in launching this in launching this in four different four different four different worlds. The first world is worlds. The first world is normal, it's a normal, it's a normal, it's a classic game. The second is a moving classic game. The second is a moving gap, where gap, where gap, where the gap between the pipes the gap between the pipes the gap between the pipes moves up and moves up and down. In the third down. In the third down. In the third world, there is a wind that world, there is a wind that world, there is a wind that blows away a bird. And the blows away a bird. And the blows away a bird. And the fourth world is fourth world is changeable, where changeable, where changeable, where all all all conditions combine. Your score is the conditions combine. Your score is the conditions combine. Your score is the average across average across average across all four worlds.

all four worlds. all four worlds. So you can't So you can't So you can't focus on just focus on just focus on just one. A controller one. A controller one. A controller that works well in a that works well in a that works well in a regular game but regular game but regular game but dies in the wind won't dies in the wind won't dies in the wind won't win. The win. The win. The leaderboard ranks you against a leaderboard ranks you against a leaderboard ranks you against a hidden hidden hidden validation set that you validation set that you validation set that you never see, never see, never see, so you can't so you can't so you can't adjust your score adjust your score adjust your score for public play. for public play. A few rules, A few rules, mostly to keep mostly to keep mostly to keep everything fair and everything fair and everything fair and nothing broken. nothing broken. nothing broken. You just...and we'll You just...and we'll You just...and we'll be sharing the be sharing the be sharing the GitHub repository for GitHub repository for GitHub repository for this soon. You are this soon. You are this soon. You are only editing the controller, i.e. only editing the controller, i.e. only editing the controller, i.e. AI controller.py. You must AI controller.py. You must AI controller.py. You must keep the keep the keep the should_flap signature exactly as should_flap signature exactly as should_flap signature exactly as it is, so you it is, so you it is, so you cannot tamper with the game or the cannot tamper with the game or the cannot tamper with the game or the evaluator. No evaluator. No evaluator. No imports, no external imports, no external imports, no external libraries, no file libraries, no file libraries, no file I/O, we I/O, we I/O, we make it self-contained. make it self-contained. make it self-contained. And to be And to be And to be fair, fair, fair, parallel parallel parallel computing computing computing is prohibited, so is prohibited, so is prohibited, so you'll be using WeGo for auto-search you'll be using WeGo for auto-search you'll be using WeGo for auto-search . . . Everyone has the Everyone has the same default model, Opus 4.8 , so the conditions are equal for everyone , so the conditions are equal for everyone , so the conditions are equal for everyone . And remember, the . And remember, the . And remember, the local score local score local score you see on you see on you see on your laptop will not your laptop will not your laptop will not match the match the match the leaderboard leaderboard leaderboard due to our hidden due to our hidden due to our hidden test set.

test set. A few tips. Um, you'll A few tips. Um, you'll get about 50 get about 50 get about 50 steps in WeGo. steps in WeGo. It's actually a 30- minute time minute time minute time slot, so I think slot, so I think slot, so I think you can run an you can run an you can run an unlimited number of unlimited number of unlimited number of steps for the rest of the day. steps for the rest of the day. You can start with the You can start with the basic version. The basic version. The basic version. The basic version already basic version already basic version already works, so works, so works, so improve the code, improve the code, improve the code, not rewrite not rewrite not rewrite the controller. Think the controller. Think the controller. Think about small, about small, about small, targeted targeted targeted tweaks like tweaks like tweaks like adjusting the adjusting the adjusting the safety margin, keyboard safety margin, keyboard safety margin, keyboard speed, bottom speed, bottom speed, bottom buffer, adjusting buffer, adjusting buffer, adjusting behavior for behavior for behavior for each world. And each world. And each world. And maybe implement maybe implement maybe implement some of the features some of the features some of the features we we we talked about today, whether it's talked about today, whether it's talked about today, whether it's checking checking checking high-scoring nodes, high-scoring nodes, high-scoring nodes, developing them, or developing them, or developing them, or using ideas using ideas using ideas from outside. And try from outside. And try from outside. And try several directions. These are several directions. These are several directions. These are the instructions for the game. By the instructions for the game. By the instructions for the game. By the way, you're playing on the way, you're playing on the way, you're playing on Nintendo Switch. The way Nintendo Switch. The way Nintendo Switch. The way we will determine we will determine we will determine the winner is to provide the winner is to provide the winner is to provide your email address instead of the your email address instead of the your email address instead of the team name. And team name. And team name. And at the end of the day, we at the end of the day, we at the end of the day, we will contact the person will contact the person who gets the who gets the who gets the highest score. In the highest score. In the highest score. In the meantime, we will continue to meantime, we will continue to meantime, we will continue to answer answer answer questions from our questions from our questions from our Slido, or please Slido, or please Slido, or please raise your hand if raise your hand if raise your hand if you have a question.

you have a question. you have a question. Okay, we're almost Okay, we're almost Okay, we're almost done. Hmm, done. Hmm, done. Hmm, the results are already the results are already the results are already coming in. I don't know coming in. I don't know coming in. I don't know who Fornog is, but it's who Fornog is, but it's who Fornog is, but it's a little suspicious because a little suspicious because a little suspicious because there's only one node. there's only one node. So we will So we will be checking your be checking your be checking your runs to runs to runs to make sure make sure make sure there is no there is no there is no fraud. Ahem. fraud. Ahem. Not a bad attempt. We Not a bad attempt. We will open this will open this will open this competition until midnight competition until midnight competition until midnight tonight. We will find a tonight. We will find a tonight. We will find a way to contact way to contact way to contact you. You can you. You can you. You can provide your provide your provide your email, or email, or email, or we already have your email we already have your email addresses from registration, addresses from registration, addresses from registration, so you can just so you can just so you can just contact us on contact us on contact us on our website, but our website, but our website, but feel free to feel free to feel free to continue working, continue working, continue working, and thank you to everyone who and thank you to everyone who and thank you to everyone who came.