Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

Hello everyone. First of Hello everyone. First of all, I'm Ryan. I'm a full-stack all, I'm Ryan. I'm a full-stack engineer here at Exa. I am very engineer here at Exa. I am very engineer here at Exa. I am very happy to present happy to present happy to present you with this great you with this great you with this great opportunity. We'll have opportunity. We'll have perfect search for perfect search for AI agents soon, but AI agents soon, but AI agents soon, but at least now you at least now you at least now you can can can use use use version control. version control. version control. Today I want Today I want Today I want to tell you about to tell you about to tell you about a few things. So, a few things. So, first, a little bit about first, a little bit about first, a little bit about what Exa is, what what Exa is, what what Exa is, what agent-based web search is, and agent-based web search is, and agent-based web search is, and why you should care at all why you should care at all why you should care at all . . . At the end of the first At the end of the first At the end of the first point, you will understand: point, you will understand: point, you will understand: I need an I need an I need an agent web search. I agent web search. I agent web search. I need it for need it for need it for my tasks, my tasks, my tasks, regardless of regardless of regardless of whether you work with whether you work with whether you work with finance or not. finance or not. finance or not. Number two: OK, I Number two: OK, I Number two: OK, I want to have it, but why do want to have it, but why do want to have it, but why do I need I need I need version control? version control? version control? At the end of the second At the end of the second At the end of the second paragraph, you will understand paragraph, you will understand paragraph, you will understand why. And this will be very why. And this will be very why. And this will be very important, because in point important, because in point important, because in point number three, a number three, a number three, a small small small practical exercise awaits you, where practical exercise awaits you, where practical exercise awaits you, where you can win a you can win a you can win a prize. I hope this prize. I hope this prize. I hope this motivates you motivates you motivates you to listen carefully to points one and to listen carefully to points one and to listen carefully to points one and two. So, I'll start by two. So, I'll start by two. So, I'll start by asking you to asking you to asking you to raise your hands to get your raise your hands to get your raise your hands to get your bearings. Who bearings. Who bearings. Who here has heard of Exa before? here has heard of Exa before? here has heard of Exa before? Okay, a lot of people Okay, a lot of people Okay, a lot of people have heard, but there are also those who have have heard, but there are also those who have have heard, but there are also those who have n't, and that's great. n't, and that's great. n't, and that's great. How many of you How many of you How many of you have used have used have used search engines search engines search engines like Google to like Google to like Google to research job-related research job-related research job-related issues in previous issues in previous issues in previous positions? For example, positions? For example, we simply used Google we simply used Google to search for to search for to search for information at home, at information at home, at information at home, at work, anywhere.

work, anywhere. work, anywhere. I hope I hope I hope most of you are cool. And the most of you are cool. And the most of you are cool. And the last part will be last part will be last part will be about how often you about how often you about how often you stopped stopped stopped using Google and using Google and using Google and instead instead instead used ChatGPT, used ChatGPT, used ChatGPT, Claude, or another agent Claude, or another agent Claude, or another agent saying, "Hey, saying, "Hey, saying, "Hey, can you find this can you find this can you find this for me?" Raise your for me?" Raise your for me?" Raise your hands if you're hands if you're hands if you're doing this right now. doing this right now. doing this right now. Wonderful, great. Okay, Wonderful, great. Okay, Wonderful, great. Okay, perfect. We will perfect. We will perfect. We will combine all three of these combine all three of these combine all three of these things, because what things, because what many of you have confirmed is what we are many of you have confirmed is what we are currently currently currently observing. So, observing. So, observing. So, AI is now searching for AI is now searching for AI is now searching for information much more information much more information much more often than humans. often than humans. often than humans. Why? Because it can Why? Because it can Why? Because it can handle so many handle so many handle so many requests back and forth, requests back and forth, requests back and forth, right? It can right? It can right? It can go online, go online, go online, run a few run a few run a few searches, searches, searches, find what you're find what you're find what you're actually looking for, actually looking for, actually looking for, browse through a lot of browse through a lot of browse through a lot of pages, and pages, and work hard to give work hard to give you the you the you the answer you need. And we answer you need. And we answer you need. And we see that more and more see that more and more see that more and more people are using people are using people are using AI for search, and AI is AI for search, and AI is AI for search, and AI is searching more often and more searching more often and more searching more often and more efficiently than humans. efficiently than humans. efficiently than humans. People are a bit lazy. You People are a bit lazy. You People are a bit lazy. You say to yourself, "Okay, say to yourself, "Okay, say to yourself, "Okay, I'm looking for this." I'll I'm looking for this." I'll I'm looking for this." I'll enter a few enter a few enter a few keywords. keywords. keywords. I hope it's I hope it's I hope it's on the front page, on the front page, on the front page, otherwise I don't care. otherwise I don't care. otherwise I don't care. Instead, AI systems Instead, AI systems Instead, AI systems really want to really want to really want to please. They are please. They are please. They are usually quite usually quite usually quite accommodating. I don't know accommodating. I don't know accommodating. I don't know if you noticed that. if you noticed that. if you noticed that. So they So they So they will keep will keep will keep looking until they looking until they looking until they find what they need.

find what they need. find what they need. All of these things mean All of these things mean that there is a that there is a that there is a lot of searching. And all these lot of searching. And all these lot of searching. And all these tools, when you tools, when you tools, when you ask them questions, ask them questions, ask them questions, think, "I need to think, "I need to think, "I need to look beyond look beyond look beyond my knowledge." "I my knowledge." "I my knowledge." "I need to go need to go need to go online." He online." He online." He will use a will use a will use a tool like Exa. And that's exactly what tool like Exa. And that's exactly what tool like Exa. And that's exactly what we provide. we provide. we provide. So we actually So we actually So we actually provide search provide search provide search for some of the for some of the for some of the world's largest world's largest world's largest agents for agents for agents for programming and programming and programming and agent applications agent applications . So when you're . So when you're . So when you're using Cursor and using Cursor and using Cursor and it's like, "Oh, if only it's like, "Oh, if only it's like, "Oh, if only I knew what was in the I knew what was in the I knew what was in the documentation, I could documentation, I could documentation, I could really help." really help." really help." He will use Exa to He will use Exa to He will use Exa to find this document and find this document and find this document and return it to return it to return it to his orchestrator his orchestrator his orchestrator to answer the to answer the to answer the question. Or you question. Or you question. Or you use Gamma and use Gamma and use Gamma and think, "Oh, if only I think, "Oh, if only I think, "Oh, if only I knew what the knew what the knew what the financial results of financial results of , well, I don't know, Bank of America were , well, I don't know, Bank of America were , well, I don't know, Bank of America were in 2019." This slide in 2019." This slide in 2019." This slide would appear. He can would appear. He can would appear. He can use Exa to use Exa to use Exa to find this information. find this information. find this information. This is where we are This is where we are . I also want to . I also want to . I also want to talk briefly about talk briefly about talk briefly about finance, because it's a finance, because it's a finance, because it's a combination of Wall Street combination of Wall Street combination of Wall Street and AI. So I hope and AI. So I hope and AI. So I hope you either work in you either work in you either work in finance or finance or finance or are interested in it. But are interested in it. But are interested in it. But we are working in we are working in we are working in many of these many of these many of these areas. This is an area areas. This is an area areas. This is an area that particularly that particularly that particularly interests me. I interests me. I interests me. I work with some work with some work with some hedge funds, hedge funds, hedge funds, sovereign wealth funds, sovereign wealth funds, sovereign wealth funds, asset managers, asset managers, asset managers, and venture capital and venture capital and venture capital firms on these firms on these firms on these issues because they issues because they issues because they need information.

need information. need information. Is it a hedge fund Is it a hedge fund Is it a hedge fund looking for "alpha" to looking for "alpha" to looking for "alpha" to make financial make financial make financial decisions? Or an asset manager decisions? Or an asset manager trying trying to predict risks to predict risks . Or a venture . Or a venture . Or a venture capitalist who wants to capitalist who wants to capitalist who wants to find find find the most promising the most promising the most promising new startup new startup new startup before The Verge or anyone before The Verge or anyone before The Verge or anyone else writes else writes else writes an article about it. an article about it. an article about it. They all use They all use They all use Exa. So I hope Exa. So I hope Exa. So I hope at the end you at the end you at the end you say, "Okay, yeah." say, "Okay, yeah." This is really a very, very This is really a very, very interesting field. But interesting field. But interesting field. But why Exa? What are we why Exa? What are we why Exa? What are we doing? We are a little doing? We are a little doing? We are a little different in that we different in that we different in that we consider ourselves a consider ourselves a consider ourselves a laboratory and laboratory and laboratory and do do do basic basic basic research to research to research to move this forward. move this forward. move this forward. So, I'll quickly So, I'll quickly So, I'll quickly touch on the basic touch on the basic touch on the basic triangle of quality, triangle of quality, triangle of quality, cost, and latency cost, and latency cost, and latency so you can understand so you can understand so you can understand why you should add X to why you should add X to why you should add X to your application, and your application, and your application, and then we'll talk then we'll talk then we'll talk about versioning, about versioning, about versioning, which will allow you to which will allow you to which will allow you to say, "Ah, now I say, "Ah, now I say, "Ah, now I see how and where this can be see how and where this can be see how and where this can be used to used to used to test my test my test my application." So, application." So, first, quality. You first, quality. You first, quality. You want your want your want your results to be of results to be of results to be of high quality. We high quality. We high quality. We care so much about care so much about care so much about this that we created our this that we created our this that we created our own generative own generative own generative benchmark. Why benchmark. Why benchmark. Why do this? Because the do this? Because the do this? Because the search field is developing search field is developing search field is developing very quickly. You very quickly. You very quickly. You publish a benchmark, publish a benchmark, publish a benchmark, and all the search and all the search and all the search providers, and later providers, and later providers, and later the labs, say, the labs, say, "We need to "We need to "We need to train on this train on this train on this to conquer this to conquer this to conquer this peak." So, if you're a peak." So, if you're a peak." So, if you're a search engine that search engine that search engine that really believes in your really believes in your really believes in your product, you want to product, you want to product, you want to create a benchmark create a benchmark create a benchmark so that people can so that people can so that people can adapt it to adapt it to adapt it to their needs and their needs and their needs and keep it up to date keep it up to date so that your system so that your system works for you, works for you, works for you, providing the most up-to-date providing the most up-to-date providing the most up-to-date information. This is the information. This is the information. This is the main reason why main reason why main reason why you use a you use a you use a search search search provider and not provider and not provider and not just ask ChatGPT

just ask ChatGPT just ask ChatGPT or OpenAI and get or OpenAI and get or OpenAI and get answers from their scales answers from their scales . But you say, " . But you say, " Okay, I want quality, Okay, I want quality, Okay, I want quality, but at the lowest price." but at the lowest price." but at the lowest price." We take care of this not We take care of this not We take care of this not only during search, only during search, only during search, because search is a lot of because search is a lot of because search is a lot of tokens that tokens that tokens that end up in the end up in the end up in the context window. And context window. And context window. And what does that mean? You what does that mean? You what does that mean? You pay for the input pay for the input pay for the input . That's why we've . That's why we've . That's why we've developed technologies developed technologies to return only to return only to return only the tokens you the tokens you the tokens you need, not just for need, not just for need, not just for one result, but one result, but one result, but for all of them, so your for all of them, so your for all of them, so your agent or other agent or other agent or other application can application can application can answer answer answer questions without questions without questions without spending a lot on spending a lot on spending a lot on input tokens. input tokens. input tokens. Finally, the delay. Finally, the delay. Finally, the delay. You can work You can work You can work very quickly here: we have very quickly here: we have very quickly here: we have a search that's a search that's a search that's less than 300 milliseconds, and less than 300 milliseconds, and less than 300 milliseconds, and tasks like " tasks like " find me all the find me all the find me all the video game publishers," video game publishers," video game publishers," because I was doing an assessment. because I was doing an assessment. because I was doing an assessment. This ran for 20 minutes This ran for 20 minutes This ran for 20 minutes and found 3,212 publishers and found 3,212 publishers and found 3,212 publishers and developers and developers and developers they they they partner with. So we partner with. So we partner with. So we cover the whole cover the whole cover the whole spectrum, no matter spectrum, no matter spectrum, no matter at what stage at what stage at what stage you implement it in you implement it in you implement it in your architecture. And your architecture. And your architecture. And now you're saying, " now you're saying, " Ryan, okay, that's Ryan, okay, that's Ryan, okay, that's great. I really great. I really great. I really like Exa, but I like Exa, but I like Exa, but I want to control its want to control its want to control its versions." Or: why do versions." Or: why do versions." Or: why do I need I need I need version control at all? version control at all? version control at all? And that's actually going to be the And that's actually going to be the And that's actually going to be the main focus of main focus of main focus of our our our discussion today—we're going discussion today—we're going discussion today—we're going to discuss a new to discuss a new to discuss a new feature called Axis feature called Axis feature called Axis Snapshot, which allows you Snapshot, which allows you Snapshot, which allows you to use our to use our to use our search as it is now search as it is now search as it is now , but see , but see , but see the results as if the results as if the results as if you were searching in the past.

you were searching in the past. you were searching in the past. First, I'll explain what it is First, I'll explain what it is First, I'll explain what it is and why it might and why it might and why it might be interesting to you. So, be interesting to you. So, be interesting to you. So, first of all, what is this? We are a first of all, what is this? We are a search engine, search engine, search engine, so we spend so we spend so we spend a lot of time a lot of time a lot of time monitoring the internet monitoring the internet . Whenever we . Whenever we . Whenever we find content, we find content, we find content, we add it to add it to add it to our index. And our index. And our index. And when we discover a when we discover a when we discover a new version of a page, new version of a page, new version of a page, or even just or even just or even just come across the come across the come across the same page in same page in same page in the past, we the past, we the past, we save a copy of it. save a copy of it. This means that for This means that for every URL in every URL in every URL in our index, which our index, which our index, which spans the entire web with spans the entire web with spans the entire web with nearly 100 billion nearly 100 billion nearly 100 billion documents and documents and documents and tracks trillions of tracks trillions of tracks trillions of links, you links, you links, you can now go back in can now go back in can now go back in time. And there are two ways to time. And there are two ways to time. And there are two ways to do this. First: I do this. First: I do this. First: I know the URL, but I know the URL, but I know the URL, but I want to see what want to see what want to see what it looked like in the it looked like in the it looked like in the past. For this, past. For this, past. For this, there is versioning of there is versioning of there is versioning of content. You can content. You can content. You can specify a date and we'll specify a date and we'll specify a date and we'll show you content show you content show you content as of that day as of that day as of that day or the closest date or the closest date that was that was that was previously available. So you can previously available. So you can previously available. So you can be sure that be sure that be sure that you will receive the version of the you will receive the version of the you will receive the version of the page that we page that we page that we published at that time. published at that time. published at that time. Broadly speaking, you can Broadly speaking, you can Broadly speaking, you can use use use version search, which version search, which version search, which allows you to not just allows you to not just allows you to not just find a URL, but perform find a URL, but perform find a URL, but perform a search as you would a search as you would a search as you would do it now, but do it now, but do it now, but taking into account taking into account taking into account past data. Let's past data. Let's past data. Let's look at two very look at two very look at two very brief examples. And brief examples. And brief examples. And then I'll talk a little bit then I'll talk a little bit then I'll talk a little bit about potential about potential usage scenarios. So, the usage scenarios. So, the first example: first example: first example: let's say you are let's say you are let's say you are interested in financial interested in financial interested in financial markets. Who here is markets. Who here is markets. Who here is interested in interested in interested in financial markets?

financial markets? financial markets? Perhaps you Perhaps you Perhaps you are trying are trying are trying to understand which to understand which to understand which startups are currently startups are currently startups are currently the most popular and the most popular and the most popular and promising. promising. promising. You've probably heard of You've probably heard of You've probably heard of OpenAI. OpenAI. OpenAI. Interesting Interesting Interesting news recently emerged when Axios and news recently emerged when Axios and news recently emerged when Axios and other publications other publications other publications reported on reported on reported on OpenAI's annual revenue, OpenAI's annual revenue, OpenAI's annual revenue, citing figures of citing figures of citing figures of around $70 billion. around $70 billion. And then there was And then there was a lot of news about a lot of news about a lot of news about whether it was 70 billion or whether it was 70 billion or whether it was 70 billion or 50 billion? 50 billion? 50 billion? It turns out it's something It turns out it's something It turns out it's something like like accounting: gross or accounting: gross or net income, do net income, do net income, do you pass on you pass on you pass on the costs of certain the costs of certain the costs of certain cloud providers cloud providers cloud providers to yourself, do to yourself, do to yourself, do you recognize this income or not? you recognize this income or not? you recognize this income or not? But if you're a person But if you're a person But if you're a person who's in the business of who's in the business of who's in the business of understanding these kinds of understanding these kinds of understanding these kinds of things, it would be really things, it would be really things, it would be really helpful if you helpful if you helpful if you could say, "Hey, I could say, "Hey, I could say, "Hey, I want to make sure I'm want to make sure I'm want to make sure I'm designing a system that's designing a system that's designing a system that's capable of looking at capable of looking at capable of looking at these kinds of things, right?" these kinds of things, right?" One that is capable, One that is capable, for example, at the for example, at the for example, at the moment you would moment you would moment you would do this search, do this search, do this search, when you would have this when you would have this when you would have this information, is it information, is it information, is it set up set up set up to detect this to detect this to detect this accounting accounting accounting note that note that note that could actually could actually could actually be very valuable for be very valuable for be very valuable for commerce? Let's say you're commerce? Let's say you're commerce? Let's say you're trying to do trying to do trying to do something, perhaps even something, perhaps even something, perhaps even more or less important more or less important , namely—understand , namely—understand , namely—understand negative negative negative media coverage of media coverage of media coverage of very important people very important people very important people or perhaps less or perhaps less or perhaps less important people. For example, important people. For example, important people. For example, Ed Sheeran recently Ed Sheeran recently Ed Sheeran recently had some negative had some negative had some negative posts, but if you posts, but if you posts, but if you look a little bit look a little bit look a little bit before that event, before that event, before that event, you can see what you can see what you can see what other things were going on, or, other things were going on, or, other things were going on, or, you know, he wasn't you know, he wasn't you know, he wasn't very reliable with very reliable with very reliable with insurance, maybe insurance, maybe insurance, maybe you wouldn't want you wouldn't want you wouldn't want to offer him to offer him to offer him some kind of insurance

some kind of insurance some kind of insurance product. But the main thing product. But the main thing product. But the main thing here is that now, when here is that now, when here is that now, when you're, for example, an you're, for example, an you're, for example, an insurance provider insurance provider insurance provider and you want to do and you want to do and you want to do due diligence on a due diligence on a due diligence on a counterparty, KYB or KYC, counterparty, KYB or KYC, counterparty, KYB or KYC, or you're in or you're in or you're in any business and you're any business and you're any business and you're trying to trying to trying to establish establish partnerships, and part of partnerships, and part of your process is your process is your process is understanding the other understanding the other understanding the other party, it would be nice to party, it would be nice to party, it would be nice to be able to be able to be able to go back and go back and go back and say, "Hey, I say, "Hey, I say, "Hey, I designed this system to designed this system to designed this system to do do do a bunch of searches on the a bunch of searches on the a bunch of searches on the internet and internet and internet and figure out who this figure out who this figure out who this person is, what kind of business they are person is, what kind of business they are , how they can be , how they can be , how they can be classified, whether classified, whether classified, whether we should we should we should insure them or not." And insure them or not." And insure them or not." And now you can now you can now you can design such a design such a design such a system, and then with system, and then with system, and then with additional data additional data additional data go back and go back and go back and test it. So these are test it. So these are test it. So these are just a few just a few just a few examples of how and examples of how and examples of how and where, for example, a where, for example, a where, for example, a snapshot can be snapshot can be snapshot can be useful for useful for useful for testing these testing these testing these systems on systems on systems on historical data. So historical data. So , here's just an example of , here's just an example of , here's just an example of that. As a side note, it is that. As a side note, it is that. As a side note, it is important important important to be aware of this to be aware of this to be aware of this as it can as it can as it can help you help you help you achieve great achieve great achieve great success in the success in the success in the future. But future. But future. But let's say you're let's say you're let's say you're trying trying trying to understand something to understand something to understand something related to, for example, the related to, for example, the related to, for example, the acquisition of acquisition of acquisition of companies. So you companies. So you companies. So you can say, "Hey, can say, "Hey, can say, "Hey, I'm trying I'm trying I'm trying to design my own to design my own to design my own system." If you system." If you system." If you do a search now, do a search now, do a search now, you'll see that you'll see that you'll see that the company has already the company has already the company has already been acquired, but I been acquired, but I been acquired, but I might want to might want to might want to go back in time go back in time go back in time and ask, "Hey, what and ask, "Hey, what and ask, "Hey, what was the day before was the day before was the day before the acquisition, the week the acquisition, the week the acquisition, the week before the acquisition, the before the acquisition, the before the acquisition, the month before the acquisition?"

month before the acquisition?" month before the acquisition?" Would I be able to Would I be able to Would I be able to find information find information find information to predict this? to predict this? to predict this? Therefore, any Therefore, any Therefore, any real-time search real-time search real-time search must must must be kept up-to- be kept up-to- be kept up-to- date. That's what date. That's what date. That's what you're actually you're actually you're actually paying for, right? That paying for, right? That paying for, right? That if you did a search if you did a search if you did a search now, you would get the now, you would get the now, you would get the most up-to-date most up-to-date most up-to-date information. But it information. But it information. But it also, in a way, " also, in a way, " leaks" knowledge about the leaks" knowledge about the leaks" knowledge about the future into the assessment of future into the assessment of future into the assessment of the past. If you're the past. If you're the past. If you're building some kind of building some kind of building some kind of predictive system, and predictive system, and predictive system, and all it does all it does is, "Okay, I'm watching is, "Okay, I'm watching is, "Okay, I'm watching the news, and it says the news, and it says the news, and it says this has been acquired, so this has been acquired, so this has been acquired, so check mark, acquired." 100 check mark, acquired." 100 % success rate of % success rate of % success rate of truly positive truly positive truly positive results. Provided results. Provided that the search is of sufficient that the search is of sufficient that the search is of sufficient quality, which you, quality, which you, quality, which you, of course, hope for. of course, hope for. of course, hope for. What's useful about What's useful about What's useful about snapshots and snapshots and snapshots and version control version control version control search is that you search is that you search is that you can limit what can limit what can limit what information is information is information is available at any given available at any given available at any given time. So in time. So in time. So in this little this little this little example, I can example, I can example, I can set the set the set the snapshot date to the moment snapshot date to the moment snapshot date to the moment right before right before the result is announced. So I'll the result is announced. So I'll know that the only thing know that the only thing know that the only thing this system is based on this system is based on this system is based on is is is information that doesn't information that doesn't information that doesn't provide accurate provide accurate provide accurate confirmation of this confirmation of this confirmation of this event, right? And then event, right? And then event, right? And then you can you can you can configure it to configure it to configure it to perform different perform different perform different tasks. For example, tasks. For example, tasks. For example, maybe I have a maybe I have a maybe I have a trait trait trait related to hiring that related to hiring that related to hiring that I'm really interested in. I I'm really interested in. I I'm really interested in. I want to make sure I want to make sure I want to make sure I 'm finding the right 'm finding the right 'm finding the right signals for hiring.

signals for hiring. signals for hiring. Maybe I have a Maybe I have a Maybe I have a sign that involves sign that involves sign that involves analyzing analyzing analyzing earnings reports, transcripts, earnings reports, transcripts, earnings reports, transcripts, or or CEO interviews on CEO interviews on podcasts. Whatever is podcasts. Whatever is podcasts. Whatever is important to you important to you , you can , you can , you can tweak the system tweak the system tweak the system and see, "Hey, and see, "Hey, and see, "Hey, would this would this would this actually improve my actually improve my actually improve my forecasting or not forecasting or not forecasting or not ?" I ?" I ?" I will share—and spend will share—and spend will share—and spend quite a bit of time on it quite a bit of time on it quite a bit of time on it —a couple of patterns —a couple of patterns —a couple of patterns that we have that we have that we have been seeing in the been seeing in the been seeing in the backtesting space lately. backtesting space lately. The first of these The first of these concerns the idea of ​​the concerns the idea of ​​the concerns the idea of ​​the traditional model. traditional model. The idea is The idea is that you simply collect that you simply collect that you simply collect the data needed to the data needed to the data needed to backtest the model you backtest the model you backtest the model you already have. You can already have. You can already have. You can think of it like this: " think of it like this: " Hey, maybe I have Hey, maybe I have Hey, maybe I have some system, it some system, it some system, it works in the world, and I works in the world, and I works in the world, and I don't need all this don't need all this don't need all this agent-based web search." agent-based web search." agent-based web search." I love my system. I love my system. I love my system. She's wonderful, isn't she? She's wonderful, isn't she? She's wonderful, isn't she? But maybe you But maybe you But maybe you can't just can't just can't just go back and go back and go back and collect the signs you collect the signs you collect the signs you had in the past, had in the past, had in the past, right? It may right? It may right? It may be that they are be that they are be that they are based on based on based on web pages that web pages that web pages that simply don't store simply don't store simply don't store versions to which you can versions to which you can versions to which you can revert. Maybe revert. Maybe revert. Maybe you're trying to you're trying to you're trying to build a system that build a system that build a system that analyzes newspaper analyzes newspaper analyzes newspaper columns or other columns or other columns or other articles, and you can't articles, and you can't articles, and you can't trust the trust the trust the publication date.

publication date. A small digression: the A small digression: the publication dates of publication dates of publication dates of articles articles articles are actually constantly updated are actually constantly updated . Both because "we . Both because "we . Both because "we really made really made really made changes," and for other changes," and for other changes," and for other reasons. To the point where reasons. To the point where reasons. To the point where "Hey, SEO really loves "Hey, SEO really loves "Hey, SEO really loves fresh articles." So we fresh articles." So we fresh articles." So we can just can just can just update it again. This update it again. This update it again. This means that you means that you means that you cannot rely cannot rely cannot rely on the publication date indication on the publication date indication on the publication date indication , which is , which is , which is important, especially important, especially important, especially if you are making if you are making if you are making significant decisions significant decisions significant decisions based on it—you based on it—you based on it—you need to be need to be need to be confident in the accuracy of confident in the accuracy of confident in the accuracy of such an indication. So you such an indication. So you such an indication. So you might decide, "Hey might decide, "Hey , I'm going to , I'm going to , I'm going to use use use snapshots." I know snapshots." I know snapshots." I know certain URLs or certain URLs or certain URLs or sites that sites that sites that interest me. I'll just interest me. I'll just interest me. I'll just go back, go back, go back, collect the data I need, collect the data I need, collect the data I need, and then convert it and then convert it and then convert it into the feature I need—an into the feature I need—an array of numbers, etc. X is array of numbers, etc. X is array of numbers, etc. X is n't even involved in n't even involved in n't even involved in forecasting, but it's a forecasting, but it's a forecasting, but it's a perfectly valid and perfectly valid and perfectly valid and relatively simple relatively simple relatively simple way that way that way that you may already be you may already be you may already be using using using versioned data to versioned data to versioned data to backtest your backtest your backtest your system. Okay, that's what it system. Okay, that's what it system. Okay, that's what it is for now. is for now. is for now. Um, what about Um, what about Um, what about the future? The the future? The the future? The future of perfect future of perfect future of perfect search. So, there are search. So, there are search. So, there are two types of templates here.

two types of templates here. two types of templates here. First—I'm going to First—I'm going to First—I'm going to add an agent add an agent add an agent function to the function to the function to the traditional model. traditional model. This is similar to what This is similar to what we talked about earlier, we talked about earlier, we talked about earlier, but the idea is but the idea is but the idea is that in a real that in a real that in a real model it model it model it would use X would use X would use X to go online, to go online, to go online, find information, find information, find information, turn it into a turn it into a turn it into a structured structured structured signal, and trade signal, and trade signal, and trade based on that. You based on that. You based on that. You can think of it as can think of it as can think of it as something like something like something like scanning scanning scanning negative media, like negative media, like negative media, like we did with our we did with our we did with our friend Sheeran friend Sheeran friend Sheeran before. Um, you can before. Um, you can before. Um, you can design your design your design your system so that it system so that it system so that it can perform can perform counterparty validation, and in a counterparty validation, and in a real production real production real production system, that might system, that might system, that might just be one of the just be one of the just be one of the components, right? The components, right? The system itself might not system itself might not have any have any have any agentivity, it would agentivity, it would agentivity, it would just collect just collect just collect a bunch of features, and then you a bunch of features, and then you might train a might train a might train a traditional traditional traditional machine learning model machine learning model machine learning model to say "yes, to say "yes, to say "yes, we invest" or " we invest" or " no, we don't invest." no, we don't invest." You can You can go back and go back and go back and backtest backtest backtest these agent functions these agent functions these agent functions one by one, one by one, one by one, say with X, and say with X, and say with X, and compare them to the compare them to the compare them to the actual positive actual positive actual positive outcome when what outcome when what outcome when what you're you're you're interested in happened: whether the interested in happened: whether the interested in happened: whether the business succeeded business succeeded business succeeded or not. You can or not. You can or not. You can protest and protest and protest and say, "Did we say, "Did we predict this correctly?" Or not?

predict this correctly?" Or not? And, most importantly, you And, most importantly, you And, most importantly, you can customize can customize can customize this system. Because this system. Because this system. Because in all these in all these in all these non-deterministic non-deterministic non-deterministic systems, the way systems, the way systems, the way they are configured they are configured they are configured is really is really is really extremely important extremely important . So if you . So if you . So if you customize your search in customize your search in customize your search in a certain way, a certain way, it might be more successful. it might be more successful. it might be more successful. If you choose a different If you choose a different If you choose a different search method, success search method, success search method, success will be less likely. will be less likely. will be less likely. This means you This means you This means you can control can control can control the experiment and then the experiment and then the experiment and then add it to your add it to your add it to your system, one system, one system, one feature at a time. And feature at a time. And feature at a time. And on the right, the idea on the right, the idea on the right, the idea is that in is that in is that in the future, predictions the future, predictions , startups, and other , startups, and other , startups, and other entities entities entities will operate in a will operate in a will operate in a mode where the entire mode where the entire mode where the entire system is potentially system is potentially system is potentially agent-like. So, the agent-like. So, the agent-like. So, the system itself can choose system itself can choose system itself can choose whether to search for something on whether to search for something on whether to search for something on the Internet by the Internet by the Internet by calling the tool. calling the tool. She can combine She can combine internal knowledge. internal knowledge. internal knowledge. It can combine It can combine It can combine structured structured structured external knowledge. And external knowledge. And external knowledge. And she can do it she can do it she can do it during execution. during execution. during execution. The idea is that The idea is that The idea is that it's similar to what it's similar to what it's similar to what people do when they people do when they people do when they do do do forecasting. They forecasting. They forecasting. They have a certain freedom of have a certain freedom of have a certain freedom of action. They are looking for this action. They are looking for this action. They are looking for this or that. Essentially, you or that. Essentially, you or that. Essentially, you can bind this can bind this can bind this interface for your interface for your interface for your model to the model to the model to the outside world. And outside world. And outside world. And then you can then you can then you can make sure that when make sure that when make sure that when the system was running, the system was running, the system was running, it could it could it could only search for the information it only search for the information it only search for the information it had at that had at that had at that time. Here's a time. Here's a time. Here's a practical example: practical example: practical example: let's say you're let's say you're let's say you're trying trying trying to improve your to improve your to improve your model for writing model for writing model for writing code. One part of code. One part of code. One part of this model would be this model would be this model would be whether it can resolve a whether it can resolve a whether it can resolve a merge request (PR) or merge request (PR) or merge request (PR) or not. You can " not. You can " freeze" the system state freeze" the system state just before just before resolving this PR in the resolving this PR in the resolving this PR in the external external external repository, and then

repository, and then repository, and then run it. And then, run it. And then, run it. And then, if it can if it can if it can complete the task, you complete the task, you complete the task, you 'll be more confident that 'll be more confident that 'll be more confident that it's the result of changing the it's the result of changing the it's the result of changing the model weights or some other model weights or some other model weights or some other component of the system component of the system component of the system that you're trying to that you're trying to that you're trying to improve through improve through improve through training, and not just a training, and not just a training, and not just a leak of future leak of future leak of future information into the information into the information into the present. OK. This is present. OK. This is present. OK. This is just a brief overview of just a brief overview of just a brief overview of Snapshot and a few Snapshot and a few Snapshot and a few ways to ways to ways to use it for use it for use it for backtesting. I'm just backtesting. I'm just backtesting. I'm just summarizing what we've already summarizing what we've already summarizing what we've already seen. So, first of all seen. So, first of all , we really , we really , we really like Exa. Why do like Exa. Why do like Exa. Why do we like Exa? we like Exa? we like Exa? Because this is a very high- Because this is a very high- Because this is a very high- quality search. Search is quality search. Search is quality search. Search is very important for very important for very important for models because they models because they models because they need to know need to know need to know relevant information relevant information , and it has to be of , and it has to be of , and it has to be of high quality. It should be high quality. It should be high quality. It should be economically viable. And economically viable. And economically viable. And this should happen with this should happen with this should happen with a delay that is a delay that is a delay that is acceptable for my acceptable for my acceptable for my system. Okay, we all system. Okay, we all system. Okay, we all learned that. And we're like learned that. And we're like , "Hey, I really want , "Hey, I really want , "Hey, I really want this." A big part of this." A big part of this." A big part of implementing this into implementing this into implementing this into my model is being my model is being my model is being able to able to able to version control it so I can version control it so I can version control it so I can prove that it adds prove that it adds prove that it adds value, not just value, not just value, not just hope it does. hope it does. Now we will move on to the Now we will move on to the most interesting part of most interesting part of most interesting part of our workshop: we will our workshop: we will our workshop: we will give you time give you time give you time to work with this to work with this to work with this tool tool tool yourself. So, yourself. So, yourself. So, there's a little QR code on the screen there's a little QR code on the screen , you can , you can , you can scan it with your scan it with your scan it with your mobile phone.

mobile phone. mobile phone. You can also You can also You can also just type this just type this just type this short URL on short URL on short URL on your computer, and the your computer, and the your computer, and the interface on your interface on your interface on your laptop laptop laptop will probably be more convenient. And will probably be more convenient. And will probably be more convenient. And now I'll give you a now I'll give you a now I'll give you a little overview of what little overview of what little overview of what we're going to we're going to we're going to do here. The idea is do here. The idea is do here. The idea is , recalling the , recalling the , recalling the example we example we example we discussed earlier, that discussed earlier, that discussed earlier, that you will try to you will try to you will try to predict predict predict company takeovers company takeovers company takeovers just before just before just before they they they happen. To happen. To happen. To tell you a little bit about tell you a little bit about tell you a little bit about the background. the background. the background. Let's say you Let's say you Let's say you work at a work at a work at a financial institution, financial institution, financial institution, and your and your and your boss's boss went boss's boss went boss's boss went to an AI to an AI engineering seminar and said, " engineering seminar and said, " Hey, I heard about this Hey, I heard about this Hey, I heard about this really cool really cool really cool web search feature. I think we web search feature. I think we web search feature. I think we should introduce it to should introduce it to should introduce it to our portfolio our portfolio our portfolio manager." And the manager." And the manager." And the portfolio manager portfolio manager portfolio manager says, "Okay, I want says, "Okay, I want says, "Okay, I want to see this. You have 20 to see this. You have 20 to see this. You have 20 minutes. Try to minutes. Try to minutes. Try to do it well." And the do it well." And the do it well." And the most confusing most confusing most confusing part here is that part here is that part here is that you don't have an AI agent you don't have an AI agent you don't have an AI agent to code, so you to code, so you to code, so you have to do have to do have to do everything manually because the everything manually because the everything manually because the usage limit has just usage limit has just usage limit has just expired. So, this is the expired. So, this is the expired. So, this is the context of the situation context of the situation context of the situation we find ourselves in. we find ourselves in.

Your task is to find the two best to find the two best to find the two best signs that accurately signs that accurately signs that accurately predict a predict a predict a company's takeover. company's takeover. And then Exa Snapshot And then Exa Snapshot will use this to will use this to will use this to go back in time go back in time go back in time 1 day, 7 days, and 30 days 1 day, 7 days, and 30 days 1 day, 7 days, and 30 days before the takeover and before the takeover and before the takeover and test your test your test your sign. Or on a sign. Or on a sign. Or on a random date if random date if random date if the company wasn't the company wasn't the company wasn't acquired, so there are acquired, so there are acquired, so there are some some some confusing options there. confusing options there. confusing options there. Um, a sign—it's just a Um, a sign—it's just a Um, a sign—it's just a yes/no question that yes/no question that yes/no question that you think you think you think implies implies implies a takeover. You can a takeover. You can a takeover. You can test as much as you test as much as you test as much as you want, there are want, there are want, there are several examples there, and I several examples there, and I several examples there, and I will approach will approach will approach each of you to each of you to each of you to chat and chat and chat and possibly help with the possibly help with the possibly help with the process. Um, and you process. Um, and you process. Um, and you can test can test can test as many as many as many options as you want. options as you want. options as you want. Scoring: You Scoring: You Scoring: You get a plus one get a plus one get a plus one point if your sign point if your sign point if your sign indicates that the company indicates that the company indicates that the company will be acquired will be acquired 170–30 days before it 170–30 days before it 170–30 days before it happens. You happens. You happens. You get minus one get minus one get minus one point if you say there point if you say there point if you say there was a takeover and was a takeover and was a takeover and it didn't happen. And it didn't happen. And there are no fines for omissions or failures, because we are there are no fines for omissions or failures, because we are very loyal. very loyal. very loyal. The best result The best result The best result is recorded is recorded is recorded automatically. You automatically. You automatically. You can try can try can try as many times as many times as many times as you want. I'll as you want. I'll as you want. I'll give us, let's say, give us, let's say, give us, let's say, 30 minutes. So let's start 30 minutes. So let's start 30 minutes. So let's start this right now. You this right now. You this right now. You have 30 minutes to have 30 minutes to have 30 minutes to try to enter the try to enter the try to enter the best sign best sign best sign you want. I will you want. I will you want. I will come to you.

come to you. come to you. Maybe we'll turn on Maybe we'll turn on Maybe we'll turn on some quiet music some quiet music some quiet music to create to create to create the atmosphere of working at an the atmosphere of working at an the atmosphere of working at an investment investment investment company with lo-fi beats. company with lo-fi beats. company with lo-fi beats. Um, and I will approach Um, and I will approach Um, and I will approach each of you for a each of you for a each of you for a conversation. Okay, I'll conversation. Okay, I'll conversation. Okay, I'll show you one quick show you one quick show you one quick example to example to example to demonstrate how demonstrate how demonstrate how this works. If you this works. If you this works. If you are already working and everything is are already working and everything is are already working and everything is working out, don't working out, don't working out, don't stop. But I'll stop. But I'll stop. But I'll just give one just give one just give one example here. So, you example here. So, you example here. So, you see this interface see this interface see this interface in front of you. You will be in front of you. You will be in front of you. You will be prompted to enter a prompted to enter a prompted to enter a new attribute. You new attribute. You new attribute. You can choose one can choose one can choose one of the examples or of the examples or of the examples or enter your own and enter your own and enter your own and click "run click "run ". Um, and the system ". Um, and the system ". Um, and the system will search for will search for will search for data snapshots. This data snapshots. This data snapshots. This worked quickly because worked quickly because worked quickly because the data was cached. This is a the data was cached. This is a performance improvement to performance improvement to show that we show that we show that we tested this tested this tested this in advance. But in advance. But in advance. But the system will test the system will test the system will test this on a set of 10 this on a set of 10 this on a set of 10 companies. So, in companies. So, in companies. So, in this set of this set of this set of companies, some have been companies, some have been companies, some have been absorbed. Some absorbed. Some absorbed. Some were not absorbed. These are were not absorbed. These are were not absorbed. These are variables that can be variables that can be variables that can be confusing. The idea confusing. The idea confusing. The idea is that we is that we is that we will take this will take this will take this specific request. specific request. specific request. So, did this So, did this So, did this company lose its company lose its company lose its CEO or CEO or CFO, or CFO, or appoint interim appoint interim appoint interim management? For management? For management? For all of this, we'll all of this, we'll all of this, we'll go back and go back and go back and do a versioned do a versioned do a versioned search. So, for search. So, for search. So, for DeliveryHero, we see: “ DeliveryHero, we see: “ Hey, there’s been a Hey, there’s been a Hey, there’s been a management change.” Yes. And management change.” Yes. And management change.” Yes. And the company the company the company was eventually absorbed. So was eventually absorbed. So was eventually absorbed. So maybe this is a maybe this is a maybe this is a prognostic prognostic prognostic indicator. Besides Quagen, I'm indicator. Besides Quagen, I'm indicator. Besides Quagen, I'm not sure I'm not sure I'm not sure I'm pronouncing it correctly, pronouncing it correctly, pronouncing it correctly, they also had a change in they also had a change in they also had a change in management, but management, but management, but they weren't ultimately they weren't ultimately they weren't ultimately acquired. Oh no, it's a acquired. Oh no, it's a acquired. Oh no, it's a distorted variable.

distorted variable. distorted variable. So your job is to So your job is to try to think, try to think, try to think, "Hey, what are one or two "Hey, what are one or two "Hey, what are one or two traits that I can traits that I can traits that I can add here that would be add here that would be add here that would be most predictive most predictive ?" So here I'm going to ?" So here I'm going to ?" So here I'm going to add another one. This one add another one. This one add another one. This one sounds like, "Hey, was it sounds like, "Hey, was it sounds like, "Hey, was it reported that reported that reported that they were considering a they were considering a they were considering a sale?" sale?" sale?" Does adding this Does adding this Does adding this improve the result improve the result improve the result or make it worse or make it worse ? And in this case, you ? And in this case, you ? And in this case, you see, "Hey, we've see, "Hey, we've see, "Hey, we've caught a lot of, um, uh caught a lot of, um, uh , we've caught a lot of , we've caught a lot of , we've caught a lot of acquisitions, so we're doing a acquisitions, so we're doing a acquisitions, so we're doing a pretty good job pretty good job pretty good job of of of finding things, but finding things, but finding things, but also, like, hey, we're also, like, hey, we're also, like, hey, we're getting a lot of getting a lot of getting a lot of false positives here." " false positives here." " So we might So we might So we might need to be more need to be more need to be more restrictive in how restrictive in how restrictive in how we combine these two we combine these two we combine these two signs. So here I've signs. So here I've signs. So here I've done a very simple done a very simple done a very simple combination thing. You combination thing. You combination thing. You can choose "or", can choose "or", can choose "or", meaning if either of meaning if either of meaning if either of your two signs gives a " your two signs gives a " your two signs gives a " yes", then we'll yes", then we'll yes", then we'll treat it all as, " treat it all as, " treat it all as, " Yes, you predict Yes, you predict Yes, you predict that the company will be that the company will be that the company will be acquired on this day." acquired on this day." acquired on this day." You can choose "and," You can choose "and," You can choose "and," and if I look, and if I look, and if I look, if I go here, if I go here, if I go here, the score improves a little bit the score improves a little bit the score improves a little bit , which , which , which means that both of these means that both of these means that both of these two features must be two features must be two features must be true. And if true. And if true. And if both of these signs are both of these signs are both of these signs are true for that true for that true for that company on that day, company on that day, company on that day, then you get a " then you get a " yes." Um, it's just an yes." Um, it's just an yes." Um, it's just an announcement. I'll announcement. I'll announcement. I'll get back to the get back to the get back to the leaderboard, but feel leaderboard, but feel leaderboard, but feel free to raise your free to raise your free to raise your hands with any hands with any hands with any questions as we questions as we questions as we move along.

move along. move along. Okay, 10 Okay, 10 Okay, 10 seconds left. If you haven't seconds left. If you haven't seconds left. If you haven't clicked "run" yet clicked "run" yet , it may be too late , it may be too late . Feel free . Feel free . Feel free to change the "and to change the "and /or" combination to try to /or" combination to try to /or" combination to try to get the latest get the latest get the latest results. Phew. results. Phew. results. Phew. Okay, congratulations. So, Okay, congratulations. So, Okay, congratulations. So, Zane, John and Ree, would Zane, John and Ree, would Zane, John and Ree, would you mind you mind you mind joining me joining me joining me on stage if you on stage if you on stage if you won the prize? If not, won the prize? If not, won the prize? If not, we'll move further down the we'll move further down the we'll move further down the list. Can we have some list. Can we have some list. Can we have some loud applause? loud applause? Come on, don't Come on, don't be shy. Don't be be shy. Don't be be shy. Don't be shy. Great shy. Great shy. Great goods and mystery goods and mystery goods and mystery boxes are waiting for boxes are waiting for boxes are waiting for you. Hey, congratulations. Congratulations, you. Hey, congratulations. Congratulations, you. Hey, congratulations. Congratulations, John. Very valuable John. Very valuable John. Very valuable achievements. Hey, congratulations. achievements. Hey, congratulations. achievements. Hey, congratulations. Yes. Um, so yeah, Yes. Um, so yeah, Yes. Um, so yeah, let me let me let me give this away. give this away. Thank you. You will now Thank you. You will now stand in line so stand in line so stand in line so we can take a photo we can take a photo . Okay, we'll take a still . Okay, we'll take a still . Okay, we'll take a still from this from this from this video later. So, video later. So, video later. So, let's just do a little let's just do a little let's just do a little digging and digging and digging and see what see what see what our

our our number one contestant did. So, number one contestant did. So, number one contestant did. So, Zane, is this company Zane, is this company Zane, is this company public? A very public? A very public? A very interesting feature. interesting feature. interesting feature. Number two: is this Number two: is this Number two: is this company different than company different than company different than Qiagen, Vitris, Bank of Cryptos, and Qiagen, Vitris, Bank of Cryptos, and Qiagen, Vitris, Bank of Cryptos, and American Electric Power? Congratulations. American Electric Power? Congratulations. American Electric Power? Congratulations. You are the reason I You are the reason I You are the reason I wanted a sample wanted a sample wanted a sample to test. I to test. I to test. I removed it because I wanted to removed it because I wanted to removed it because I wanted to simplify, but it's a simplify, but it's a simplify, but it's a good example of good example of good example of why backtesting is necessary why backtesting is necessary and why it's worth and why it's worth doing it with a delayed doing it with a delayed doing it with a delayed sample. So I'll sample. So I'll sample. So I'll take this as a take this as a take this as a feature, not a feature, not a feature, not a bug, of not having a bug, of not having a bug, of not having a delayed sample, delayed sample, delayed sample, and it shows why and it shows why and it shows why overtraining is so overtraining is so overtraining is so dangerous, because I don't dangerous, because I don't dangerous, because I don't think this think this think this particular algorithm particular algorithm particular algorithm will be successful in will be successful in will be successful in the future, but who the future, but who the future, but who knows? Number two, Ree. knows? Number two, Ree. knows? Number two, Ree. So, okay, he " So, okay, he " stole" Zayn. Maybe stole" Zayn. Maybe they would sit next to each other. they would sit next to each other. they would sit next to each other. And finally, John. So, has an And finally, John. So, has an active active investor or large investor or large investor or large shareholder publicly shareholder publicly shareholder publicly called for the sale of the called for the sale of the called for the sale of the company as a whole in the last 12 months, is the company as a whole in the last 12 months, is the company's current share price at least company's current share price at least 60% below its 60% below its 60% below its peak price over the peak price over the peak price over the last 5 years, or is last 5 years, or is last 5 years, or is it called it called it called Dominion? A bit of Dominion? A bit of Dominion? A bit of retraining too, but I would retraining too, but I would retraining too, but I would say it would say it would say it would probably work probably work probably work better. So this gives an better. So this gives an better. So this gives an example of how and example of how and example of how and why it is important why it is important why it is important to conduct backtesting to conduct backtesting , right? Because this is, , right? Because this is, , right? Because this is, you know, a somewhat you know, a somewhat you know, a somewhat limited example. I limited example. I limited example. I feel like I regret feel like I regret feel like I regret deleting my deleting my deleting my delayed sample, delayed sample, delayed sample, although I think although I think although I think people would be even people would be even people would be even more confused more confused more confused if I did. But if I did. But if I did. But that gives you an idea that gives you an idea that gives you an idea of ​​why it's valuable, of ​​why it's valuable, of ​​why it's valuable, right? Because if you right? Because if you right? Because if you know what the know what the know what the existing set is, you existing set is, you existing set is, you can fit can fit can fit it to the results, it to the results, it to the results, but the idea is that but the idea is that but the idea is that with this with this with this capability you can capability you can capability you can test it, right? You test it, right? You test it, right? You can go back and

can go back and can go back and say, "Hey, I'm say, "Hey, I'm say, "Hey, I'm going to go through my going to go through my going to go through my list and highlight these 10 list and highlight these 10 list and highlight these 10 companies that did companies that did companies that did something and 20 something and 20 something and 20 companies that didn't, to companies that didn't, to companies that didn't, to train and train and train and improve my improve my improve my model, right?" "And model, right?" "And model, right?" "And make sure it make sure it make sure it continues to perform continues to perform continues to perform better than just on better than just on better than just on publicly available publicly available publicly available datasets." So, datasets." So, datasets." So, let's let's let's summarize. I'm also summarize. I'm also summarize. I'm also going to...Oh my god, there are going to...Oh my god, there are going to...Oh my god, there are so many people here. so many people here. so many people here. I have to I have to I have to scroll for a very long time. I scroll for a very long time. I scroll for a very long time. I will put you all on the will put you all on the will put you all on the thank you page. I should thank you page. I should thank you page. I should be on the be on the be on the thank you page by now. thank you page by now. thank you page by now. On the thank you page On the thank you page On the thank you page you will find a you will find a you will find a link to our link to our link to our documentation. We documentation. We documentation. We love documentation. love documentation. love documentation. You have a link to You have a link to You have a link to our blog. It our blog. It our blog. It reads better than reads better than reads better than documentation and is documentation and is documentation and is more more more people-oriented. And finally, people-oriented. And finally, people-oriented. And finally, you can send you can send you can send me an me an me an email. Now I regret it email. Now I regret it email. Now I regret it because there are 100 people in the room because there are 100 people in the room because there are 100 people in the room , so I'm expecting a , so I'm expecting a , so I'm expecting a bunch of letters. But it is bunch of letters. But it is bunch of letters. But it is also a corporate also a corporate also a corporate function. We have function. We have function. We have public access, but public access, but public access, but if you are interested in if you are interested in if you are interested in getting it for getting it for getting it for your team, your team, your team, write to me and I will write to me and I will write to me and I will connect you with our connect you with our corporate corporate client department. Let's client department. Let's client department. Let's summarize the main summarize the main summarize the main points. First, we points. First, we points. First, we love Exa. Why? Because love Exa. Why? Because love Exa. Why? Because it's a great it's a great it's a great agent web search. agent web search. agent web search. This is profitable. This is very This is profitable. This is very This is profitable. This is very cost-effective cost-effective . This works with . This works with whatever delay I need whatever delay I need whatever delay I need . I want this in . I want this in . I want this in my model.

my model. Secondly, the picture. It Secondly, the picture. It Secondly, the picture. It allows me allows me allows me to test my to test my to test my model on historical model on historical model on historical data. It allows you data. It allows you data. It allows you to see how to see how to see how well Exa improves well Exa improves well Exa improves your model. And your model. And third, be sure third, be sure third, be sure to add delayed to add delayed to add delayed sampling to your sampling to your sampling to your competition, otherwise you competition, otherwise you competition, otherwise you may get may get may get overtraining. Thank you overtraining. Thank you overtraining. Thank you all.