Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data

So, let's begin. So, let's begin. Nice Nice Nice to meet to meet to meet everyone. My name is everyone. My name is everyone. My name is Johan Raju. I am a Johan Raju. I am a Johan Raju. I am a solutions architect at solutions architect at solutions architect at Bright Data. I have been working at the Bright Data. I have been working at the Bright Data. I have been working at the company for about 6 company for about 6 company for about 6 years and am involved in years and am involved in years and am involved in the field of data collection and the field of data collection and the field of data collection and web scraping. web scraping. We originally started We originally started as a proxy as a proxy infrastructure company with the infrastructure company with the infrastructure company with the world's largest world's largest world's largest network of network of network of residential IP residential IP addresses. And then we addresses. And then we addresses. And then we moved on to the moved on to the data collection platform. We are now the data collection platform. We are now the world's largest world's largest web data collection platform, collecting web data collection platform, collecting petabytes of data per petabytes of data per petabytes of data per month, second month, second month, second only to Google. So, a little only to Google. So, a little only to Google. So, a little information for you information for you information for you about where we about where we about where we started and how we started and how we started and how we got to where we are got to where we are got to where we are today. today. today. So, I will cover two So, I will cover two So, I will cover two topics. First, I'll topics. First, I'll topics. First, I'll talk about talk about talk about building AI agents using building AI agents using real- real- time web data, some of the time web data, some of the time web data, some of the tools that are tools that are tools that are commonly commonly commonly used for AI used for AI agents compared to agents compared to agents compared to relying on RAG relying on RAG models with pre- models with pre- models with pre- indexed data indexed data , what the benefits are, what we're , what the benefits are, what we're , what the benefits are, what we're seeing in the industry, seeing in the industry, seeing in the industry, what changes are expected, what changes are expected, what changes are expected, and what's required to make and what's required to make and what's required to make these agents work. And these agents work. And these agents work. And finally, I'll have a finally, I'll have a finally, I'll have a hands-on class hands-on class hands-on class that you can go through that you can go through that you can go through with me, where with me, where with me, where I have a I have a I have a product analysis agent product analysis agent product analysis agent who will essentially take a who will essentially take a who will essentially take a product from Amazon, product from Amazon, product from Amazon, get opinions get opinions get opinions on Reddit, and then on Reddit, and then on Reddit, and then give you an analysis: give you an analysis: give you an analysis: Is this product worth buying Is this product worth buying Is this product worth buying , yes or no? So, , yes or no? So, having worked in the having worked in the scraping industry for a scraping industry for a scraping industry for a long time, I've long time, I've long time, I've noticed one thing: noticed one thing: noticed one thing: when you think about when you think about when you think about building a

building a building a scraping system or collecting scraping system or collecting scraping system or collecting data, what does it data, what does it data, what does it actually take? actually take? The first The first scraping clients I scraping clients I scraping clients I worked with when we were just worked with when we were just worked with when we were just starting out would always starting out would always starting out would always say, "Hey, I want to... say, "Hey, I want to... Here are my target sites Here are my target sites , and here's what I want to , and here's what I want to , and here's what I want to collect, right?" collect, right?" collect, right?" Usually people Usually people Usually people start with start with start with small amounts. This is what small amounts. This is what small amounts. This is what makes up the makes up the makes up the overall traditional overall traditional scraping architecture. scraping architecture. Let's say I'm an Let's say I'm an e-commerce user e-commerce user and I want to collect data from and I want to collect data from and I want to collect data from e-commerce. My e-commerce. My e-commerce. My biggest targets biggest targets biggest targets might be Amazon and Walmart might be Amazon and Walmart , because that's where I can , because that's where I can , because that's where I can find a lot of find a lot of find a lot of products, get products, get products, get price analytics, price analytics, price analytics, review data, and review data, and review data, and general information general information general information for the vast for the vast for the vast majority of products. majority of products. So, when does scraping So, when does scraping become difficult? This is a become difficult? This is a become difficult? This is a change that I really change that I really change that I really see in see in see in customer service today, customer service today, customer service today, happening in the happening in the happening in the industry where it's industry where it's industry where it's not just about not just about not just about Amazon and Walmart anymore. Now you Amazon and Walmart anymore. Now you Amazon and Walmart anymore. Now you need Shopee. So, need Shopee. So, need Shopee. So, now you need now you need now you need stores on Shopify. stores on Shopify. stores on Shopify. Now you need Etsy Now you need Etsy Now you need Etsy and, you know, a lot of and, you know, a lot of and, you know, a lot of other sites. If other sites. If other sites. If we take a step back and we take a step back and we take a step back and think about target think about target think about target sites, it usually takes sites, it usually takes a developer 3 to 5 days to create scrapers for them. And this is not 3 to 5 days to create scrapers for them. And this is not only development, but also only development, but also only development, but also quality control, quality control, quality control, launch in the launch in the launch in the development environment, testing, development environment, testing, development environment, testing, and the entire end-to-end and the entire end-to-end and the entire end-to-end process. If we're process. If we're process. If we're talking about, say, talking about, say, talking about, say, 10 sites, it 10 sites, it 10 sites, it could take some could take some could take some time. Now imagine time. Now imagine time. Now imagine you want to add 100 you want to add 100 you want to add 100 or 1000 sites. This is where

or 1000 sites. This is where or 1000 sites. This is where things start to get things start to get things start to get complicated. Before the complicated. Before the complicated. Before the advent of AI, developers advent of AI, developers advent of AI, developers had to manually had to manually had to manually study the site and study the site and study the site and understand where understand where the selectors were located. Can I the selectors were located. Can I target these target these target these data points via the API? data points via the API? data points via the API? And actually And actually And actually create crawlers create crawlers create crawlers based on this knowledge. I have based on this knowledge. I have based on this knowledge. I have worked with many worked with many scrapers for over 10 scrapers for over 10 years, managing and years, managing and years, managing and maintaining these maintaining these maintaining these crawlers. But when crawlers. But when crawlers. But when it comes to it comes to it comes to scaling, you scaling, you scaling, you think, "Okay, I'm think, "Okay, I'm think, "Okay, I'm adding new sites, but adding new sites, but adding new sites, but I also need to I also need to I also need to manage the manage the manage the scrapers that are already there." scrapers that are already there." Let's say Amazon Let's say Amazon makes a change tomorrow. And two makes a change tomorrow. And two makes a change tomorrow. And two other sites will do the same thing other sites will do the same thing other sites will do the same thing , and you manage , and you manage , and you manage a thousand sites. This a thousand sites. This a thousand sites. This requires much requires much requires much more engineering more engineering more engineering resources and time to resources and time to resources and time to understand what exactly understand what exactly understand what exactly has changed. How can I has changed. How can I has changed. How can I counteract this? How to counteract this? How to counteract this? How to fix this? And does fix this? And does fix this? And does it affect my it affect my it affect my customers, my SLAs, and customers, my SLAs, and customers, my SLAs, and data quality? Previously, data quality? Previously, data quality? Previously, we had a "bottleneck" we had a "bottleneck" we had a "bottleneck" in the form of the need in the form of the need in the form of the need to scale resources to scale resources , and to manage , and to manage , and to manage a thousand sites, we a thousand sites, we a thousand sites, we needed two to needed two to needed two to four times as many four times as many four times as many engineers. But now, engineers. But now, engineers. But now, in the era of AI, this is no longer in the era of AI, this is no longer in the era of AI, this is no longer necessary, because there are agents necessary, because there are agents necessary, because there are agents that can that can that can independently find independently find independently find sites and create sites and create sites and create scrapers for them scrapers for them scrapers for them much easier than a much easier than a much easier than a whole team of whole team of whole team of developers. This is a developers. This is a developers. This is a classic example of an classic example of an classic example of an e-commerce site e-commerce site targeting targeting page selectors.

page selectors. page selectors. As you can see, in this As you can see, in this As you can see, in this example we example we example we focus only on focus only on focus only on the price and name. What the price and name. What the price and name. What happens if the price happens if the price happens if the price changes? If the changes? If the changes? If the price selector price selector price selector is changed, say, is changed, say, is changed, say, tomorrow. This will require tomorrow. This will require tomorrow. This will require some downtime some downtime some downtime on your part because, you on your part because, you know, if you're the one know, if you're the one know, if you're the one parsing the site, parsing the site, parsing the site, you're not getting you're not getting you're not getting the price right now, and that could the price right now, and that could the price right now, and that could affect the rest of affect the rest of affect the rest of your pipeline where your pipeline where your pipeline where you might be you might be you might be parsing products and parsing products and parsing products and so on. This so on. This so on. This would normally require the would normally require the would normally require the team to step in, team to step in, team to step in, fix it, and then fix it, and then fix it, and then possibly recover the possibly recover the possibly recover the lost data. But lost data. But lost data. But now, with the help of AI now, with the help of AI agents, this can be agents, this can be agents, this can be automated and automated and automated and performed performed performed automatically. So automatically. So automatically. So when you create when you create when you create parsers, it's always worth parsers, it's always worth parsers, it's always worth remembering that every remembering that every remembering that every website is unique. website is unique. Here are the sites from my Here are the sites from my example, like Amazon and example, like Amazon and example, like Amazon and Walmart, have different Walmart, have different Walmart, have different layouts, even though both layouts, even though both layouts, even though both may have data may have data such as price, title, description. such as price, title, description. Therefore, you Therefore, you need to need to need to create a separate create a separate create a separate parser for each site. And more parser for each site. And more parser for each site. And more importantly, importantly, importantly, every site strives every site strives every site strives to protect its data.

to protect its data. Sites usually Sites usually want to make sure no want to make sure no want to make sure no one else has the one else has the one else has the same product same product same product analytics as them, analytics as them, analytics as them, or, depending on the type of or, depending on the type of or, depending on the type of data, try to data, try to data, try to keep it secure. keep it secure. They always They always implement various implement various implement various types of types of types of bot protection and captchas to bot protection and captchas to bot protection and captchas to confuse confuse confuse parsers and bots that parsers and bots that parsers and bots that try to collect try to collect try to collect this data. I have seen all this data. I have seen all this data. I have seen all possible types of possible types of possible types of locks. I've seen both the locks. I've seen both the locks. I've seen both the simplest sites simplest sites simplest sites that can be that can be that can be accessed through data accessed through data centers, and sites centers, and sites centers, and sites protected by Cloudflare, Akamai, protected by Cloudflare, Akamai, protected by Cloudflare, Akamai, etc., where you really etc., where you really etc., where you really need a resident need a resident need a resident IP address, a real IP address, a real IP address, a real browser, and you have to browser, and you have to browser, and you have to look like a look like a look like a real user real user . You know, if you . You know, if you click too quickly on a click too quickly on a page or page or page or scroll too quickly, it scroll too quickly, it scroll too quickly, it can activate can activate can activate bot protection. bot protection. bot protection. If you block If you block If you block images or something images or something images or something like that. As I like that. As I like that. As I mentioned earlier, mentioned earlier, mentioned earlier, engineering was a “ engineering was a “ bottleneck” until the bottleneck” until the bottleneck” until the field of field of field of AI came along, and now we have AI AI came along, and now we have AI agents ready agents ready agents ready to build crawlers. to build crawlers. to build crawlers. And you would have to have a And you would have to have a And you would have to have a deep understanding of the deep understanding of the deep understanding of the structure of each structure of each structure of each site. Many sites, site. Many sites, site. Many sites, as I said, because as I said, because as I said, because their layouts are their layouts are their layouts are so different, so different, so different, try to make it try to make it as difficult as possible for you to collect data as difficult as possible for you to collect data . Therefore, you . Therefore, you . Therefore, you need a deep need a deep need a deep level of expert level of expert level of expert knowledge in this field knowledge in this field knowledge in this field to access to access to access this data. Here is a this data. Here is a this data. Here is a diagram of what a diagram of what a diagram of what a traditional traditional traditional parsing architecture looks like parsing architecture looks like . So, to . So, to . So, to go over this briefly: on the go over this briefly: on the go over this briefly: on the left side we have, left side we have, left side we have, you know, the subject, you know, the subject, you know, the subject, which is what triggers

which is what triggers which is what triggers this crawl. Some people this crawl. Some people this crawl. Some people like like like to use to use to use schedulers, schedulers, schedulers, you know, cron jobs, you know, cron jobs, you know, cron jobs, whatever, when they whatever, when they whatever, when they want to get want to get want to get data daily, weekly, data daily, weekly, data daily, weekly, etc., and that kicks the etc., and that kicks the etc., and that kicks the process right up process right up process right up to the next level. to the next level. Some Some prefer certain prefer certain prefer certain settings, settings, settings, certain URLs, certain URLs, certain URLs, the idea is the same. This the idea is the same. This the idea is the same. This is passed is passed is passed directly to the directly to the directly to the proxy layer. So, a proxy layer. So, a proxy layer. So, a proxy is essentially proxy is essentially proxy is essentially your way of hiding your way of hiding your way of hiding yourself from being yourself from being yourself from being detected. There are detected. There are detected. There are different types of proxies different types of proxies different types of proxies that can be that can be that can be used: used: used: static IP addresses, static IP addresses, static IP addresses, for example, from data for example, from data centers or ISPs, or centers or ISPs, or centers or ISPs, or rotating resident rotating resident rotating resident IP addresses originating IP addresses originating IP addresses originating from laptops, from laptops, from laptops, tablets, tablets, tablets, TVs, which TVs, which TVs, which look look look more more more authentic to websites because they authentic to websites because they authentic to websites because they “understand”: “Okay, “understand”: “Okay, “understand”: “Okay, this is registered to a this is registered to a this is registered to a real device.” " real device.” " We will let them in." And We will let them in." And We will let them in." And there are also tools with there are also tools with there are also tools with built-in anti-bot built-in anti-bot capabilities, such as the Web capabilities, such as the Web capabilities, such as the Web Unlocker browser API from Bright Data Unlocker browser API from Bright Data , which is about , which is about browser automation. Where the browser automation. Where the product itself controls product itself controls product itself controls the process: “Okay, is the process: “Okay, is the process: “Okay, is my request blocking?

my request blocking? »"Should I »"Should I »"Should I try again?"" try again?"" Am I using the Am I using the Am I using the correct IP address?" correct IP address?" correct IP address?" This is all part of the This is all part of the This is all part of the level when we level when we level when we talk about anti-bot talk about anti-bot protection. Then, the protection. Then, the protection. Then, the next level is next level is next level is parsing the data and parsing the data and parsing the data and checking whether the checking whether the checking whether the data is authentic. data is authentic. This is an important part of the This is an important part of the process after process after process after you have landed on the you have landed on the you have landed on the site and collected data. site and collected data. site and collected data. Do I get a price 90% of the Do I get a price 90% of the time and do I want to time and do I want to time and do I want to make sure I make sure I make sure I get it get it get it that often? Do that often? Do that often? Do I need I need I need feedback data? Do feedback data? Do feedback data? Do I need ratings, you know I need ratings, you know , things like that? , things like that? , things like that? You need to You need to You need to think, “Okay, how do think, “Okay, how do think, “Okay, how do I make sure that I make sure that I make sure that as the as the as the data goes through the pipeline, data goes through the pipeline, data goes through the pipeline, it is accurately it is accurately it is accurately checked against the checked against the checked against the validation rules? validation rules? » Then we move on » Then we move on » Then we move on to the post-processing stage to the post-processing stage to the post-processing stage at the data entry level. at the data entry level. at the data entry level. So, removing So, removing So, removing duplicate records, duplicate records, duplicate records, enriching your enriching your enriching your data, data, data, preprocessing batches. This preprocessing batches. This preprocessing batches. This happens when happens when happens when , you know, some , you know, some , you know, some want to deliver them want to deliver them want to deliver them to cloud storage, to cloud storage, to cloud storage, others to their others to their others to their internal systems. internal systems. internal systems. Once they Once they Once they get there, what do you get there, what do you get there, what do you do with that data?

do with that data? do with that data? How do you ensure that you How do you ensure that you How do you ensure that you don't provide the don't provide the don't provide the same record twice or same record twice or same record twice or provide "dirty" provide "dirty" provide "dirty" data? So, you know, data? So, you know, data? So, you know, moving data in a moving data in a moving data in a medallion medallion medallion architecture: bronze, architecture: bronze, architecture: bronze, silver, and then the silver, and then the silver, and then the final migration final migration final migration to your database. And to your database. And to your database. And if something in this if something in this if something in this process fails, you process fails, you process fails, you need to start need to start need to start this whole process over again this whole process over again this whole process over again . And that's exactly what . And that's exactly what . And that's exactly what I was talking about: if something I was talking about: if something I was talking about: if something breaks at the breaks at the breaks at the anti-bot level, or the anti-bot level, or the anti-bot level, or the selectors change selectors change , you have to , you have to , you have to start the whole process over start the whole process over start the whole process over again. Sometimes when again. Sometimes when again. Sometimes when filling in data, you filling in data, you filling in data, you may not meet an may not meet an may not meet an SLA or something similar. SLA or something similar. So now I'm So now I'm going to talk specifically going to talk specifically going to talk specifically about artificial intelligence. about artificial intelligence. How many of you How many of you today today today use use use tools tools tools like Claude, GPT, like Claude, GPT, any real AI any real AI brains? Okay, cool. brains? Okay, cool. How many of you have entered How many of you have entered a query and seen the words a query and seen the words a query and seen the words "thinking", "looking for "thinking", "looking for "thinking", "looking for an answer" or " an answer" or " searching the Internet"? searching the Internet"? Okay, cool. Perfectly.

Okay, cool. Perfectly. So here you see how So here you see how So here you see how I've divided it into two I've divided it into two I've divided it into two parts: on the left is parts: on the left is parts: on the left is where you see the AI where you see the AI where you see the AI actually thinking or actually thinking or actually thinking or searching for information on searching for information on searching for information on the web. That's when he the web. That's when he the web. That's when he decides: yes, for decides: yes, for decides: yes, for the request you gave, the request you gave, the request you gave, I need to go and I need to go and I need to go and find the answer. I find the answer. I find the answer. I can't use can't use can't use my pre- my pre- my pre- trained data. So trained data. So trained data. So here's a simple here's a simple here's a simple example of a prompt: " example of a prompt: " When is the best time When is the best time When is the best time to travel to San to travel to San Francisco?" If I Francisco?" If I Francisco?" If I say so, the AI ​​will say so, the AI ​​will say so, the AI ​​will usually turn usually turn usually turn to the RAG system, to the RAG system, to the RAG system, asynchronously asynchronously asynchronously using its using its using its trained data, which trained data, which trained data, which says that the best says that the best says that the best time is summer. And he time is summer. And he time is summer. And he will give you this will give you this will give you this answer. Now, answer. Now, answer. Now, if I change the query and if I change the query and if I change the query and ask, "When is the ask, "When is the ask, "When is the best time to best time to best time to travel to San travel to San Francisco this year?" Francisco this year?" » Then the AI ​​has to go » Then the AI ​​has to go » Then the AI ​​has to go online and online and online and see what's see what's see what's happening to happening to happening to understand the situation. understand the situation. understand the situation. Is Is Is the user asking me about the user asking me about current events? How do I current events? How do I make make make real-time decisions real-time decisions real-time decisions this year to this year to this year to advise them on the advise them on the advise them on the best time to best time to best time to travel? So these are two travel? So these are two travel? So these are two different paths that AI different paths that AI different paths that AI can take to give can take to give can take to give you an answer. Some you an answer. Some you an answer. Some things I notice things I notice are that responses are that responses are that responses can sometimes get can sometimes get can sometimes get cached.

cached. cached. For example, if someone For example, if someone For example, if someone asks, "What's the asks, "What's the asks, "What's the weather like in New weather like in New York today?" For example, York today?" For example, York today?" For example, several times a day, several times a day, several times a day, this might be a good this might be a good this might be a good answer to answer to answer to cache because you cache because you only need one only need one person to ask it. And you can person to ask it. And you can person to ask it. And you can know what the know what the know what the weather will be like for the day and weather will be like for the day and weather will be like for the day and just provide that just provide that just provide that information. So you information. So you information. So you don't have to don't have to don't have to spend tokens or, spend tokens or, spend tokens or, you know, money to you know, money to you know, money to go go go online and look for this online and look for this online and look for this data. So, as I data. So, as I data. So, as I mentioned, there are two ways mentioned, there are two ways mentioned, there are two ways to collect these to collect these to collect these analytics. analytics. Synchronous path and Synchronous path and asynchronous path. asynchronous path. asynchronous path. Both are useful for Both are useful for Both are useful for different occasions. The different occasions. The different occasions. The synchronous way is synchronous way is synchronous way is if you really if you really if you really need to go need to go need to go online and find online and find online and find what you're looking for, and the what you're looking for, and the what you're looking for, and the asynchronous way is if asynchronous way is if asynchronous way is if you can just you can just you can just rely on rely on rely on pre-trained pre-trained pre-trained data. And both data. And both data. And both require different require different require different settings. We also settings. We also settings. We also offer a offer a offer a tool, which tool, which tool, which I'll talk about on the I'll talk about on the I'll talk about on the next slide, where next slide, where next slide, where you can you can you can use use use BrightData tools BrightData tools BrightData tools to go to go to go online and collect online and collect online and collect data. Another thing data. Another thing data. Another thing I noticed is that if you I noticed is that if you I noticed is that if you go to any go to any go to any site, and if you're site, and if you're site, and if you're familiar with robots.txt, familiar with robots.txt, familiar with robots.txt, many sites many sites many sites usually block usually block usually block bots like Claude or bots like Claude or bots like Claude or GPT. So if you GPT. So if you GPT. So if you ask these engines a ask these engines a ask these engines a question and they question and they question and they try to find try to find try to find data on the internet, data on the internet, data on the internet, they may not be able to they may not be able to they may not be able to get it from certain get it from certain get it from certain sites because they sites because they sites because they block access for block access for block access for GPT or Claude. This is where we GPT or Claude. This is where we GPT or Claude. This is where we come in handy.

come in handy. Our advantage is Our advantage is that we can open up that we can open up that we can open up access to the web: access to the web: access to the web: whether whether whether you have a scraper, an you have a scraper, an you have a scraper, an agent, or you agent, or you agent, or you use something use something use something like Claude or GPT, like Claude or GPT, like Claude or GPT, we have the tools to we have the tools to we have the tools to allow them to allow them to allow them to go online and go online and go online and collect data without collect data without collect data without blocking. And we have been blocking. And we have been blocking. And we have been masters at this for a masters at this for a masters at this for a long time. long time. long time. Now I will talk about the Now I will talk about the Web Model Context Web Model Context Protocol. This is MCP. It Protocol. This is MCP. It Protocol. This is MCP. It usually usually usually integrates with integrates with integrates with solutions such as solutions such as solutions such as Claude code, LangChain, and Claude code, LangChain, and Claude code, LangChain, and others. You can others. You can others. You can think of it as a brain that think of it as a brain that needs to be needs to be needs to be interacted with. And this interacted with. And this interacted with. And this brain essentially brain essentially brain essentially connects to connects to connects to our MCP endpoint, our MCP endpoint, our MCP endpoint, which encompasses over 60 APIs which encompasses over 60 APIs which encompasses over 60 APIs that we control in that we control in that we control in our library. our library. our library. You You You can see some examples here, can see some examples here, can see some examples here, such as a search such as a search such as a search engine. So, these are the engine. So, these are the engine. So, these are the Google results. Google results. Google results. Scraping pages as Scraping pages as Scraping pages as markdown files markdown files markdown files and scraping pages and scraping pages and scraping pages as HTML. So, if I as HTML. So, if I as HTML. So, if I want to target want to target want to target specific sites like Amazon specific sites like Amazon , Reddit, LinkedIn, or Instagram, that's , Reddit, LinkedIn, or Instagram, that's , Reddit, LinkedIn, or Instagram, that's included as well included as well included as well . Actually, as an . Actually, as an . Actually, as an example: 2 weeks ago example: 2 weeks ago example: 2 weeks ago I was working with someone I was working with someone I was working with someone who was trying to who was trying to who was trying to create a create a social media monitoring tool social media monitoring tool using MCP. One of using MCP. One of using MCP. One of the requests was: "Hello the requests was: "Hello , this is my name. I want , this is my name. I want , this is my name. I want you to find you to find you to find all my all my all my social social social media profiles on the Internet and tell me media profiles on the Internet and tell me media profiles on the Internet and tell me how many how many how many followers I have." So he

followers I have." So he followers I have." So he used it on used it on used it on Claude code. The system Claude code. The system Claude code. The system made a call to made a call to made a call to our MCP, first our MCP, first our MCP, first using a using a using a search engine, search engine, search engine, found results, found results, found results, his Instagram, his Instagram, his Instagram, TikTok, LinkedIn pages, returned TikTok, LinkedIn pages, returned TikTok, LinkedIn pages, returned them, made an API request them, made an API request them, made an API request to our to our to our data pipeline, and then provided data pipeline, and then provided data pipeline, and then provided results regarding the results regarding the results regarding the number of number of number of followers and followers and followers and returned returned returned context to him. That's the context to him. That's the context to him. That's the power of what it can power of what it can power of what it can do, and how much of the do, and how much of the do, and how much of the internet internet internet MCP can open up. This MCP can open up. This MCP can open up. This also works for also works for browser automation. Let's say browser automation. Let's say creating tasks creating tasks creating tasks that require a that require a that require a browser, like clicks, browser, like clicks, browser, like clicks, scrolling, maybe scrolling, maybe scrolling, maybe automating automating automating an order on DoorDash, or an order on DoorDash, or an order on DoorDash, or automating on a automating on a automating on a travel site travel site travel site to find the best to find the best to find the best time to buy time to buy time to buy a plane ticket or a plane ticket or a plane ticket or book a flight. book a flight. book a flight. It's very profitable for such things It's very profitable for such things It's very profitable for such things . I'll show you a . I'll show you a . I'll show you a short video about short video about short video about how MCP works. how MCP works. Let me Let me open On the left you open On the left you open On the left you see Claude with the see Claude with the Bright Data MCP configured, and on the right Claude without the Bright Data MCP.

Claude without the Bright Data MCP. You can see You can see that the request is the same: that the request is the same: that the request is the same: use the browser use the browser use the browser to get the content of to get the content of to get the content of this page—this is a this page—this is a this page—this is a link to Glassdoor. link to Glassdoor. On the left, you can see that with On the left, you can see that with Bright Data's MCP, he Bright Data's MCP, he Bright Data's MCP, he successfully went to successfully went to successfully went to Glassdoor and got Glassdoor and got Glassdoor and got results, while results, while results, while on the right, access to on the right, access to on the right, access to Glassdoor was Glassdoor was Glassdoor was blocked. This is a blocked. This is a blocked. This is a perfect example perfect example perfect example of how sites of how sites of how sites like Glassdoor, which I have like Glassdoor, which I have like Glassdoor, which I have worked with in the context of worked with in the context of worked with in the context of data collection, are very data collection, are very data collection, are very secure resources secure resources . This is a great example . This is a great example . This is a great example of why MCP is of why MCP is of why MCP is useful when useful when useful when combined with something combined with something combined with something like Claude, GPT, or like Claude, GPT, or like Claude, GPT, or any other any other any other intelligent intelligent intelligent model you model you model you want to run it on. want to run it on. Okay, now I'll Okay, now I'll tell you about a tell you about a tell you about a usage example. usage example. Working with AI companies, I see companies, I see companies, I see examples of clients I examples of clients I examples of clients I speak with who are speak with who are speak with who are using Bright using Bright using Bright Data in some way.

Data in some way. This includes, for example, This includes, for example, training models, training models, training models, retraining or retraining or retraining or enriching them, providing AI enriching them, providing AI agents with agents with agents with real-time access, as we real-time access, as we real-time access, as we mentioned, or mentioned, or mentioned, or enriching RAGs. So, in enriching RAGs. So, in enriching RAGs. So, in this example, I'm going to this example, I'm going to this example, I'm going to talk about talk about talk about vacancies, okay? vacancies, okay? vacancies, okay? Let's say someone Let's say someone Let's say someone comes to me and comes to me and comes to me and says, "Listen, I says, "Listen, I says, "Listen, I need data about need data about need data about vacancies, I want to vacancies, I want to vacancies, I want to train a model train a model train a model based on this data. based on this data. based on this data. For example, for For example, for For example, for resume selection, as resume selection, as resume selection, as an option." So, it's an option." So, it's important to understand that important to understand that when you think about when you think about when you think about job data, it's not the job data, it's not the job data, it's not the same as collecting other same as collecting other same as collecting other forms of data that you can forms of data that you can forms of data that you can just download, just download, just download, forget about, or forget about, or forget about, or train a model on train a model on . You need . You need . You need constant updates constant updates constant updates on job openings. Which on job openings. Which on job openings. Which companies companies companies are posting, how many are posting, how many are posting, how many vacancies are available, vacancies are available, vacancies are available, what is the salary, location, what is the salary, location, what is the salary, location, etc. Therefore, it is important that etc. Therefore, it is important that etc. Therefore, it is important that the data is the data is the data is structured and structured and structured and has consistent has consistent has consistent output. Actual output. Actual output. Actual performance performance performance is also key because is also key because is also key because if you're building a if you're building a if you're building a model to analyze model to analyze model to analyze trends or anything trends or anything trends or anything else in the labor market else in the labor market , you need the , you need the , you need the data to come in data to come in data to come in constantly. It also constantly. It also constantly. It also needs to be very needs to be very needs to be very reliable because reliable because reliable because if something breaks if something breaks if something breaks or has an outage, or has an outage, or has an outage, it can negatively it can negatively it can negatively impact your entire impact your entire impact your entire data pipeline. As data pipeline. As data pipeline. As for time for time for time limits, it's not limits, it's not limits, it's not just about getting just about getting just about getting job data job data job data starting starting starting today. Some people today. Some people today. Some people need archived need archived need archived job data,

job data, job data, for example, from past for example, from past for example, from past months or years. So it's months or years. So it's months or years. So it's very important to cover very important to cover very important to cover all of this, because if you all of this, because if you all of this, because if you want to get the full want to get the full want to get the full picture, for example in picture, for example in picture, for example in this example with this example with this example with vacancies, to vacancies, to vacancies, to understand how to train understand how to train understand how to train the model to select the the model to select the the model to select the best candidates best candidates best candidates for the position. You need to for the position. You need to for the position. You need to understand how understand how trends have changed. Maybe trends have changed. Maybe salaries for salaries for software engineers have increased over time, and you have increased over time, and you need to understand need to understand need to understand why, what the demand is, what the why, what the demand is, what the why, what the demand is, what the current offerings are, and current offerings are, and current offerings are, and based on that you based on that you based on that you can build your can build your can build your model. This is an example of a model. This is an example of a model. This is an example of a use case, such as an AI use case, such as an AI agent for agent for agent for CV selection. So, that's exactly what CV selection. So, that's exactly what CV selection. So, that's exactly what we we we discussed here. So, you discussed here. So, you discussed here. So, you immediately need immediately need immediately need real- real- real- time access to go through the whole time access to go through the whole time access to go through the whole process and, you know, process and, you know, process and, you know, collect the data and be collect the data and be collect the data and be able to train able to train able to train this model with this model with this model with it. Good. So now it. Good. So now it. Good. So now I will move on to a I will move on to a I will move on to a practical exercise. practical exercise. So, I have a So, I have a Google notebook. And if any of Google notebook. And if any of Google notebook. And if any of you are doing this you are doing this you are doing this with me, there are two things with me, there are two things with me, there are two things you need to you need to you need to run this script.

run this script. run this script. The first is the Bright Data API token The first is the Bright Data API token The first is the Bright Data API token . And the second is the Gemini API key . And the second is the Gemini API key . And the second is the Gemini API key . I have a list of . I have a list of . I have a list of instructions on Google Drive instructions on Google Drive instructions on Google Drive on how on how on how to set up both. And to set up both. And , in fact, I've already , in fact, I've already , in fact, I've already prepared everything for you prepared everything for you prepared everything for you so you can so you can so you can launch. So I'll launch. So I'll launch. So I'll tell you a little bit about what tell you a little bit about what we do here. we do here. we do here. First, we First, we First, we access the access the Bright Data API endpoint to Bright Data API endpoint to Bright Data API endpoint to collect data from Amazon. We collect data from Amazon. We collect data from Amazon. We collect the response collect the response collect the response from Amazon, then from Amazon, then from Amazon, then call the Reddit API, call the Reddit API, call the Reddit API, get get get the raw data from the raw data from the raw data from Reddit, and then Reddit, and then Reddit, and then perform AI perform AI analysis on it. Now, before we analysis on it. Now, before we analysis on it. Now, before we launch this, I launch this, I launch this, I want to show you the want to show you the want to show you the exact data we exact data we exact data we get from get from get from each of these APIs. So, each of these APIs. So, each of these APIs. So, with Amazon, we get with Amazon, we get product-level data. I chose a product-level data. I chose a thermos mug as an thermos mug as an thermos mug as an example. And, in fact, we example. And, in fact, we example. And, in fact, we have absolutely all the have absolutely all the have absolutely all the data about this product. data about this product. That is, price, That is, price, image, image, image, link, etc. I'll link, etc. I'll link, etc. I'll show you this once show you this once show you this once it works.

it works. My screen is a little small , so it might not , so it might not , so it might not show everything in show everything in show everything in full, but we full, but we full, but we also have the Reddit API, and in also have the Reddit API, and in also have the Reddit API, and in this script I this script I this script I created, I'm going to run a created, I'm going to run a created, I'm going to run a keyword search keyword search keyword search for this for this for this particular particular particular thermos mug. And I'll be thermos mug. And I'll be thermos mug. And I'll be putting out posts that putting out posts that putting out posts that , you know, , you know, , you know, are about whether are about whether are about whether I should I should I should buy this or not. Is buy this or not. Is buy this or not. Is it worth it? Isn't it it worth it? Isn't it it worth it? Isn't it worth it? What worth it? What worth it? What do people say? So do people say? So do people say? So let's get this going let's get this going let's get this going live. live. Okay, we've uploaded Okay, we've uploaded our keys. Okay, we our keys. Okay, we our keys. Okay, we got the data from Amazon.

got the data from Amazon. You can see the price , rating, other , rating, other , rating, other sellers who sellers who sellers who sell it. All the sell it. All the sell it. All the factual data about the factual data about the factual data about the product itself, product itself, product itself, the characteristics, the characteristics, the characteristics, in short—everything you in short—everything you in short—everything you want. Now we'll want. Now we'll want. Now we'll run a Reddit search run a Reddit search for the Stanley Quencher thermos mug for the Stanley Quencher thermos mug just to just to just to get reviews on the get reviews on the get reviews on the product itself. And while that's product itself. And while that's product itself. And while that's working, I'll working, I'll working, I'll tell you a little bit about what tell you a little bit about what tell you a little bit about what I mentioned earlier: the I mentioned earlier: the model context protocol, MCP servers. model context protocol, MCP servers. So, we have different So, we have different So, we have different levels. I know I've levels. I know I've levels. I know I've talked about things talked about things like like like browser automation and search browser automation and search browser automation and search engines, but we also engines, but we also engines, but we also cover e- cover e- cover e- commerce, social commerce, social commerce, social media, financial media, financial media, financial analytics, analytics, analytics, research, and research, and research, and business intelligence.

business intelligence. So, all of this can be So, all of this can be connected to the connected to the connected to the MCP access point. You know, like the MCP access point. You know, like the MCP access point. You know, like the App Store. And we App Store. And we App Store. And we continue to add continue to add continue to add new features. new features. new features. We currently have over 60 We currently have over 60 We currently have over 60 tools tools tools available for available for available for use by AI use by AI agents. Some people agents. Some people agents. Some people prefer prefer prefer to use to use to use scrapers themselves, scrapers themselves, scrapers themselves, but we let but we let but we let your AI agent your AI agent your AI agent choose choose choose which tools which tools which tools to use. So it will be to use. So it will be to use. So it will be much more much more much more convenient and easier. convenient and easier. As long as it works, I can As long as it works, I can answer answer any questions any questions any questions you have. Yes, the question you have. Yes, the question you have. Yes, the question was how was how was how can you collect can you collect can you collect resume-level data that resume-level data that resume-level data that regular bots regular bots regular bots can't, right?

can't, right? So, the advantage of Bright Data that allows So, the advantage of Bright Data that allows us to us to us to do this, do this, do this, while regular bots while regular bots while regular bots don't, is that don't, is that don't, is that we have the largest proxy we have the largest proxy we have the largest proxy infrastructure in the world infrastructure in the world infrastructure in the world . And when it . And when it . And when it comes to multiple comes to multiple comes to multiple queries, maybe queries, maybe queries, maybe multiple days of multiple days of multiple days of page crawling, page crawling, page crawling, like Glassdoor, like I like Glassdoor, like I like Glassdoor, like I gave in the example, it gave in the example, it gave in the example, it can work, can work, can work, right? But right? But right? But let's say after let's say after let's say after a while Glassdoor a while Glassdoor a while Glassdoor discovers, "Hey, it's discovers, "Hey, it's discovers, "Hey, it's using a data using a data center IP, it's center IP, it's center IP, it's accessing the accessing the accessing the page, but no page, but no page, but no real action is real action is real action is being taken on it," and being taken on it," and being taken on it," and then they start then they start then they start adding all these layers of adding all these layers of adding all these layers of bot protection. bot protection. That's where companies like us, Bright companies like us, Bright Data, come into play, because you Data, come into play, because you Data, come into play, because you need to not only have a need to not only have a need to not only have a scraper or bot, but also scraper or bot, but also scraper or bot, but also think about how to bypass think about how to bypass think about how to bypass site protection, CAPTCHA, and site protection, CAPTCHA, and site protection, CAPTCHA, and how to do it how to do it how to do it consistently without consistently without consistently without causing a causing a causing a lockout. We've lockout. We've lockout. We've been doing this for been doing this for been doing this for years, and we have years, and we have years, and we have tools that are tools that are tools that are integrated into this integrated into this integrated into this anti- anti- anti- bot logic, where we bot logic, where we bot logic, where we dramatically change the dramatically change the dramatically change the approach to the site approach to the site approach to the site every time. But every time. But every time. But sometimes people who are sometimes people who are sometimes people who are just starting out in just starting out in just starting out in scraping scraping try to do try to do try to do it themselves and it themselves and it themselves and manage the manage the manage the process on their own. You also process on their own. You also process on their own. You also need to factor in need to factor in need to factor in engineering time and engineering time and engineering time and resources, as trying resources, as trying resources, as trying to figure out exactly what to figure out exactly what to figure out exactly what changed on the site also changed on the site also changed on the site also takes time, and takes time, and takes time, and probably even probably even probably even more than running the more than running the more than running the scraper itself. Okay scraper itself. Okay , great. Looks like we , great. Looks like we , great. Looks like we got the data. So, I got the data. So, I got the data. So, I can show you.

can show you. He grabbed several He grabbed several Reddit posts about the Reddit posts about the Reddit posts about the Tumblr itself. So, Tumblr itself. So, Tumblr itself. So, a few comments a few comments a few comments about, you know, what about, you know, what about, you know, what people have said about the people have said about the people have said about the glass itself. It seems like people glass itself. It seems like people glass itself. It seems like people were saying it was embarrassing were saying it was embarrassing were saying it was embarrassing and were just and were just and were just obsessed, but here's the obsessed, but here's the obsessed, but here's the real real real answer. So, you answer. So, you answer. So, you can see if can see if can see if it is worth buying. It's a it is worth buying. It's a it is worth buying. It's a good good good discount, you know, discount, you know, discount, you know, the reviews are mostly the reviews are mostly the reviews are mostly positive, but there are positive, but there are positive, but there are a few conflicting a few conflicting a few conflicting opinions. Demand is high, opinions. Demand is high, opinions. Demand is high, and Amazon has low inventory, and Amazon has low inventory, and Amazon has low inventory, as the data as the data as the data confirms. And, confirms. And, confirms. And, you know, a few more you know, a few more you know, a few more analytical points analytical points analytical points about the glass itself about the glass itself about the glass itself and the number of and the number of and the number of reviews about it. reviews about it. reviews about it. He actually He actually He actually analyzed all analyzed all analyzed all the data received through the data received through the data received through our API and helped our API and helped our API and helped decide: yes, decide: yes, decide: yes, should I should I should I buy this or not? This is buy this or not? This is buy this or not? This is just one example of just one example of just one example of using data, using data, using data, starting with Amazon, starting with Amazon, starting with Amazon, running it through running it through running it through Reddit, and then Reddit, and then Reddit, and then making decisions, making decisions, making decisions, but think about it more but think about it more but think about it more broadly, it doesn't have to broadly, it doesn't have to broadly, it doesn't have to be be be e-commerce.

e-commerce. e-commerce. This could be any This could be any use case; As long as use case; As long as we have the tools to we have the tools to we have the tools to access the access the access the page and page and page and data, it's up to data, it's up to data, it's up to you what you what you what logic to choose and how to logic to choose and how to logic to choose and how to build that build that build that pipeline. And build pipeline. And build pipeline. And build it with it with it with scalability. Because scalability. Because scalability. Because we are best at we are best at we are best at providing data at providing data at providing data at scale. scale. Steeply. I think we Steeply. I think we can end this here. can end this here. can end this here. If you have any If you have any If you have any questions, I will be questions, I will be questions, I will be available, or you can available, or you can available, or you can visit the visit the visit the Bright Data booth. Thank you all for Bright Data booth. Thank you all for Bright Data booth. Thank you all for your time.