Navigate Select ESC Close

How I Scrape Any Site Using This MCP Server + Claude, LangGraph & AI Agents

2025-05-07 Education
41.9k
1.1k
43
Tech With Tim
Tech With Tim
2.0m subscribers

Unlock all features

FREE: Get instant access to 10 AI summaries, chats, or transcripts per day.

Description

Create a BrightData account to start using this MCP server: https://brdta.com/techwithtim_mcp So you've built an AI agent. It can reason, it can plan, but then you ask it to get real time data from the web, and everything breaks. You're hit with rate limits, CAPTCHAs, JavaScript, roadblocks, or worse, the site just outright blocks you, meaning you can't get the data you need. If that sounds familiar, then keep watching in this video. Want to make real money with coding? I share high-signal insights on careers, monetization, and leverage in my free newsletter. Join here and get my guide How to Make Money With Coding instantly: https://techwithtim.net/newsletter 🎞 Video Resources 🎞 BrightData MCP Docs: https://github.com/luminati-io/brightdata-mcp Code In This Video: https://github.com/techwithtim/BrightDataMCPServerAgent UV Video: https://www.youtube.com/watch?v=6pttmsBSi8M&ab_channel=TechWithTim MCP Video: https://www.youtube.com/watch?v=L94WBLL0KjY&ab_channel=TechWithTim Anthropic Console: https://console.anthropic.com/settings/keys ⏳ Timestamps ⏳ 00:00 | Overview 00:19 | Web Scraping MCP Server 00:49 | What are MCPs? 02:07 | MCP Server Demo 07:31 | Setup in Claude Desktop 14:07 | Use in Custom Python AI Agent Hashtags #BrightData #WebScraping #AiAgents

Top Comments (10)

@StarStrider99 2025-05-08

So helpful! I just finished my web scraper agent from this tutorial, and it works perfectly! I'm going to try making a nice frontend for it now.

1
@christophedms613 2025-05-07

Very Good, great explanation, good job Tim hope you'll do the next video

0
@dwarfminer 2025-05-11

Thanks a lot, Tim! this video was incredibly helpful! I followed your video and managed to create my own project using the MCP server setup. However, as it grew, it became a huge monolith 😅. I’ve been trying to split things into separate files to keep it more maintainable, but I’d really appreciate in future videos a follow-up showing how to modularize the project properly. That would be awesome!

0
@watch_what_is_happening 2025-05-09

tim, it is working great. thank you very much

0
@Lokimando 2025-05-09

Great tutorial as always , thanks Tim. Is there any easy way to run the custom agent in a docker container ?

1
@Jocke1234 2025-12-12

You probably always delete your api keys after making the videos, but still i found it funny, that you blurred out the left side of the mcp access details, and left the right side unblurred, which i believe also has the api key as the bearer in the request :D

0
@IvoDachev-k5b 2025-05-08

What is th cost? And how this compare to Crawl4AI ( which is 100% free)?

12
@StuartLong-x7k 2025-09-29

This is super helpful! Is there a specific way to integrate HasData with the MCP server for smoother scraping?

0
@hamburger--fries 2025-06-05

Any site? Doubt it. If you have a html page with no JSON or consistent patterns from page to page how would you scrape it? For example an OCR scan of a book output as html-v1 from the year 1999 how would you scrape it without a highly complex beautifulsoup custom Python that can handle the different patterns. Where by the python script is modified for each different pattern.

2
@aidenkearney1902 2025-05-07

Thank you Tim! What about rotating proxy servers for bots?

0

Unlock the Data Inside
Turn Videos into Knowledge

  • Get FREE 10/day: transcripts, summaries, chats
  • Chat with videos, export text & PDF
  • $1 free API credit for RAG, chatbots & research

Free forever plan • All features unlocked

App screenshot