1.firecrawl / firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
+10,252 stars
from 187,424 · score 9.24
Trending repositories
Ranked by Attention Score: stars gained over the last 30 complete days
43 repositories by Attention Score · Oct 4, 2026 – Nov 2, 2026 (UTC)
2.D4Vinci / Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
+6,471 stars
from 85,273 · score 8.78
3.unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
+2,180 stars
from 84,542 · score 7.69
4.ScrapeGraphAI / Scrapegraph-ai
Python scraper based on AI
+969 stars
from 31,453 · score 6.88
5.Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
+669 stars
from 20,399 · score 6.51
6.scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
+654 stars
from 64,508 · score 6.48
7.getmaxun / maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
+520 stars
from 17,609 · score 6.26
8.dengmeiluan / ticket-monitoring
✈️ 自托管的多渠道机票价格监控：航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
+417 stars
from 1 · score 6.04
9.apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
+363 stars
from 25,948 · score 5.90
10.projectdiscovery / katana
A next-generation crawling and spidering framework.
+197 stars
from 17,587 · score 5.29
11.NaiboWang / EasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
+193 stars
from 44,566 · score 5.27
12.DedSecInside / TorBot
Dark Web OSINT Tool
+183 stars
from 4,975 · score 5.21
13.adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
+151 stars
from 6,899 · score 5.02
14.iawia002 / lux
👾 Fast and simple video download library and CLI tool written in Go
+113 stars
from 31,705 · score 4.74
15.s0md3v / Photon
Incredibly fast crawler designed for OSINT.
+91 stars
from 13,226 · score 4.52
16.jhao104 / proxy_pool
Python ProxyPool for web spider
+62 stars
from 23,716 · score 4.14
17.BlessedRebuS / Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
+55 stars
from 770 · score 4.03
18.orangecoding / fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
+52 stars
from 1,555 · score 3.97
19.gocolly / colly
Elegant Scraper and Crawler Framework for Golang
+50 stars
from 25,521 · score 3.93
20.guyueyingmu / avbook
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
+47 stars
from 10,052 · score 3.87
21.injetlee / Python
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
+46 stars
from 10,843 · score 3.85
22.shengqiangzhang / examples-of-web-crawlers
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
+29 stars
from 14,677 · score 3.40
23.crawlab-team / crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
+21 stars
from 12,266 · score 3.09
24.codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
+20 stars
from 15,162 · score 3.04
25.ssssssss-team / spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
+19 stars
from 11,345 · score 3.00
26.hardkoded / puppeteer-sharp
Headless Chrome .NET API
+12 stars
from 3,920 · score 2.56
27.elixir-crawly / crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
+10 stars
from 1,117 · score 2.40
28.apache / stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
+7 stars
from 1,004 · score 2.08
29.bda-research / node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
+6 stars
from 6,790 · score 1.95
30.JayBizzle / Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
+6 stars
from 2,402 · score 1.95
31.codelibs / fess
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
+5 stars
from 1,137 · score 1.79
32.binux / pyspiderArchived
A Powerful Spider(Web Crawler) System in Python.
+4 stars
from 16,746 · score 1.61
33.code4craft / webmagic
A scalable web crawler framework for Java.
+4 stars
from 11,666 · score 1.61
34.fredwu / crawler
A high performance web crawler / scraper in Elixir.
+4 stars
from 957 · score 1.61
35.0xInfection / XSRFProbe
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
+3 stars
from 1,305 · score 1.39
36.Nekmo / dirhunt
Find web directories without bruteforce
+2 stars
from 2,006 · score 1.10
37.howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
+2 stars
from 1,737 · score 1.10
38.dotnetcore / DotnetSpider
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
+1 stars
from 4,140 · score 0.69
39.leonardocardoso / SwiftLinkPreviewArchived
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
+1 stars
from 1,385 · score 0.69
40.felipecsl / wombat
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
+1 stars
from 1,359 · score 0.69
41.s0rg / crawley
The unix-way web crawler
+1 stars
from 339 · score 0.69
42.karust / gogetcrawl
Extract web archive data using Wayback Machine and Common Crawl
+1 stars
from 185 · score 0.69
43.duyet / pricetrack
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
+1 stars
from 139 · score 0.69