1firecrawl / firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
+2,346 stars
from 195,330 · score 7.76
Trending repositories
Ranked by Attention Score: stars gained over the last 7 complete days
28 repositories by Attention Score · Oct 27, 2026 – Nov 2, 2026 (UTC)
2D4Vinci / Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
+2,042 stars
from 89,702 · score 7.62
3unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
+461 stars
from 86,261 · score 6.14
4dengmeiluan / ticket-monitoring
✈️ 自托管的多渠道机票价格监控：航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
+247 stars
from 143 · score 5.51
5ScrapeGraphAI / Scrapegraph-ai
Python scraper based on AI
+202 stars
from 32,220 · score 5.31
6Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
+173 stars
from 20,895 · score 5.16
7scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
+138 stars
from 65,024 · score 4.93
8getmaxun / maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
+89 stars
from 18,040 · score 4.50
9projectdiscovery / katana
A next-generation crawling and spidering framework.
+54 stars
from 17,730 · score 4.01
10apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
+47 stars
from 26,264 · score 3.87
11DedSecInside / TorBot
Dark Web OSINT Tool
+43 stars
from 5,115 · score 3.78
12adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
+28 stars
from 7,022 · score 3.37
13BlessedRebuS / Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
+23 stars
from 802 · score 3.18
14s0md3v / Photon
Incredibly fast crawler designed for OSINT.
+22 stars
from 13,295 · score 3.14
15NaiboWang / EasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
+20 stars
from 44,739 · score 3.04
16iawia002 / lux
👾 Fast and simple video download library and CLI tool written in Go
+14 stars
from 31,804 · score 2.71
17jhao104 / proxy_pool
Python ProxyPool for web spider
+12 stars
from 23,766 · score 2.56
18injetlee / Python
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
+11 stars
from 10,878 · score 2.48
19guyueyingmu / avbook
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
+11 stars
from 10,088 · score 2.48
20orangecoding / fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
+11 stars
from 1,596 · score 2.48
21ssssssss-team / spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
+3 stars
from 11,361 · score 1.39
22codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
+2 stars
from 15,180 · score 1.10
23crawlab-team / crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
+2 stars
from 12,285 · score 1.10
24gocolly / colly
Elegant Scraper and Crawler Framework for Golang
+1 stars
from 25,570 · score 0.69
25leonardocardoso / SwiftLinkPreviewArchived
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
+1 stars
from 1,385 · score 0.69
260xInfection / XSRFProbe
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
+1 stars
from 1,307 · score 0.69
27elixir-crawly / crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
+1 stars
from 1,126 · score 0.69
28apache / stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
+1 stars
from 1,010 · score 0.69