1.firecrawl / firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
+38,224 stars
from 159,452 · score 10.55
Trending repositories
Ranked by Attention Score: stars gained over the last 90 complete days
46 repositories by Attention Score · Aug 5, 2026 – Nov 2, 2026 (UTC)
2.D4Vinci / Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
+19,806 stars
from 71,938 · score 9.89
3.unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
+11,111 stars
from 75,611 · score 9.32
4.ScrapeGraphAI / Scrapegraph-ai
Python scraper based on AI
+3,556 stars
from 28,866 · score 8.18
5.Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
+2,072 stars
from 18,996 · score 7.64
6.scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
+1,744 stars
from 63,418 · score 7.46
7.apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
+1,228 stars
from 25,083 · score 7.11
8.getmaxun / maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
+1,168 stars
from 16,961 · score 7.06
9.DedSecInside / TorBot
Dark Web OSINT Tool
+688 stars
from 4,470 · score 6.54
10.adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
+665 stars
from 6,385 · score 6.50
11.NaiboWang / EasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
+631 stars
from 44,128 · score 6.45
12.projectdiscovery / katana
A next-generation crawling and spidering framework.
+606 stars
from 17,178 · score 6.41
13.iawia002 / lux
👾 Fast and simple video download library and CLI tool written in Go
+366 stars
from 31,452 · score 5.91
14.s0md3v / Photon
Incredibly fast crawler designed for OSINT.
+290 stars
from 13,027 · score 5.67
15.jhao104 / proxy_pool
Python ProxyPool for web spider
+279 stars
from 23,499 · score 5.63
16.BlessedRebuS / Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
+273 stars
from 552 · score 5.61
17.gocolly / colly
Elegant Scraper and Crawler Framework for Golang
+225 stars
from 25,346 · score 5.42
18.orangecoding / fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
+223 stars
from 1,384 · score 5.41
19.injetlee / Python
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
+165 stars
from 10,724 · score 5.11
20.guyueyingmu / avbook
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
+128 stars
from 9,971 · score 4.86
21.shengqiangzhang / examples-of-web-crawlers
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
+89 stars
from 14,617 · score 4.50
22.codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
+76 stars
from 15,106 · score 4.34
23.ssssssss-team / spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
+60 stars
from 11,304 · score 4.11
24.crawlab-team / crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
+59 stars
from 12,228 · score 4.09
25.JayBizzle / Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
+41 stars
from 2,367 · score 3.74
26.hardkoded / puppeteer-sharp
Headless Chrome .NET API
+30 stars
from 3,902 · score 3.43
27.apache / stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
+22 stars
from 989 · score 3.14
28.codelibs / fess
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
+21 stars
from 1,121 · score 3.09
29.code4craft / webmagic
A scalable web crawler framework for Java.
+20 stars
from 11,650 · score 3.04
30.elixir-crawly / crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
+15 stars
from 1,112 · score 2.77
31.0xInfection / XSRFProbe
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
+14 stars
from 1,294 · score 2.71
32.Nekmo / dirhunt
Find web directories without bruteforce
+12 stars
from 1,996 · score 2.56
33.bda-research / node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
+10 stars
from 6,786 · score 2.40
34.binux / pyspiderArchived
A Powerful Spider(Web Crawler) System in Python.
+9 stars
from 16,741 · score 2.30
35.dotnetcore / DotnetSpider
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
+7 stars
from 4,134 · score 2.08
36.fredwu / crawler
A high performance web crawler / scraper in Elixir.
+5 stars
from 956 · score 1.79
37.howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
+4 stars
from 1,735 · score 1.61
38.s0rg / crawley
The unix-way web crawler
+4 stars
from 336 · score 1.61
39.karust / gogetcrawl
Extract web archive data using Wayback Machine and Common Crawl
+4 stars
from 182 · score 1.61
40.myreader-io / myGPTReaderArchived
A community-driven way to read and chat with AI bots - powered by chatGPT.
+2 stars
from 4,397 · score 1.10
41.fanyong920 / jvppeteer
Java API For Chrome and Firefox
+2 stars
from 802 · score 1.10
42.leonardocardoso / SwiftLinkPreviewArchived
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
+1 stars
from 1,385 · score 0.69
43.felipecsl / wombat
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
+1 stars
from 1,359 · score 0.69
44.postmodern / spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
+1 stars
from 835 · score 0.69
45.duyet / pricetrack
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
+1 stars
from 139 · score 0.69
46.veliovgroup / spiderable-middleware
Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ideal for PWAs, SPAs, and modern JS-driven apps, websites, and webpages
+1 stars
from 42 · score 0.69