Tags / Automation, bots, and scraping /
Web scraper
A tool, framework, or script that crawls websites or extracts data from web pages.
Repositories
41
Net stars gained, last 7 days
+18,109
Top-level Tag
Net stars gained each day, last 90 days
Peak days
- 1
Oct 2, 2026 +3,855 net stars (median day +1,752)
- Panniantong/Agent-Reach+1,460 (38%)
- lexmount/moli+1,412 (37%)
- firecrawl/firecrawl+420 (11%)
- 2
Oct 4, 2026 +3,531 net stars (median day +1,752)
- lexmount/moli+1,497 (42%)
Published v1.1.13 on Oct 4, 2026
- Panniantong/Agent-Reach+1,146 (32%)
- firecrawl/firecrawl+359 (10%)
- 3
Oct 5, 2026 +3,546 net stars (median day +1,752)
- lexmount/moli+1,802 (51%)
Published v1.1.13 on Oct 4, 2026 · Published v1.1.14 on Oct 5, 2026
- Panniantong/Agent-Reach+966 (27%)
- firecrawl/firecrawl+333 (9%)
- 4
Nov 3, 2026 +4,026 net stars (median day +1,752)
- D4Vinci/Scrapling+1,989 (49%)
- Usagi-org/ai-goofish-monitor+423 (11%)
- Panniantong/Agent-Reach+358 (9%)
A peak day has at least 2 times the net stars of the median day of this chart, and at least 100 more. For each peak day, the list shows the repositories that gained the most net stars that day. It also shows their GitHub releases of that day or the day before (UTC), and their creation date when it was 14 days or less before. These events occur near the same day. They do not show why the stars came.
Largest
Most stars in total now.
readme:
Scrape all URLs of a website with a single request
description:
Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
readme:
Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.
description:
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown.
readme:
一个功能强大的**多平台自媒体数据采集工具**，支持小红书、抖音、快手、B站、微博、贴吧、知乎等主流平台的公开信息抓取。
description:
Scrapy, a fast high-level web crawling & scraping framework for Python.
readme:
RSSHub delivers millions of contents aggregated from all kinds of sources
readme:
The most common one is the `SmartScraperGraph`, which extracts information from a single page given a user prompt and a source URL.
readme:
Once you've loaded the HTML, you can use jQuery-style selectors to find elements within the document.
readme:
It also detects any content featured *inside* pages & extracts it out into a folder:
Newest Admissions
Entered the index by the rising rule, newest first.
readme:
聚合卡网、发卡站的 ChatGPT / Codex、Claude / Claude Code、Gemini、Grok、Cursor 等报价
readme:
抓取抖音单条视频完整评论区的 Agent Skill。
readme:
made for web scraping, JS reverse engineering, data collection, and getting past bot checks.
readme:
5 家浏览器渠道（去哪儿 / 飞猪 / 携程 / 同程 / 途牛）+ 5 家 App 渠道
description:
Rust web scraper & crawler for AI agents: URLs, PDFs and Office files → clean Markdown/JSON.
readme:
Coleta a cada 3 horas
readme:
DonSeTch gives any AI agent full web research from a single local process: fetch, search, crawl, screenshot.
description:
Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.
- sqzw-x/amane1.2K
readme:
内置 25 个影片站点, 按内容类型配置站点顺序, 每个字段可单独调整优先级或排除站点.
readme:
Moli helps your AI agent fetch and extract web pages, search the web, and automate browser tasks.
How a repository gets this Tag
An agent reads the description, the GitHub topics, and the start of the README of each repository. It chooses Tags only from the Tag list in the AwesomeRepos source code. For each Tag, it quotes the text. A code check keeps the Tag only when the quote is in that text. The quote under each repository is that evidence. A Tag does not rank repositories. It does not say that one repository can replace another. To report a wrong Tag, use "Report a problem with this page" on the page of the repository. All numbers on this page count only visible repositories.