Ranked by Attention Score: stars gained over the last 180 complete days
40 repositories by Attention Score · May 8, 2026 – Nov 3, 2026 (UTC)
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
TranslatedXiaohongshu notes | comment crawler, Douyin videos | comment crawler, Kuaishou videos | comment crawler, Bilibili videos | comment crawler, Weibo posts | comment crawler, Baidu Tieba posts | Baidu Tieba comment and reply crawler | Zhihu Q&A and articles | comment crawler
Python scraper based on AI
TranslatedA multi-task real-time/scheduled monitoring and intelligent analysis system for Xianyu (闲鱼), built with Playwright and AI, and equipped with a full-featured back-end management UI. It helps users find the products they like among the vast number of items on Xianyu.
一款在线的 微信公众号文章批量下载 工具，支持导出阅读量与评论数据，无需搭建任何环境，可通过 在线网站 使用，支持 docker 私有化部署和 Cloudflare 部署。 支持下载各种文件格式，其中 HTML 格式可100%还原文章排版与样式。
Scrapy, a fast high-level web crawling & scraping framework for Python.
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
🧡 Everything is RSSible
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
Turn any website into a structured API. Extract, automate, search and monitor the web.
小红书（XiaoHongShu、RedNote）链接提取/作品采集工具
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
A cli tool to browse and play anime
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Download pictures (or videos) along with their captions and other metadata from Instagram.
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
A next-generation crawling and spidering framework.
📊 APIs for Web Scraping, Browser Automation, and E2E Testing. CDP Mode & Stealthy Playwright Mode bypass bot-detection/CAPTCHAs.
Python ProxyPool for web spider
Incredibly fast crawler designed for OSINT.
The RSS feed for websites missing it
Elegant Scraper and Crawler Framework for Golang
A standalone version of the readability lib
Crawl a site to generate knowledge files to create your own custom GPT from a URL
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
A little like that j-thing, only in Go.
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
A scalable web crawler framework for Java.
Pythonic HTML Parsing for Humans™
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.