Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

🧡 Everything is RSSible

• rss
• twitter
• weibo
• bilibili

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

• cheerio
• jquery
• htmlparser2
• dom

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

Crawl a site to generate knowledge files to create your own custom GPT from a URL

• ai

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

TranslatedDouyin / TikTok platform tool for downloading works and collecting data

• douyin
• python
• tiktok
• api

TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool

• python
• json
• linux
• macos

TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.

• download
• wechat
• wechat-article
• wechat-download

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/

• llm
• proxy

A standalone version of the readability lib

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• crawler
• javascript
• spider
• extract-data

The next web scraper. See through the <html> noise.

Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.

A AI Browser run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.

• anti-bot
• automation
• firefox
• headless-browser

TranslatedAn Agent Skill for scraping the complete comment section of a single Douyin video (top-level comments + nested replies): it copies the page's real requests inside a logged-in Chrome and leaves the signing to the page's SDK

• agent-skill
• claude-code
• comments-scraper
• douyin

TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.

• ai
• price-comparison
• self-hosted
• web-scraping