Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

🧡 Everything is RSSible

• rss
• twitter
• weibo
• bilibili

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

• cheerio
• jquery
• htmlparser2
• dom

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

Crawl a site to generate knowledge files to create your own custom GPT from a URL

• ai

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.

• download
• wechat
• wechat-article
• wechat-download

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/

• llm
• proxy

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• crawler
• javascript
• spider
• extract-data

Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.

Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...

The unofficial HLTV Node.js API

• hltv
• parser
• scraper

TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.

• ai
• price-comparison
• self-hosted
• web-scraping

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi

Practical pinned gist which show your latest medium status 📌

• awesome-pinned-gists
• medium-stats
• github-gists
• github-action