Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
85,578 visible repositories in the index
1–24 of 110 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
🧡 Everything is RSSible
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Crawl a site to generate knowledge files to create your own custom GPT from a URL
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
- douyin
- python
- tiktok
- api
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.
- download
- wechat-article
- wechat-download
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
A standalone version of the readability lib
Get the main content of any page as Markdown.
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
- python
- python-script
- image-download
- google-images
This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles and it does not require an API key nor a headless browser, like other selenium based solutions do!
TranslatedProvince, city, district/county and township level (three-level or four-level) city data, with pinyin annotations, coordinates, and administrative region boundary ranges; latest collection on April 3, 2026; provides csv format files, supports online conversion into multi-level linked js code and general json format, provides software to convert into shp, geojson, sql and import into databases; includes js collection source code that runs in the browser, combining administrative division data from the Ministry of Civil Affairs of the People's Republic of China, China National Geographical Names Information Library, the Bureau of Statistics, Amap (Gaode Map), and Tencent Maps
Turn any webpage into structured data using LLMs
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Official Bright Data CLI - scrape, search, and extract structured web data directly from your terminal.
💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js
- offline-web-pages
- downloader
- scraper
- assets
The next web scraper. See through the <html> noise.
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
📜 Extract meaningful content from the chaos of a web page