Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Repository Index
142,273 visible repositories in the index
1–24 of 66 · Most stars
🧡 Everything is RSSible
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Crawl a site to generate knowledge files to create your own custom GPT from a URL
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.
- download
- wechat-article
- wechat-download
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
Get the main content of any page as Markdown.
Turn any webpage into structured data using LLMs
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Official Bright Data CLI - scrape, search, and extract structured web data directly from your terminal.
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
Datascraper that is currently adult focused (OnlyFans, etc)- Updated regularly
- onlyfans
- scraper
- archiver
- fansly
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
- automation
- self-hosted
- job-search
- jobhunt
Stay on top of trending topics on social media and the web with AI
🔥 今日热榜 API,一个聚合热门数据的 API 接口,支持 RSS 模式 及 Vercel 部署 | 前端页面:https://github.com/imsyy/DailyHot
Extract any website’s design system into tokens in seconds: logo, colors, typography, borders & more. One command.
Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
TrendPublish: 全自动 AI 内容生成与发布系统 | 微信公众号自动化 | 多源数据抓取 (Twitter/X、网站) | DeepseekAI、千问、讯飞模型 | 智能内容分析排序 | 定时发布 | 多模板支持 | Node.js | TypeScript | AI 技术趋势跟踪工具
- ai
- weixin