Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

🧡 Everything is RSSible

• rss
• twitter
• weibo
• bilibili

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

• browser
• rust
• servo
• browser-automation

Crawl a site to generate knowledge files to create your own custom GPT from a URL

• ai

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

TranslatedDouyin / TikTok platform tool for downloading works and collecting data

• douyin
• python
• tiktok
• api

TranslatedA multi-task real-time/scheduled monitoring and intelligent analysis system for Xianyu (闲鱼), built with Playwright and AI, and equipped with a full-featured back-end management UI. It helps users find the products they like among the vast number of items on Xianyu.

• ai
• playwright
• automation
• gemini

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

• ai-tools
• automation
• claude-ai
• claude-skills

A little like that j-thing, only in Go.

• selector-strings
• jquery
• html-parsing
• goquery

Pythonic HTML Parsing for Humans™

• html
• scraping
• python
• requests

Download pictures (or videos) along with their captions and other metadata from Instagram.

• instagram
• instagram-client
• instagram-feed
• downloader

Browser automation and testing without limits. SeleniumBase's CDP Mode bypasses bot-detection with ease. Check out the examples!

• python
• selenium-python
• e2e-testing
• seleniumbase

TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool

• python
• json
• linux
• macos

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/

• llm
• proxy

A standalone version of the readability lib

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

• jsoup
• html
• java
• dom

The RSS feed for websites missing it

• php
• rss-bridge
• unlicense
• atom-feed

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

• python
• machine-learning
• natural-language-processing
• web-mining

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web