Repository Index

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

• crawler
• crawling
• crawling-python
• playwright

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Scrapy, a fast high-level web crawling & scraping framework for Python.

• python
• scraping
• crawling
• framework

🧡 Everything is RSSible

• rss
• twitter
• weibo
• bilibili

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

• cheerio
• jquery
• htmlparser2
• dom

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

• browser
• rust
• servo
• browser-automation

Crawl a site to generate knowledge files to create your own custom GPT from a URL

• ai

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

A next-generation crawling and spidering framework.

• crawler
• web-spider
• gocrawler
• spider-framework

TranslatedA multi-task real-time/scheduled monitoring and intelligent analysis system for Xianyu (闲鱼), built with Playwright and AI, and equipped with a full-featured back-end management UI. It helps users find the products they like among the vast number of items on Xianyu.

• ai
• playwright
• automation
• gemini

TranslatedDouyin / TikTok platform tool for downloading works and collecting data

• douyin
• python
• tiktok
• api

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

• ai-tools
• automation
• claude-ai
• claude-skills

A little like that j-thing, only in Go.

• selector-strings
• jquery
• html-parsing
• goquery

Pythonic HTML Parsing for Humans™

• html
• scraping
• python
• requests

Download pictures (or videos) along with their captions and other metadata from Instagram.

• instagram
• instagram-client
• instagram-feed
• downloader

Incredibly fast crawler designed for OSINT.

• crawler
• spider
• python
• osint

Browser Automation, End-to-End Testing, and Stealth in one library. Bypass bot-detection/CAPTCHAs with SeleniumBase CDP Mode.

• python
• selenium-python
• e2e-testing
• seleniumbase

TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool

• python
• json
• linux
• macos

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

A standalone version of the readability lib

The RSS feed for websites missing it

• php
• rss-bridge
• unlicense
• atom-feed