Repository Index

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

• ai
• ai-agents
• crawler
• data-extraction

TranslatedXiaohongshu notes | comment crawler, Douyin videos | comment crawler, Kuaishou videos | comment crawler, Bilibili videos | comment crawler, Weibo posts | comment crawler, Baidu Tieba posts | Baidu Tieba comment and reply crawler | Zhihu Q&A and articles | comment crawler

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

• ai-tools
• automation
• claude-ai
• claude-skills

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

• python
• news
• crawler
• crawling

A scalable web crawler framework for Java.

• crawler
• java
• scraping
• framework

抓取抖音单条视频完整评论区（一级 + 楼中楼）的 Agent Skill：在已登录的 Chrome 里照抄页面真实请求，签名交给页面 SDK

• agent-skill
• claude-code
• comments-scraper
• douyin

AI 订阅与卡网比价平台，说出需求，让 AI 帮你选。聚合 ChatGPT / Codex、Claude、Gemini 等代充或账号报价，对比价格走势、充值渠道与质保。AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.

• ai
• price-comparison
• self-hosted
• web-scraping