Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
60,121 visible repositories in the index
1–19 of 19 · Most stars
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
TranslatedAn Android APP with custom screen tapping based on Accessibility, Advanced Selectors, and Subscription Rules
- android
- kotlin
- kotlin-multiplatform
- compose
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
- douyin
- python
- tiktok
- api
Get the main content of any page as Markdown.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
TranslatedSync articles to multiple content platforms with one click. Supports major platforms including Toutiao (今日头条), WordPress, Zhihu (知乎), Jianshu (简书), Juejin (掘金), CSDN, and typecho. Publish once, sync to multiple platforms. Free up personal productivity.
- chrome-extension
- chrome
- markdown
- multiplatform
Translated✨WeChat Official Account assistant that fits reading habits, WeChat Official Account to MarkDown, WeChat Official Account to PDF, scheduled updates of subscribed Official Account articles, generate RSS feeds for WeChat Official Accounts, export WeChat Official Account subscription feeds, supports Webhook/API/AI Agent integration for WeChat Official Accounts, subscribe to WeChat Official Accounts, WeChat Official Account assistant, WeChat Official Account reading, WeChat Official Account interface, WeChat Official Account crawler, WeChat Official Account monitoring, tag-based subscription to WeChat Official Accounts, WeChat Official Account source, WeChat Official Account book reading, WeChat Official Account articles, WeChat Official Account framework, WeChat Official Account management, WeChat Official Account source, WeChat Official Account platform, WeChat Official Account code, WeChat Official Account system, WeChat Official Account source code
- rss-feed
- rss-hub
- weichat
- weixin
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
面向 AI Agent 的微信公众号创作与发布 CLI：Markdown 排版、AI 配图、预览与草稿创建；支持由浏览器 Agent 保存知乎、CSDN、头条未发布草稿。
- claude-skills
- markdown-converter
- markdown-to-wechat
A personal knowledge base that builds and maintains itself. Drop in sources — Claude (or Codex/Gemini) reads them, extracts knowledge, and maintains a persistent interlinked wiki. Works with Claude Code, Codex, OpenCode, Gemini CLI. No API key needed.
Translated📚 WeChat Reading (微信读书) lossless full-book export tool — one-click export of dual-font, finely typeset PDF / EPUB / plain Markdown notes (compatible with Obsidian & Notion), solving the problem of missing characters when turning pages on the web version.
🐘 A PHP client for interacting with Gotenberg.
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Screenshot automation framework. Define once, regenerate forever. Visual picker, multi-viewport, light/dark mode, CI-ready.
Download Medium posts as clean Markdown, preserving structure, images, links, code blocks, and common embeds for plain Markdown or Jekyll workflows.
- ruby
- medium
- medium-articles
- markdown
Confluence Cloud to Markdown crawler with incremental updates and local link rewriting