Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

TranslatedA one-click tool for the daily tasks of Arknights, supporting all clients. (Arknights assistant: one click to finish all your daily tasks!)

• arknights
• maa
• computer-vision

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

TranslatedProduction-grade platform for building agentic IM bots - Agent, knowledge base orchestration, plugin system / Bots for Discord / Slack / LINE / Telegram / WeChat(企业微信 (Qiye Weixin, "Enterprise WeChat"), 企微智能机器人 (Qiwei Zhineng Jiqiren, "Enterprise WeChat smart bot"), 公众号 (Gongzhonghao, "official account")) / 飞书 (Feishu, "flying book") / 钉钉 (Dingding, "nail nail") / QQ / Matrix e.g. Integrated with ChatGPT(GPT), DeepSeek, Dify, n8n, Langflow, Coze, Claude, Gemini, GLM, Ollama, SiliconFlow, Kimi, openclaw / hermes agent, deerflow

• qq
• openai
• agent
• llm

Bridge local AI coding agents (Claude Code, Cursor, Gemini CLI, Codex) to messaging platforms (Feishu/Lark, DingTalk, Slack, Telegram, Discord, LINE, WeChat Work). Chat with your AI dev assistant from anywhere — no public IP required for most platforms.

Connect APIs, remarkably fast. Free for developers.

• eventsourcing
• event-sourcing
• automation
• workflow

🤖 Multi-platform IM AI Agent for Telegram, WhatsApp, Lark, and WeChat. Connects ChatGPT / Claude / Kimi / DeepSeek / Ollama / Pi for auto-replies, community analysis, contact management, and inactive-friend detection.

• wechat
• wechatbot
• chatgpt
• openai

Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.

• perimeterx
• nopecha
• ai-agents
• anti-captcha

Mautic: Open Source Marketing Automation Software.

• marketing-automation
• mautic
• php
• hacktoberfest

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

Mechanize is a ruby library that makes automated web interaction easy.

• scraping
• web
• ruby

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

• tokenization
• web-security
• compliance
• security

:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine

• meta-search-engine
• rust
• search-engine
• fast

TranslatedOpen-source GrokBot alternative built on DeepSeek Harness. Build your personal bot team: multiple personal agents with persistent memory, identities and permissions, extensible through DSH plugins. MIT.

• dsh-plugin
• agent-memory
• ai-agent
• bot-marketplace

Feishu(飞书)/Lark Open API Go SDK, Support ALL Open API and Event Callback.

• lark
• feishu
• sdk
• go

🐘 A PHP client for interacting with Gotenberg.

• gotenberg
• pdf
• word
• excel

TranslatedBullFlow — Open-source enterprise RAG solution: knowledge base → automated marketing content → omnichannel publishing. The entire knowledge base pipeline runs on local open-source models, with no paid models needed; AI Agent creates image-text posts / images / videos, with one-click publishing to 17 platforms including Douyin, Xiaohongshu, WeChat Official Accounts, LinkedIn, and YouTube. Commercial-grade, multi-tenant, billing & admin built in. MIT

• ai-agents
• bge-m3
• content-generation
• elasticsearch

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi

Visual Integration Designer for Apache Camel

• camel
• camel-k
• apache-camel
• integration