Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

Connect APIs, remarkably fast. Free for developers.

• eventsourcing
• event-sourcing
• automation
• workflow

Mautic: Open Source Marketing Automation Software.

• marketing-automation
• mautic
• php
• hacktoberfest

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

• tokenization
• web-security
• compliance
• security

TranslatedOpen-source GrokBot alternative built on DeepSeek Harness. Build your personal bot team: multiple personal agents with persistent memory, identities and permissions, extensible through DSH plugins. MIT.

• dsh-plugin
• agent-memory
• ai-agent
• bot-marketplace

🐘 A PHP client for interacting with Gotenberg.

• gotenberg
• pdf
• word
• excel

TranslatedBullFlow — Open-source enterprise RAG solution: knowledge base → automated marketing content → omnichannel publishing. The entire knowledge base pipeline runs on local open-source models, with no paid models needed; AI Agent creates image-text posts / images / videos, with one-click publishing to 17 platforms including Douyin, Xiaohongshu, WeChat Official Accounts, LinkedIn, and YouTube. Commercial-grade, multi-tenant, billing & admin built in. MIT

• ai-agents
• bge-m3
• content-generation
• elasticsearch

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi

Recieve, send, edit and delete Discord messages in node-red.

Download Medium posts as clean Markdown, preserving structure, images, links, code blocks, and common embeds for plain Markdown or Jekyll workflows.

• ruby
• medium
• medium-articles
• markdown

My algorithmic trading strategies with the Gekko cryptocurrency trading bot.

• gekko-strategies
• trading-strategies
• machine-learning-algorithms
• neural-networks

Confluence Cloud to Markdown crawler with incremental updates and local link rewriting

Text Generation using Markov Chains fed by 4chan APIs

• scrapper
• data-mining
• markov-chain

A hilarious Sinatra-based Markov bot for Slack.

• markov-bot
• sinatra
• heroku
• ruby

A package to searching courses - Super Mario Maker

• json
• mario-maker
• nodejs
• javascript

"Execute Right To Be Forgotten" Marketing Automation Action