Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
Repository Index
47,133 repositories indexed
1–21 of 21 · Most stars
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Sim is the collaborative workspace to build, deploy, and monitor AI agents and workflows. Used by 100,000+ builders.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
TranslatedProduction-grade platform for building agentic IM bots - Agent, knowledge base orchestration, plugin system / Bots for Discord / Slack / LINE / Telegram / WeChat(企业微信 (Qiye Weixin, "Enterprise WeChat"), 企微智能机器人 (Qiwei Zhineng Jiqiren, "Enterprise WeChat smart bot"), 公众号 (Gongzhonghao, "official account")) / 飞书 (Feishu, "flying book") / 钉钉 (Dingding, "nail nail") / QQ / Matrix e.g. Integrated with ChatGPT(GPT), DeepSeek, Dify, n8n, Langflow, Coze, Claude, Gemini, GLM, Ollama, SiliconFlow, Kimi, openclaw / hermes agent, deerflow
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
Fully functional Twitter clone built in flutter framework using Firebase realtime database and storage
- twitter-clone
- flutter-apps
- social
File Attachment toolkit for Ruby applications
- file-upload
- ruby
- rack
- storage
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
TranslatedBullFlow — Open-source enterprise RAG solution: knowledge base → automated marketing content → omnichannel publishing. The entire knowledge base pipeline runs on local open-source models, with no paid models needed; AI Agent creates image-text posts / images / videos, with one-click publishing to 17 platforms including Douyin, Xiaohongshu, WeChat Official Accounts, LinkedIn, and YouTube. Commercial-grade, multi-tenant, billing & admin built in. MIT
- ai-agents
- bge-m3
- content-generation
- elasticsearch
Selfhosted. URL to PNG utility featuring parallel rendering using Playwright for screenshots and with storage caching via Local, S3, or CouchDB
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Ragpicker is a Plugin based malware crawler with pre-analysis and reporting functionalities. Use this tool if you are testing antivirus products, collecting malware for another analyzer/zoo.
POMWright is a complementary test framework for Playwright written in TypeScript.
- playwright
- playwright-test
- automation
- fixture
Lightweight, self hosted platform for building webhooks, scheduled tasks and pages. Implement your logic with Bash, Lua or mRuby
- automation
- bash
- devops
- golang
The Raspberry Pi Telegram Bot is a lightweight, customizable bot built for Raspberry Pi users. It leverages Python and the Telegram Bot API to enable users to execute commands, receive updates, and automate processes from their Telegram account. Ideal for DIY projects, home automation, and remote control applications, this bot is easy to set up .
- bot
- home-automation
- iot
- notifications
Docker-aware backup agent using labels to automate backups for PostgreSQL, MySQL, MongoDB, and Redis to local or S3 compatible storage.
Local-first automation Agent & deterministic workflow engine — the secure control layer between AI and your data. Data stays local. Connect your own LLM / files / notes / databases via Connector API. ReAct reasoning · DAG/WAL/Saga/idempotent execution · MCP · offline-capable.
Crypto trading arbitrage bot
Super simple interface to send discord messages through webhooks in vlang
- discord
- discord-webhook
- discord-webhooks
- notifications