Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 46 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
Self-Hosting Guide. Learn all about locally hosting (on premises & private web servers) and managing software applications by yourself or your organization. Including Cloud, LLMs, WireGuard, Automation, Home Assistant, and Networking.
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
C# application with primary purpose of farming Steam cards from multiple accounts simultaneously.
- steam
- steam-api
- steam-account
- steambot
Saves Discord chat logs to a file
TranslatedLeave the dailies to March 7th, and keep the Trailblaze for yourself. An open-source automation assistant for "Honkai: Star Rail".
- honkai-star-rail
- star-rail
- march7thassistant
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, Ghostarchive, IPFS, Telegraph, and file systems.
- telegram
- archive
- wayback-machine
- save-the-internet
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
- fredy
- immowelt
- appartment
- appartments
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
Powerful and flexible library for logging on files, console, memory, email, rest, eventlog, syslog, slack, telegram, redis, logstash, elasticsearch, influxdb, graylog, Sentry, Twilio, ide debug messages and throw events for Delphi/Firemonkey/freepascal/.NET (Windows/Linux/OSX/IOS/Android).
The Sofie TV Studio Automation System, used in live TV news production by the Norwegian public service broadcaster NRK since September 2018
- sofie
- nrk
- sofie-server
- mos-protocol
WarcDB: Web crawl data as SQLite databases.
LittleHorse Server: Business-as-Code Platform for Microservices, Workflows, and Agents
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
TranslatedBullFlow — Open-source enterprise RAG solution: knowledge base → automated marketing content → omnichannel publishing. The entire knowledge base pipeline runs on local open-source models, with no paid models needed; AI Agent creates image-text posts / images / videos, with one-click publishing to 17 platforms including Douyin, Xiaohongshu, WeChat Official Accounts, LinkedIn, and YouTube. Commercial-grade, multi-tenant, billing & admin built in. MIT
- ai-agents
- bge-m3
- content-generation
- elasticsearch
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.
The type-safe, event-driven workflow orchestration library that scales with your business. Build robust, distributed workflows in Go with compile-time safety, automatic retries, and horizontal scaling out of the box. Integrate with your existing tech stack using adapters 🚀
- eventdrivenarchitecture
- golang
- kafka
- reflex