Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

Self-Hosting Guide. Learn all about locally hosting (on premises & private web servers) and managing software applications by yourself or your organization. Including Cloud, LLMs, WireGuard, Automation, Home Assistant, and Networking.

• self-hosted
• selfhosted
• home-assistant
• wireguard

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• crawler
• javascript
• spider
• extract-data

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine

• meta-search-engine
• rust
• search-engine
• fast

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

• java
• elasticsearch
• full-text-search
• lucene

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

• agpl
• ai-agents
• anti-bot
• crawling

DocSearch - Scraper

• algolia
• docsearch
• documentation
• scraper

TranslatedBullFlow — Open-source enterprise RAG solution: knowledge base → automated marketing content → omnichannel publishing. The entire knowledge base pipeline runs on local open-source models, with no paid models needed; AI Agent creates image-text posts / images / videos, with one-click publishing to 17 platforms including Douyin, Xiaohongshu, WeChat Official Accounts, LinkedIn, and YouTube. Commercial-grade, multi-tenant, billing & admin built in. MIT

• ai-agents
• bge-m3
• content-generation
• elasticsearch

TranslatedMaiBot Baidu search plugin, supporting search commands and LLM Tool, integrated with the official Baidu Qianfan API.

An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.

• elixir
• phoenix
• otp
• react

Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.

• ai
• claude
• content-extraction
• go

Browser extension to simulate browsing behaviour in search engines.

• automation
• audit
• browser-extension
• firefox

Search npm packages by web scraping and without indexing all npm packages

A package to searching courses - Super Mario Maker

• json
• mario-maker
• nodejs
• javascript