Repository Index

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Python ProxyPool for web spider

• crawler
• proxy
• spider
• redis

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

Incredibly fast crawler designed for OSINT.

• crawler
• spider
• python
• osint

TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool

• python
• json
• linux
• macos

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• ruby
• scraper
• crawler
• dsl

A high performance web crawler / scraper in Elixir.

• elixir
• crawler
• spider
• scraper

Extract structured data from web sites. Web sites scraping.

• golang
• golang-library
• extract-data
• chrome-fetcher

TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.

• ai
• price-comparison
• self-hosted
• web-scraping

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi

Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app

• api
• redash
• tracking
• crawler

An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.

• elixir
• phoenix
• otp
• react

Credential Phish Analysis and Automation