Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Repository Index
47,133 repositories indexed
1–24 of 70 · Most stars
Scrapy, a fast high-level web crawling & scraping framework for Python.
- python
- scraping
- crawling
- framework
AI Agent Assistant & development framework that integrates lots of IM platforms, LLMs, plugins and AI feature, and can be your openclaw alternative. ✨
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
- browser
- rust
- servo
- browser-automation
The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/products/free-netbox-cloud/
- dcim
- ipam
- netbox
- network-automation
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
- golang
- chrome
- chrome-debugging-protocol
- testing
Incredibly fast crawler designed for OSINT.
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
Connect APIs, remarkably fast. Free for developers.
- eventsourcing
- event-sourcing
- automation
- workflow
Blazing fast deployer and server automation tool
- ruby
- rails
- mina
- deployment
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
SEO Tools for Laravel
- php
- opengraph
- meta-tags
- opengraph-tags
Fast website link checker in Go
An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, Ghostarchive, IPFS, Telegraph, and file systems.
- telegram
- archive
- wayback-machine
- save-the-internet
A framework to build Slack apps using Python
- slack
- slack-api
- bolt
- web-api
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
LibreTime: Radio Broadcast & Automation Platform
- radio
- broadcasting
- airtime
- liquidsoap
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
A Discord API wrapper library made in C