Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
Repository Index
47,133 repositories indexed
25–41 of 41 · Most stars
A high performance web crawler / scraper in Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox
:scissors: High performance, multi-threaded image scraper
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
The unofficial HLTV Node.js API
TranslatedAn Agent Skill for scraping the complete comment section of a single Douyin video (top-level comments + nested replies): it copies the page's real requests inside a logged-in Chrome and leaves the signing to the page's SDK
- agent-skill
- claude-code
- comments-scraper
- douyin
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Webpage summary extractor using Facebook Open Graph and arc90's readability
- readability
- scraper
- opengraph
Browser extension to simulate browsing behaviour in search engines.
- automation
- audit
- browser-extension
- firefox
Go package for scraping website recipes
- go
- golang
- recipe
- recipes
Seamlessly fetch paginated data from any source. Simple and high performance API scraping included!
- api-scraper
- indexer
- pagination
- paginator
pocketbase LLM docs scraper