Sign in

Repository Index

56,139 visible repositories in the index

Tags: Automation, bots, and scraping
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

25–40 of 40 · Most stars

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

  • crawling
  • csharp
  • elixir
  • ffi

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

  • go
  • golang
  • goquery
  • colly

Seamlessly fetch paginated data from any source. Simple and high performance API scraping included!

  • api-scraper
  • indexer
  • pagination
  • paginator