Repository Index

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

A little like that j-thing, only in Go.

• selector-strings
• jquery
• html-parsing
• goquery

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

• crawler
• dotnetcore
• cross-platform
• csharp

An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, Ghostarchive, IPFS, Telegraph, and file systems.

• telegram
• archive
• wayback-machine
• save-the-internet

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• ruby
• scraper
• crawler
• dsl

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

Extract structured data from web sites. Web sites scraping.

• golang
• golang-library
• extract-data
• chrome-fetcher

A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library

• golang
• unmarshall
• goquery
• selectors

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

• go
• golang
• goquery
• colly

api for liquipedia.net

• dota2
• liquipedia
• python3
• counter-strike

Download Medium posts as clean Markdown, preserving structure, images, links, code blocks, and common embeds for plain Markdown or Jekyll workflows.

• ruby
• medium
• medium-articles
• markdown