Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping
Repository Index
47,133 repositories indexed
49–65 of 65 · Most stars
Mac app for website auditing and crawling
- seo
- website-auditing
- swift
- mac
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules
- cloudflare
- firewall
- user-agent
- firewall-rules
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Extract web archive data using Wayback Machine and Common Crawl
- golang
- crawler
- commoncrawl
- wayback-machine
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
- go
- golang
- goquery
- colly
Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ideal for PWAs, SPAs, and modern JS-driven apps, websites, and webpages
- meteor
- middleware
- crawler
- seo
Take HTML Snapshots for your Angular, Ember, Durandal or any JavaScript applications
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action
Confluence Cloud to Markdown crawler with incremental updates and local link rewriting
SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.
:arrow_forward: A .NET core library to handle the login to Backpack.tf. Backpack.tf is a trading site for Team Fortress 2, Counter-Strike: Global Offensive, and Dota 2. Community item pricing, item trading and stats, and much more.
- teamfortress2
- steam
- steambot
- steamweb
A package to searching courses - Super Mario Maker
- json
- mario-maker
- nodejs
- javascript
:repeat: An API wrapper for the TF2 Outpost platform. A platform to find great deals for your Team Fortress 2, Counter-Strike: Global Offensive and Dota 2 items with zero hassle.
- teamfortress2
- steam
- steambot
- steam-api
A package to fetching the maker profile - Super Mario Maker
- json
- mario-maker
- nodejs
- javascript