The unix-way web crawler
Repository Index
55,640 visible repositories in the index
97–120 of 133 · Most stars
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Mac app for website auditing and crawling
- seo
- website-auditing
- swift
- mac
Roguelite dungeon crawler game
:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules
- cloudflare
- firewall
- user-agent
- firewall-rules
TranslatedAll-in-one CLI & Skill to download anything: videos, docs, cloud drives, music, papers & books. An all-in-one download and archiving tool for AI Agents and terminals.
- ai-agent
- claude-code
- cli
- downloader
A Payload CMS 3 plugin for Auth.js/NextAuth 5
- payload-plugin
- payloadcms
- authjs
- next-auth
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Voight-Kampff is a Ruby gem that detects bots, spiders, crawlers and replicants
Extract web archive data using Wayback Machine and Common Crawl
- golang
- crawler
- commoncrawl
- wayback-machine
Astray is a lua based maze, room and dungeon generation library for dungeon crawlers and rougelike video games
- lua
- maze
- room
- rougelike
:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app
Egoboo is a working cool 3D dungeon crawling game in the spirit of NetHack for Windows, Linux and Mac.
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
- go
- golang
- goquery
- colly
Ragpicker is a Plugin based malware crawler with pre-analysis and reporting functionalities. Use this tool if you are testing antivirus products, collecting malware for another analyzer/zoo.
a dungeon crawler written in TypeScript using React and svg
Various Jupyter notebooks about Common Crawl data
- jupyter-notebook
- common-crawl
- aws-athena
- webarchiving
A first person dungeon-crawler for DOS, written in C++, using software rendering and fixed point math
- dungeon
- sdl
- msdos
- rpg
Appium Native Crawler CLI - Features include: Screenshots, Performance, Accessibility Detection, Google Translate, Applitools, Monkey Tester
Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ideal for PWAs, SPAs, and modern JS-driven apps, websites, and webpages
- meteor
- middleware
- crawler
- seo
Take HTML Snapshots for your Angular, Ember, Durandal or any JavaScript applications
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action