Repository Index

Elegant Scraper and Crawler Framework for Golang

• golang
• scraper
• framework
• crawler

A next-generation crawling and spidering framework.

• crawler
• web-spider
• gocrawler
• spider-framework

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

Pholcus is a distributed high-concurrency crawler software written in pure golang

• spider
• crowler

A Chrome DevTools Protocol driver for web automation and scraping.

• cdp
• chrome-headless
• chrome-devtools
• chrome-devtools-protocol

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

• bugbounty
• crawling
• hacking
• osint

Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes

• crawler
• proxypool

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

• ai
• anti-detect
• golang
• markdown

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

• endpoints
• endpoint-discovery
• bugbounty
• crawler

A powerful browser crawler for web vulnerability scanners

• headless-chrome
• crawler
• chrome-devtools
• golang

Gospider - Fast web spider written in Go

• spider
• crawler
• go
• gospider

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

• go
• scraping
• scraper
• crawler

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

• golang
• go
• cloning
• crawler

Extract structured data from web sites. Web sites scraping.

• golang
• golang-library
• extract-data
• chrome-fetcher

The unix-way web crawler

• cli
• crawler
• golang-application
• unix-way

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

• golang
• web-crawler
• crawling
• scraping

Extract web archive data using Wayback Machine and Common Crawl

• golang
• crawler
• commoncrawl
• wayback-machine

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

• go
• golang
• goquery
• colly

Confluence Cloud to Markdown crawler with incremental updates and local link rewriting