Repository Index

Elegant Scraper and Crawler Framework for Golang

• golang
• scraper
• framework
• crawler

A next-generation crawling and spidering framework.

• crawler
• web-spider
• gocrawler
• spider-framework

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

A Chrome DevTools Protocol driver for web automation and scraping.

• cdp
• chrome-headless
• chrome-devtools
• chrome-devtools-protocol

Extract structured data from web sites. Web sites scraping.

• golang
• golang-library
• extract-data
• chrome-fetcher

The unix-way web crawler

• cli
• crawler
• golang-application
• unix-way

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

• golang
• web-crawler
• crawling
• scraping

Extract web archive data using Wayback Machine and Common Crawl

• golang
• crawler
• commoncrawl
• wayback-machine

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

• go
• golang
• goquery
• colly

Confluence Cloud to Markdown crawler with incremental updates and local link rewriting