Repository Index

👾 Fast and simple video download library and CLI tool written in Go

• downloader
• go
• crawler
• scraper

Elegant Scraper and Crawler Framework for Golang

• golang
• scraper
• framework
• crawler

Weighs the soul of incoming HTTP requests to stop AI crawlers

• defense
• security

A next-generation crawling and spidering framework.

• crawler
• web-spider
• gocrawler
• spider-framework

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.

• bittorrent
• dht
• prowlarr
• radarr

teler-waf is a Go HTTP middleware that protects local web services from OWASP Top 10 threats, known vulnerabilities, malicious actors, botnets, unwanted crawlers, and brute force attacks.

• ids
• teler
• waf
• teler-ids

The unix-way web crawler

• cli
• crawler
• golang-application
• unix-way

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

• golang
• web-crawler
• crawling
• scraping

Extract web archive data using Wayback Machine and Common Crawl

• golang
• crawler
• commoncrawl
• wayback-machine

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

• go
• golang
• goquery
• colly

Confluence Cloud to Markdown crawler with incremental updates and local link rewriting