👾 Fast and simple video download library and CLI tool written in Go
Repository Index
72,362 visible repositories in the index
1–24 of 27 · Most stars
Elegant Scraper and Crawler Framework for Golang
Weighs the soul of incoming HTTP requests to stop AI crawlers
- defense
- security
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
Declarative data automation language and Go runtime for structured extraction workflows.
- golang
- query-language
- dsl
- go
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
- gau
- wayback-machine
- hacktoberfest
- security
A Mighty CLI for AWS
A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
- bittorrent
- dht
- prowlarr
- radarr
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
- crawler
- proxypool
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
Gospider - Fast web spider written in Go
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
- bugbounty
- bugbounty-tool
- crawler
- dirscanner
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
teler-waf is a Go HTTP middleware that protects local web services from OWASP Top 10 threats, known vulnerabilities, malicious actors, botnets, unwanted crawlers, and brute force attacks.
- ids
- teler
- waf
- teler-ids
The unix-way web crawler
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping