Elegant Scraper and Crawler Framework for Golang
Repository Index
132,241 visible repositories in the index
1–24 of 29 · Most stars
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
- crawler
- proxypool
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
Gospider - Fast web spider written in Go
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
Polite, slim and concurrent web crawler.
- crawler
- robots-txt
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
open source grok bot. gawk bots automate your menial work via AI models and build you microapps to manage the outcome, so that you have a false sense of control.
- agents
- autonomous-agents
- claude-code
- knowledge-graph
A Unix-style personal search engine and web crawler for your digital footprint.
- search
- poseidon
- webcrawler
- personal-search
The fastest dork scanner written in Go.
- golang
- dork-scanner
- crawler
- dorking
Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...
小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对
- biquge
- qidian
- fiction
- novels
一个方便安全研究人员获取每日安全日报的爬虫和推送程序,目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客,持续更新中。
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
- web-scraping
- man-in-the-middle
- proxy-server
- mitm
:paw_prints: Creeper - The Next Generation Crawler Framework (Go)
Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.
- crawler
- passive-vulnerability-scanner
- poc
- sql-injection
OSINT Swiss Army Knife
- go
- golang
- osint
- haveibeenpwnd