Elegant Scraper and Crawler Framework for Golang
Repository Index
132,241 visible repositories in the index
1–24 of 45 · Most stars
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
TranslatedZotero translators for Chinese web page scraping 🎉 This is Zotero translators for Chinese Sites(beta), not the official Zotero repo
- zotero
- translators
- javascript
- cnki
TranslatedToday's Hot List (今日热榜), an aggregator website that gets the hot headlines from major popular websites. Written in Go, it uses multiple goroutines to fetch information asynchronously and quickly. Preview: https://mo.fish
- hot
- golang
- spider
- today-s-hot-list
An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.
- ai-agent
- automation
- data-extraction
- event-driven-architecture
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
freehire — the open-source search engine for job seekers
- ats
- careers
- hiring
- job-aggregator
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
Node.js scraper to get data from Google Play
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
- search-engine
- python
- scraping
- crawler
Fast website link checker in Go
Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.
多播放源自动采集在线影视网站 golang vue
- axios
- elementplus
- film
- gin
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
Web Scraper in Go, similar to BeautifulSoup
- golang
- go
- webscraper
- webscraping
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
- google-scholar
- google-scholar-scraper
- python
- python-scraper
Retrieve author and publication information from Google Scholar in a friendly, Pythonic way without having to worry about CAPTCHAs!
- scholar
- googlescholar
- scholarly-articles
- scholarly-communications
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
- python
- html5
- css
- web-scraping
Golang实现的IP代理池
- go
- ip
- proxypool
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
- automation
- playwright
- python
- scraping