Real time google image scraper. The process is automated by sending HTTP requests to retrieve image data which is then parsed and saved.
- google-image-scraper
- google-images-crawler
- image-scrape
- image-scraper
Repository Index
72,862 visible repositories in the index
97–120 of 135 · Most stars
Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App
- crawl4ai
- documentation
- llm
- playwright
蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统
NewPipe's core library for extracting data from streaming sites
Polite, slim and concurrent web crawler.
- crawler
- robots-txt
Find web directories without bruteforce
- pentesting
- security
- security-tools
- websec
Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
- fredy
- immowelt
- appartment
- appartments
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
Crawly, a high-level web crawling & scraping framework for Elixir.
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
A high performance web crawler / scraper in Elixir.
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
Translated✈️ Self-hosted multi-channel airfare price monitoring: flight detail collection → DingTalk rich-media push → four-layer strong alerts when the target price is met. Domestic routes and international routes (destinations on five continents) share the same collect-judge-push pipeline; data stays on your own machine, all configuration is done on a web page, open source and free.
- crawler
- dingtalk
- flight
- playwright
Xiryn — the open-source personal agent. Does everything Muse, Grok Bot and Dots do, on your own computer: each agent gets its own Linux desktop in Docker, any model with your keys, an encrypted vault, and control from Telegram.
WarcDB: Web crawl data as SQLite databases.
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
The unix-way web crawler
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
- golang
- web-crawler
- crawling
- scraping