Repository Index

Real time google image scraper. The process is automated by sending HTTP requests to retrieve image data which is then parsed and saved.

• google-image-scraper
• google-images-crawler
• image-scrape
• image-scraper

Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App

• crawl4ai
• documentation
• llm
• playwright

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

• crawler
• crawling
• spider
• webcrawler

The goal of this repository is to document the most common techniques to bypass AppLocker.

• applocker
• bypass
• awl
• rules

A toy physically based GPU path tracer (C++/OpenGL/GLSL)

• glsl
• pathtracer
• raytracer
• opengl

NewPipe's core library for extracting data from streaming sites

• newpipe
• youtube
• crawler
• extractor

Polite, slim and concurrent web crawler.

• crawler
• robots-txt

Find web directories without bruteforce

• pentesting
• security
• security-tools
• websec

Free database schema discovery and comprehension tool

• schemacrawler
• jdbc
• database-diagrams
• schema

Async Python 3.6+ web scraping micro-framework based on asyncio

• asyncio
• aiohttp
• asyncio-spider
• crawler

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• ruby
• scraper
• crawler
• dsl

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

• csrf
• crawler
• csrf-attacks
• token-generation

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

• java
• elasticsearch
• full-text-search
• lucene

Crawly, a high-level web crawling & scraping framework for Elixir.

• elixir
• erlang
• scraper
• scraping

Generate smart and powerful wordlists

• wordlist
• password
• hacking-tool
• password-cracking

Cross-platform port of Arx Fatalis, a first-person role-playing game

• opengl
• game
• c-plus-plus
• cmake

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

• agpl
• ai-agents
• anti-bot
• crawling

SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.

• agent-skills
• claude-code
• codex
• geo

A high performance web crawler / scraper in Elixir.

• elixir
• crawler
• spider
• scraper

Show an agent a problem: mark up, record, send

• ai-agents
• claude-code
• coding-agents
• developer-tools

Flutter UI challenge (with Box2D physic)- Smart washing machine app

• ui
• challenge
• ui-challenge
• flutter-apps

TranslatedSelf-hosted web and AI crawler analytics for Generative Engine Optimization (GEO). Track visitors, AI search referrals, crawl logs, content coverage, and conversions. Jointly published by 猎看科技 (Liekan Technology, "hunt and see technology") · 北京日报社智源GEO (Beijing Daily Zhiyuan GEO, "Beijing Daily Press Zhiyuan GEO"). MIT licensed.

• ai-analytics
• geo
• mit
• postgresql