Repository Index

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

Open Source Web Crawler for Java

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

• crawler
• dotnetcore
• cross-platform
• csharp

Headless Chrome .NET API

• puppeteer
• chrome
• chromium
• automation

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

• php
• user-agent
• crawler
• spider

Find web directories without bruteforce

• pentesting
• security
• security-tools
• websec

Async Python 3.6+ web scraping micro-framework based on asyncio

• asyncio
• aiohttp
• asyncio-spider
• crawler

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• ruby
• scraper
• crawler
• dsl

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

• java
• elasticsearch
• full-text-search
• lucene

Crawly, a high-level web crawling & scraping framework for Elixir.

• elixir
• erlang
• scraper
• scraping

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

A high performance web crawler / scraper in Elixir.

• elixir
• crawler
• spider
• scraper

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

• spider
• ruby
• spider-links
• crawler

Java API For Chrome and Firefox

• puppeteer
• java
• crawler
• chrome-headless

Translated✈️ Self-hosted multi-channel airfare price monitoring: flight detail collection → DingTalk rich-media push → four-layer strong alerts when the target price is met. Domestic routes and international routes (destinations on five continents) share the same collect-judge-push pipeline; data stays on your own machine, all configuration is done on a web page, open source and free.

• crawler
• dingtalk
• flight
• playwright

The unix-way web crawler

• cli
• crawler
• golang-application
• unix-way

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

• golang
• web-crawler
• crawling
• scraping

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl

:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules

• cloudflare
• firewall
• user-agent
• firewall-rules

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi

Extract web archive data using Wayback Machine and Common Crawl

• golang
• crawler
• commoncrawl
• wayback-machine

Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.web.app

• api
• redash
• tracking
• crawler

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

• go
• golang
• goquery
• colly