Repository Index

UAParser.js: The Essential Web Development Tool for User-Agent Detection. Detect Browsers, OS, Devices, Bots, Apps, AI Crawlers, and more. Run in Browser or Node

• user-agent-parser
• browser-detection
• user-agent
• client-hints

TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database

• javbus
• avmoo
• javlibrary
• spider

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• crawler
• javascript
• spider
• extract-data

Shattered Pixel Dungeon is an open-source traditional roguelike dungeon crawler with randomized levels and enemies, and hundreds of items to collect and use. It's based on the source code of Pixel Dungeon, by Watabou.

• android
• game
• game-development
• ios

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

Open Source Web Crawler for Java

A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.

• bittorrent
• dht
• prowlarr
• radarr

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

• crawler
• dotnetcore
• cross-platform
• csharp

Headless Chrome .NET API

• puppeteer
• chrome
• chromium
• automation

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

• php
• user-agent
• crawler
• spider

Find web directories without bruteforce

• pentesting
• security
• security-tools
• websec

Free database schema discovery and comprehension tool

• schemacrawler
• jdbc
• database-diagrams
• schema

Async Python 3.6+ web scraping micro-framework based on asyncio

• asyncio
• aiohttp
• asyncio-spider
• crawler

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• ruby
• scraper
• crawler
• dsl

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

• csrf
• crawler
• csrf-attacks
• token-generation

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

• java
• elasticsearch
• full-text-search
• lucene

Crawly, a high-level web crawling & scraping framework for Elixir.

• elixir
• erlang
• scraper
• scraping

Generate smart and powerful wordlists

• wordlist
• password
• hacking-tool
• password-cracking

Cross-platform port of Arx Fatalis, a first-person role-playing game

• opengl
• game
• c-plus-plus
• cmake

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.

• agent-skills
• claude-code
• codex
• geo

A high performance web crawler / scraper in Elixir.

• elixir
• crawler
• spider
• scraper