Incredibly fast crawler designed for OSINT.
Repository Index
72,362 visible repositories in the index
1–22 of 22 · Most stars
TranslatedOneForAll is a powerful subdomain collection tool
- subdomain
- subdomain-scanner
- subdomain-enumeration
- subdomain-bruteforcing
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
- gau
- wayback-machine
- hacktoberfest
- security
The Ultimate Information Gathering Toolkit
- cms-detection
- directory-finder
- dns-lookup
- information-gathering
A list of AI agents and robots to block.
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
- scanner
- crawler
- information-gathering
- admin-scanner
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
All In One Web Recon
- python3
- pentesting
- pentest-tool
- pentesting-tools
Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
- bugbounty
- bugbounty-tool
- crawler
- dirscanner
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
- crawler
- information-gathering
- cms-detector
- cloudflare-detection
Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App
- crawl4ai
- documentation
- llm
- playwright
The goal of this repository is to document the most common techniques to bypass AppLocker.
- applocker
- bypass
- awl
- rules
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
- csrf
- crawler
- csrf-attacks
- token-generation
Generate smart and powerful wordlists
- wordlist
- password
- hacking-tool
- password-cracking
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
- anti-crawling
- cloud-native
- crawler
- deception
teler-waf is a Go HTTP middleware that protects local web services from OWASP Top 10 threats, known vulnerabilities, malicious actors, botnets, unwanted crawlers, and brute force attacks.
- ids
- teler
- waf
- teler-ids
:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules
- cloudflare
- firewall
- user-agent
- firewall-rules
Ragpicker is a Plugin based malware crawler with pre-analysis and reporting functionalities. Use this tool if you are testing antivirus products, collecting malware for another analyzer/zoo.
A Python DNS crawler to find identical domain names under different TLDs.
- python
- cybersecurity
- dns
- crawler