Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
Repository Index
47,133 repositories indexed
1–5 of 5 · Most stars
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
- anti-crawling
- cloud-native
- crawler
- deception
:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules
- cloudflare
- firewall
- user-agent
- firewall-rules
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action