Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
Repository Index
72,362 visible repositories in the index
1–5 of 5 · Most stars
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
- search-engine
- no-cloud
- small-web
- internet-search
Free database schema discovery and comprehension tool
- schemacrawler
- jdbc
- database-diagrams
- schema
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
DuckDB extension to fetch pages from Wayback Machine & Common Crawl
- archived
- common-crawl
- duckdb
- duckdb-extension