Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
Repository Index
55,640 visible repositories in the index
1–6 of 6 · Most stars
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Free database schema discovery and comprehension tool
- schemacrawler
- jdbc
- database-diagrams
- schema
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
DuckDB extension to fetch pages from Wayback Machine & Common Crawl
- archived
- common-crawl
- duckdb
- duckdb-extension