Repository Index

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.

• ai
• claude
• content-extraction
• go

A toolkit makes it easier to archive webpages to IPFS

• archiving
• webpage
• ipfs