Repository Index

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

Go-native distributed job scheduler with delayed tasks, batched Redis coordination, retries, lease-based recovery, and versioned queue partitioning.

• distributed
• distributed-scheduler
• golang
• scheduler