Repository Index

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

Real-time monitor and web admin for Celery distributed task queue

• celery
• task-queue
• monitoring
• administration

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

• crawler
• dotnetcore
• cross-platform
• csharp

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

Distributed Automation for XNAT

• xnat
• python
• dax

Distributed, Scalable, Fault-Tolerant, Minimalistic Workflow Engine - No DAGs, No YAML, No Cumbersome Diagrams, Just Code

• business-process
• kafka
• kafka-streams
• micorservices

Go-native distributed job scheduler with delayed tasks, batched Redis coordination, retries, lease-based recovery, and versioned queue partitioning.

• distributed
• distributed-scheduler
• golang
• scheduler