TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Repository Index
47,133 repositories indexed
1–14 of 14 · Most stars
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
Maze Applied Reinforcement Learning Framework
The type-safe, event-driven workflow orchestration library that scales with your business. Build robust, distributed workflows in Go with compile-time safety, automatic retries, and horizontal scaling out of the box. Integrate with your existing tech stack using adapters 🚀
- eventdrivenarchitecture
- golang
- kafka
- reflex
state machine which runs code
- golang
- workflows
- state-machine
- actor-model
Ratus is a RESTful asynchronous task queue server. It translated concepts of distributed task queues into a set of resources that conform to REST principles and provides a consistent HTTP API for various backends.
- distributed-systems
- priority-queue
- task-queue
- task-scheduler
Distributed batch data processing framework
Durable execution framework for AI agents in Go. Keeps agent state, tool calls, and execution loops resilient across process crashes.
- agentic-ai
- ai
- ai-agents
- golang
Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.
- ruby
- rake
- pwrake
- parallel
Distributed Automation for XNAT
- xnat
- python
- dax
Distributed, Scalable, Fault-Tolerant, Minimalistic Workflow Engine - No DAGs, No YAML, No Cumbersome Diagrams, Just Code
- business-process
- kafka
- kafka-streams
- micorservices
Go-native distributed job scheduler with delayed tasks, batched Redis coordination, retries, lease-based recovery, and versioned queue partitioning.
- distributed
- distributed-scheduler
- golang
- scheduler