Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–13 of 13 · Most stars
The open source and enterprise-ready webhooks service 🦀
- webhooks
- rust
- webhook-server
- webhook-service
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
REST job server for Apache Spark
Run MapReduce jobs on Hadoop or Amazon Web Services
Ruby and Rails efficient Kafka processing framework
- karafka-framework
- kafka
- kafka-client
- kafka-clients
Ultra quick message queue and streaming server
- message-queue
- server
- amqp
- background-jobs
RedisSMQ: A high-performance Redis message queue for Node.js
- redis
- background-jobs
- brokers
- dead-letter-queue
A simple jobs queue that just works (for Meteor.js)
- meteor
- jobs
- scheduled-tasks
- schedule
Type-safe, persistent, embedded task queues and background job runner w/ SQLite. Web monitoring UI included.
- go
- golang
- job-queue
- job-scheduler
A type-safe queue for background tasks using PocketBase.
- background-jobs
- pocketbase
- queue
- typescript
A micro workers class that easily manages a pool of threads to optimise simultaneous jobs and jobs endings, for MicroPython (used on Pycom modules & ESP32)
- pycom
- micropython
- lopy
- wipy