Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
Repository Index
47,133 repositories indexed
1–14 of 14 · Most stars
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
The open source and enterprise-ready webhooks service 🦀
- webhooks
- rust
- webhook-server
- webhook-service
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
REST job server for Apache Spark
Run MapReduce jobs on Hadoop or Amazon Web Services
Ruby and Rails efficient Kafka processing framework
- karafka-framework
- kafka
- kafka-client
- kafka-clients
Ultra quick message queue and streaming server
- message-queue
- server
- amqp
- background-jobs
RedisSMQ: A high-performance Redis message queue for Node.js
- redis
- background-jobs
- brokers
- dead-letter-queue
Simple but powerful DAG scheduler and dashboard
- directed-acyclic-graph
- airflow
- dashboard
- go
Joblib Apache Spark Backend
Build and deploy a serverless data pipeline on AWS with no effort.
- aws-cdk
- data-pipeline
- stepfunctions
- glue
AsyncJob is an asynchronous queue job manager with light code, clear and speed. I hope so ! 😬
- async
- queue
- simple
- lightweight