Repository Index

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

• python
• luigi
• orchestration-framework
• scheduling

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

• workflow-schedule
• azkaban
• airflow
• task-scheduler

Parallel computing with task scheduling

• dask
• python
• pydata
• numpy

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

• workflow
• governance
• azkaban
• davinci

A distributed task scheduler for Dask

• pydata
• dask
• distributed-computing
• python

RedisSMQ: A high-performance Redis message queue for Node.js

• redis
• background-jobs
• brokers
• dead-letter-queue

Simple but powerful DAG scheduler and dashboard

• directed-acyclic-graph
• airflow
• dashboard
• go

Canonical GTFS Validator project for schedule (static) files.

• gtfs-dataset
• gtfs-validator
• gtfs-zip
• gtfs-feed

Modular, Production-Ready, Open-Source EHR

• ehr
• electronic-health-record
• fhir
• health

A simple jobs queue that just works (for Meteor.js)

• meteor
• jobs
• scheduled-tasks
• schedule

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis

Type-safe, persistent, embedded task queues and background job runner w/ SQLite. Web monitoring UI included.

• go
• golang
• job-queue
• job-scheduler

Facilitates simple durable and asynchronous messaging in dotnet apps.

• async
• csharp
• dotnet
• messaging

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

• notebooks
• pyspark-notebook
• juypter-notebook
• spark

A library for scheduling when to dispatch a message to a channel

• streaming
• messaging
• scheduling
• go

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

• spark
• tweets
• twitter-data
• twitter-json

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

• etl
• etl-framework
• etl-pipeline
• python