Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Repository Index
47,133 repositories indexed
1–24 of 27 · Most stars
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.
The open source and enterprise-ready webhooks service 🦀
- webhooks
- rust
- webhook-server
- webhook-service
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
REST job server for Apache Spark
Run MapReduce jobs on Hadoop or Amazon Web Services
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
Ruby and Rails efficient Kafka processing framework
- karafka-framework
- kafka
- kafka-client
- kafka-clients
👜 Tiny and fast reactive/iterable programming library
- callbacks
- reactive
- iterables
- observables
Ultra quick message queue and streaming server
- message-queue
- server
- amqp
- background-jobs
🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications
- event-driven
- generics
- go
- golang
RedisSMQ: A high-performance Redis message queue for Node.js
- redis
- background-jobs
- brokers
- dead-letter-queue
An observable microservice bus library for .NET, that wraps the underlying message transports in simple Rx based interfaces.
- messaging
- microservices
- servicebus
- observable
Queue plugin for CakePHP - simple, pure PHP and without dependencies.
- asynchronous
- background-jobs
- cronjob
- queue
A simple jobs queue that just works (for Meteor.js)
- meteor
- jobs
- scheduled-tasks
- schedule
Type-safe, persistent, embedded task queues and background job runner w/ SQLite. Web monitoring UI included.
- go
- golang
- job-queue
- job-scheduler
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
- antelope
- api
- big-data
- blockchain
Realistic synthetic events for testing, demos, and pipelines — streamed live or generated in bulk
- clickhouse
- data-engineering
- data-generation
- devops
A type-safe queue for background tasks using PocketBase.
- background-jobs
- pocketbase
- queue
- typescript