47,133 repositories indexed
1–24 of 39 · Most stars
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
A high-performance observability data pipeline.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
An orchestration platform for the development, production, and observation of data assets.
Premium Queue package for handling distributed jobs and messages in NodeJS.
Apache Doris is a real-time analytics and hybrid search database for AI agents.
Cloud-native OSS search engine for observability
BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL
Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
The polyglot queue: Fast and reliable background jobs in Go, Java, JS, Ruby, and Rust on Postgres/SQLite.
Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.
Near-zero config .NET library that makes advanced application features like Task Scheduling, Caching, Queuing, Event Broadcasting, and more a breeze!
Queueing jobs in Postgres from Node.js like a boss
The open source and enterprise-ready webhooks service 🦀
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Kubernetes-native platform to run massively parallel data/streaming jobs
REST job server for Apache Spark
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
Ruby and Rails efficient Kafka processing framework
Pluggable foundation blocks for building distributed apps.
A light-weight job scheduling system built on top of Resque