For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.
Repository Index
83,087 visible repositories in the index
1–24 of 139 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
CMAK is a tool for managing Apache Kafka clusters
🥊 Tiny 200 byte functional event emitter / pubsub.
- mitt
- event-handlers
- pubsub
- tiny
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
- machine-learning
- computer-vision
- data-format
- deep-learning
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
Dockerfile for Apache Kafka
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
- version-control
- experiment-manager
- version
- control
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
Data-Centric Pipelines and Data Versioning
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time