For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.
Repository Index
57,138 visible repositories in the index
1–24 of 592 · Most stars
Open Data Platform for analysts, quants and AI agents.
- python
- machine-learning
- finance
- stocks
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Run frontier AI locally.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
Apache Kafka - A distributed event streaming platform
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
TranslatedAlibaba's MySQL binlog incremental subscription & consumption component
Apache Flink
A realtime distributed messaging platform
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- python
- workflow
- data-engineering
- data-science
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
High-Performance server for NATS.io, the cloud and edge native messaging system.
- go
- golang
- messaging
- message-bus
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more
- go
- golang
- data-structure
- map
TranslatedDataX is the open-source version of Alibaba Cloud DataWorks Data Integration.
The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles