For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.
Repository Index
100,822 visible repositories in the index
1–24 of 238 · Most stars
Open Data Platform for analysts, quants and AI agents.
- python
- machine-learning
- finance
- stocks
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Run frontier AI locally.
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Logstash - transport and process your logs, events, or other data
Node-based Visual Programming Toolbox
- photogrammetry
- computer-vision
- 3d-reconstruction
- alicevision
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
- fluentd
- c
- logging
- data-collector
Broadcast, Presence, and Postgres Changes via WebSockets
- elixir
- postgres
- postgresql
- realtime
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
- ai-agents
- data-engineering
- document-ai
- generative-ai
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷