Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

All-in-one IoT Platform - Device management, data collection, processing and visualization.

• visualization
• iot
• iot-platform
• mqtt

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

• deep-learning
• document-parsing
• machine-learning
• nlp

Logstash - transport and process your logs, events, or other data

• etl-framework
• streaming
• logging
• java

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl

Building event-driven applications the easy way in Go.

• watermill
• event-driven
• event-sourcing
• sagas

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus

Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

• data
• data-cleaning
• data-pipelines
• data-processing

Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows

• fluentd
• c
• logging
• data-collector

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

• data-analysis
• data-science
• large-language-models
• llm

Python Stream Processing

• kafka-streams
• kafka
• python
• asyncio

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL

• rust
• database
• sql
• streaming

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

• machine-learning
• python
• data-engineering
• distributed-computing

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

• cloud-native
• streaming
• rust
• real-time

Distributed stream processing engine in Rust

• data
• dev-tools
• infrastructure
• kafka

A lightweight data processing framework built on DuckDB and 3FS.

• data-processing
• duckdb

A light-weight, flexible, and expressive statistical data testing library

• pandas
• validation
• schema
• dataframes