Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
87,075 visible repositories in the index
1–24 of 30 · Most stars
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Logstash - transport and process your logs, events, or other data
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
RedisShake is a Valkey/Redis data processing and migration tool.
- redis
- redis-cluster
- redis-proxy
A system for agentic LLM-powered data processing and ETL
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
- machine-learning
- convolutional-networks
- embedding-vectors
- embeddings
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Spring Batch is a framework for writing batch applications using Java and Spring
- batch
- batch-processing
- java
- spring
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- claude-code
- codex
- data-processing
- harness-engineering
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
Scalable data pre processing and curation toolkit for LLMs
- data-curation
- llm
- data
- data-prep
Data processing & ETL framework for Ruby
- etl
- etl-ruby
- ruby
- data
Extract Transform Load for Python 3.5+
- data-processing
- extract-transform-load
- bonobo
- python3
Python Streaming DataFrames for Kafka
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
The simple, stupid batch framework for Java
- java
- batch-processing
- batch