Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
71,235 visible repositories in the index
1–24 of 24 · Most stars
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Logstash - transport and process your logs, events, or other data
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
RedisShake is a Valkey/Redis data processing and migration tool.
- redis
- redis-cluster
- redis-proxy
A system for agentic LLM-powered data processing and ETL
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
- machine-learning
- convolutional-networks
- embedding-vectors
- embeddings
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Spring Batch is a framework for writing batch applications using Java and Spring
- batch
- batch-processing
- java
- spring
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- claude-code
- codex
- data-processing
- harness-engineering
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- sql
- redpanda
- single-binary
Data processing & ETL framework for Ruby
- etl
- etl-ruby
- ruby
- data
Python Streaming DataFrames for Kafka
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
The simple, stupid batch framework for Java
- java
- batch-processing
- batch
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
- java
- smooks
- enterprise-integration
- etl
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
Code to download, process, and analyse the Online Encyclopedia of Integer Sequences
Internet Archive's Sparkling Data Processing Library
High performant ETL engine written in TypeScript
- data-processing
- etl
- etl-framework
- streaming-data