Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
104,105 visible repositories in the index
1–24 of 31 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
Logstash - transport and process your logs, events, or other data
Incremental engine for long horizon agents 🌟 Star if you like it!
Fancy stream processing made operationally mundane
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
RedisShake is a Valkey/Redis data processing and migration tool.
A system for agentic LLM-powered data processing and ETL
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Spring Batch is a framework for writing batch applications using Java and Spring
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
Bifrost ---- 面向生产环境的 MySQL,MariaDB,kafka 同步到Redis,MongoDB,ClickHouse,StarRocks,Doris,Kafka等服务的异构中间件
Scalable data pre processing and curation toolkit for LLMs
Data processing & ETL framework for Ruby
Extract Transform Load for Python 3.5+
Python Streaming DataFrames for Kafka
Concurrent Python made simple
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.