Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–24 of 37 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Always know what to expect from your data.
Incremental engine for long horizon agents 🌟 Star if you like it!
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
A lightweight data processing framework built on DuckDB and 3FS.
A light-weight, flexible, and expressive statistical data testing library
Kubernetes-native platform to run massively parallel data/streaming jobs
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Concurrent and multi-stage data ingestion and data processing with Elixir
Data Contracts engine for the modern data stack. https://www.soda.io
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
Python Stream Processing
Scalable data pre processing and curation toolkit for LLMs
Extract Transform Load for Python 3.5+
Stream Processing and Complex Event Processing Engine
Python Streaming DataFrames for Kafka
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Concurrent Python made simple
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.