Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–24 of 40 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
Always know what to expect from your data.
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
Agent-ready RPA suite with out-of-the-box automation tools. Built for individuals and enterprises.
A lightweight data processing framework built on DuckDB and 3FS.
A light-weight, flexible, and expressive statistical data testing library
Visualize and compare datasets, target values and associations, with one line of code.
End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Data Contracts engine for the modern data stack. https://www.soda.io
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
Python Stream Processing
Scalable data pre processing and curation toolkit for LLMs
Optimized Agentic and LLM Bulk Processing Over Your Data
Machine learning with dataframes
Extract Transform Load for Python 3.5+
Python Streaming DataFrames for Kafka
Data and tools for generating and inspecting OLMo pre-training data.