Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–24 of 94 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
Always know what to expect from your data.
Incremental engine for long horizon agents 🌟 Star if you like it!
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
8 Lessons, Kick-start Your Cybersecurity Learning.
Countly is a privacy-first, AI-powered analytics and engagement platform for understanding and optimizing customer journeys across digital applications, from desktop and mobile to IoT and connected environments.
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Open-Source Personal Cloud OS for Always-On Agents
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Agent-ready RPA suite with out-of-the-box automation tools. Built for individuals and enterprises.
A lightweight data processing framework built on DuckDB and 3FS.
A light-weight, flexible, and expressive statistical data testing library
Visualize and compare datasets, target values and associations, with one line of code.
Kubernetes-native platform to run massively parallel data/streaming jobs
End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow