Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
142,273 visible repositories in the index
1–24 of 29 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
The leader in Customer Data Infrastructure
A system for agentic LLM-powered data processing and ETL
🔥🔥 AllData大模型与AI数据底座,以数据平台为底座,以数据中台为桥梁,以机器学习平台为工厂,以大模型应用为上游产品,提供全链路数字化解决方案。 产品手册:https://www.yuque.com/aolingdata/product 产品演示+AI社群+商务采购:https://docs.qq.com/doc/DVHlkSEtvVXVCdEFo
Flow collector, enricher and visualizer
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Markdown 批量导出工具、开放式跨平台博客解决方案,随意组合写作平台(语雀/Notion/FlowUs/飞书/我来Wolai)和博客平台(Hexo/Vitepress/Halo/Confluence/WordPress等)
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Hop Orchestration Platform
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
Python Fast Dataflow programming framework for Data pipeline work( Web Crawler,Machine Learning,Quantitative Trading.etc)
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Input pipeline framework
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
The data transformation tool the open-source community deserves.