Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–24 of 51 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Flink CDC is a streaming data integration tool
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
Privacy and Security focused Segment-alternative, in Golang and React
A system for agentic LLM-powered data processing and ETL
A data integration framework
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
A Dockerized python Script to fetch Garmin health data and populate that in a InfluxDB Database, for visualization long term health trends with Grafana
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
🔥🔥 AllData大模型与AI数据底座,以数据平台为底座,以数据中台为桥梁,以机器学习平台为工厂,以大模型应用为上游产品,提供全链路数字化解决方案。 产品手册:https://www.yuque.com/aolingdata/product 产品演示+AI社群+商务采购:https://docs.qq.com/doc/DVHlkSEtvVXVCdEFo
深圳地铁大数据客流分析系统🚇🚄🌟
Flow collector, enricher and visualizer
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
LarkMidTable 是一站式开源的数据中台,实现中台的 基础建设,数据治理,数据开发,监控告警,数据服务,数据的可视化,实现高效赋能数据前台并提供数据服务的产品。
Markdown 批量导出工具、开放式跨平台博客解决方案,随意组合写作平台(语雀/Notion/FlowUs/飞书/我来Wolai)和博客平台(Hexo/Vitepress/Halo/Confluence/WordPress等)
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.