Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
96,838 visible repositories in the index
1–24 of 33 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Logstash - transport and process your logs, events, or other data
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
Incremental engine for long horizon agents 🌟 Star if you like it!
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
A data integration framework
The Data Transfer Project makes it easy for platforms to build interoperable user data portability features. We are establishing a common framework, including data models and protocols, to enable direct transfer of data both into and out of participating online service providers.
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Scalable and efficient data transformation framework - backwards compatible with dbt.
Spring Batch is a framework for writing batch applications using Java and Spring
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
高性能表格数据导出器
Data processing & ETL framework for Ruby
cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
Python Streaming DataFrames for Kafka
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Apache InLong - a one-stop, full-scenario integration framework for massive data
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
The simple, stupid batch framework for Java
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration