Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–10 of 10 · Most stars
Logstash - transport and process your logs, events, or other data
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
This repository is a getting started guide to Singer.
- etl
- etl-framework
- python
- singer
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
The Frank!Framework is an easy-to-use, stateless integration framework which allows (transactional) messages to be modified and exchanged between different systems.
- java
- system-integration
- open-source
- community-driven
EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.
- dotnet
- dotnet-core
- etl
- etl-framework
High performant ETL engine written in TypeScript
- data-processing
- etl
- etl-framework
- streaming-data
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
- etl
- etl-framework
- etl-pipeline
- python