Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–13 of 13 · Most stars
SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.
Logstash - transport and process your logs, events, or other data
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Data processing & ETL framework for Ruby
- etl
- etl-ruby
- ruby
- data
Apache InLong - a one-stop, full-scenario integration framework for massive data
- inlong
- one-stop-service
- data-streaming
- event-streaming
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
The simple, stupid batch framework for Java
- java
- batch-processing
- batch
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation