Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–24 of 38 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
Apache Calcite
The Data Transfer Project makes it easy for platforms to build interoperable user data portability features. We are establishing a common framework, including data models and protocols, to enable direct transfer of data both into and out of participating online service providers.
The open source and enterprise-ready webhooks service 🦀
Scalable and efficient data transformation framework - backwards compatible with dbt.
In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.
A cluster computing framework for processing large-scale geospatial data
Unofficial open source mtproto server written in golang with compatible telegram client.
[Go] Lightweight eventbus with async compatibility for Go
Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake
Lightweight and extensible compatibility layer between dataframe libraries!
parallel graph management and execution in heterogeneous computing
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.
A compatibility layer for event-driven abstractions
The easiest way to get started with event sourcing in Laravel
The Apache Flink SQL Cookbook is a curated collection of examples, patterns, and use cases of Apache Flink SQL. Many of the recipes are completely self-contained and can be run in Ververica Platform as is.
Opinionated Domain Driven Design Template for Go
A smart set of common classes and implementations to improve your development productivity.
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Event Sourcing, CQRS and DDD framework for C#/.NET Core.