Repository Index

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

• rust
• database
• serverless
• bigdata

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

• machine-learning
• computer-vision
• data-format
• deep-learning

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

• big-data
• data-ingestion
• flink
• paimon

DuckLake is an integrated data lake and catalog format

A native Rust library for Delta Lake, with bindings into Python

• delta
• rust
• delta-lake
• databricks

An open lakehouse engine — Postgres-native, vector-native and Python-native. Speaks PostgreSQL, runs on your object storage, written in Rust.

• lakehouse
• spark
• flink
• streaming

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

• data
• spark
• java
• iceberg

Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.

• apache-iceberg
• apache-spark
• data-engineering
• delta-lake