Repository Index

This is a repo with links to everything you'd ever want to learn about data engineering

• apachespark
• awesome
• bigdata
• data

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

• bigdata
• cloud-native
• filesystem
• object-storage

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

• iot
• bigdata
• time-series
• database

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

TranslatedBig Data Beginner's Guide :star:

• hadoop
• hdfs
• yarn
• mapreduce

A curated list of awesome big data frameworks, ressources and other awesomeness.

• awesome
• bigdata
• data-warehouse
• data-analytics

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

• filesystem
• cloud-native
• golang
• redis

TranslatedFocused on big data learning and interviews, the road to becoming a big data god begins. Flink/Spark/Hadoop/Hbase/Hive...

• flink
• spark
• hadoop
• hdfs

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

• dataframe
• python
• bigdata
• tabular-data

Upserts, Deletes And Incremental Processing on Big Data.

• hudi
• apachehudi
• datalake
• bigdata

Apache Avro is a data serialization system.

• csharp
• ruby
• cplusplus
• python

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

• database
• griddb
• nosql
• iot

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

• spark
• csharp
• dotnet
• analytics

An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.

• business-intelligence
• sql
• dashboard
• data-visualization

Distributed Big Data Orchestration Service

• big-data
• bigdata
• orchestration
• configuration

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

• spark
• big-data
• bioinformatics
• genomics

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

• livy
• bigdata
• spark
• apachelivy

Lightweight real-time big data streaming engine over Apache Pekko

• akka
• stream-processing
• bigdata
• scala

Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️

• acid
• database
• key-value-store
• document-database

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x

• spark
• kotlin
• bigdata
• scala

High-performance, time-indexed object storage for robotics and industrial IoT

• blob-storage
• storage-engine
• storage
• historian

AthenaCLI is a CLI tool for AWS Athena service that can do auto-completion and syntax highlighting.

• aws-cli
• athena-cli
• aws-athena
• python

BENERATOR is a leading software solution to generate, obfuscate, pseudonymize and migrate data for development, testing, and training purposes with a model-driven approach.

• testdata
• anonymization
• migration
• performance-testing