This is a repo with links to everything you'd ever want to learn about data engineering
Repository Index
47,133 repositories indexed
1–24 of 27 · Most stars
RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
大数据入门指南 :star:
A curated list of awesome big data frameworks, ressources and other awesomeness.
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
Upserts, Deletes And Incremental Processing on Big Data.
Apache Avro is a data serialization system.
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.
Distributed Big Data Orchestration Service
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
Lightweight real-time big data streaming engine over Apache Pekko
Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️
This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
High-performance, time-indexed object storage for robotics and industrial IoT
AthenaCLI is a CLI tool for AWS Athena service that can do auto-completion and syntax highlighting.
BENERATOR is a leading software solution to generate, obfuscate, pseudonymize and migrate data for development, testing, and training purposes with a model-driven approach.