This is a repo with links to everything you'd ever want to learn about data engineering
Repository Index
47,133 repositories indexed
1–24 of 35 · Most stars
RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
TranslatedBig Data Beginner's Guide :star:
- hadoop
- hdfs
- yarn
- mapreduce
A curated list of awesome big data frameworks, ressources and other awesomeness.
- awesome
- bigdata
- data-warehouse
- data-analytics
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
TranslatedFocused on big data learning and interviews, the road to becoming a big data god begins. Flink/Spark/Hadoop/Hbase/Hive...
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
- dataframe
- python
- bigdata
- tabular-data
A JavaScript library for arbitrary-precision decimal and non-decimal arithmetic
- bignumber
- javascript
- arbitrary-precision
- decimal-places
Upserts, Deletes And Incremental Processing on Big Data.
- hudi
- apachehudi
- datalake
- bigdata
Apache Avro is a data serialization system.
- csharp
- ruby
- cplusplus
- python
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
Arbitrary-precision arithmetic library for PHP
- php
- arbitrary-precision
- biginteger
- bigdecimal
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.
Distributed Big Data Orchestration Service
- big-data
- bigdata
- orchestration
- configuration
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
Lightweight real-time big data streaming engine over Apache Pekko
- akka
- stream-processing
- bigdata
- scala
Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️
This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
High-performance, time-indexed object storage for robotics and industrial IoT
- blob-storage
- storage-engine
- storage
- historian