Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
47,133 repositories indexed
1–24 of 41 · Most stars
Apache Flink
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
CMAK is a tool for managing Apache Kafka clusters
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
Distributed Big Data Orchestration Service
- big-data
- bigdata
- orchestration
- configuration
Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.
- database
- data-analysis
- data-warehouse
- olap
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
- apache-spark
- big-data
- connected-components
- dataframe
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
Sparkling Water provides H2O functionality inside Spark cluster
- h2o
- spark
- machine-learning
- integration
🏅State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes
Mirror of Apache Samza
- scala
- big-data
- samza
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Low-code tool for automating actions on real time data | Stream processing for the users.
- flink
- flink-kafka
- apache-flink
- gui
A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.