Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
109,091 visible repositories in the index
1–24 of 93 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
Apache Flink
All-in-one IoT Platform - Device management, data collection, processing and visualization.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Open-Source Web UI for Apache Kafka Management
CMAK is a tool for managing Apache Kafka clusters
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Apache Beam is a unified programming model for Batch and Streaming data processing.
The Open Source Feature Store for AI/ML
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Upserts, Deletes And Incremental Processing on Big Data.
Data-Centric Pipelines and Data Versioning
A Cloud Native Batch System (Project under CNCF)
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Apache Calcite
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
Apache Ignite
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
Privacy and Security focused Segment-alternative, in Golang and React