Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
47,133 repositories indexed
1–13 of 13 · Most stars
Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.
- kafka
- diskless
- automq
- serverless
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
A framework for building messaging apps with .NET and C#.
- command-processor
- command-dispatcher
- brighter
- asynchronously
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
An extensible distributed system for reliable nearline data streaming at scale
- distributed-systems
- data-streaming
- scalability
- kafka-mirror-maker
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
Phoebus is a distributed framework for large scale graph processing written in Erlang.
The type-safe, event-driven workflow orchestration library that scales with your business. Build robust, distributed workflows in Go with compile-time safety, automatic retries, and horizontal scaling out of the box. Integrate with your existing tech stack using adapters 🚀
- eventdrivenarchitecture
- golang
- kafka
- reflex
The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.
- fairness
- fairness-ml
- fairness-ai
Scramjet Platform is a distributed runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters without exposing network ports to the public internet.
- nodejs
- stream-processing
- transform-as-a-service
- real-time