Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
47,133 repositories indexed
1–19 of 19 · Most stars
Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.
- kafka
- diskless
- automq
- serverless
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
- security
- kinesis
- serverless
- terraform
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
A framework for building messaging apps with .NET and C#.
- command-processor
- command-dispatcher
- brighter
- asynchronously
JeroMQ is a pure Java implementation of the ZeroMQ messaging library, offering high-performance asynchronous messaging for distributed or concurrent applications.
- jeromq
- libzmq
- messaging
- networking
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
An extensible distributed system for reliable nearline data streaming at scale
- distributed-systems
- data-streaming
- scalability
- kafka-mirror-maker
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
Phoebus is a distributed framework for large scale graph processing written in Erlang.
Postgres change events (CDC) in Elixir [WAL]
- cdc
- change-data-capture
- elixir
- postgres
Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.
Scramjet Platform is a distributed runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters without exposing network ports to the public internet.
- nodejs
- stream-processing
- transform-as-a-service
- real-time
A library for scheduling when to dispatch a message to a channel
Production-grade Server-Sent Events (SSE) for Fiber v3. Event coalescing, priority lanes, topic wildcards, adaptive throttling, built-in auth, Prometheus metrics, graceful drain.
- event-streaming
- fasthttp
- fiber
- gofiber