Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.
- kafka
- diskless
- automq
- serverless
Repository Index
47,133 repositories indexed
49–72 of 665 · Most stars
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Apache Iceberg
- iceberg
- apache
- hacktoberfest
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Efficient reliable UDP unicast, UDP multicast, and IPC message transport
- messaging
- java
- c-plus-plus
- ipc
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Pentaho Data Integration ( ETL ) a.k.a Kettle
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
Distributed Application Framework for .NET
- masstransit
- rabbitmq
- c-sharp
- azure-service-bus
Arkime is an open source, large scale, full packet capturing, indexing, and database system.
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability