SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
Repository Index
60,621 visible repositories in the index
1–24 of 32 · Most stars
All-in-one IoT Platform - Device management, data collection, processing and visualization.
- visualization
- iot
- iot-platform
- mqtt
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles
🥊 Tiny 200 byte functional event emitter / pubsub.
- mitt
- event-handlers
- pubsub
- tiny
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.
- rust
- database
- serverless
- bigdata
Enterprise job scheduling middleware with distributed computing ability.
- scheduler
- workflow
- distributed
- mapreduce
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
- camel
- integration
- java
- microservices
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
- data
- python
- data-engineering
- data-lake
Translatedcube studio is an open-source, cloud-native, one-stop machine learning / deep learning / large model AI platform. It covers the full mlops algorithm pipeline, a computing power rental platform, online notebook development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference services with VGPU virtualization, edge computing, a labeling platform with automated labeling, SFT fine-tuning / reward model / reinforcement learning training for large models such as deepseek, multi-machine large model inference with vllm/ollama/mindie, a private knowledge base, and an AI model marketplace. It supports domestic Chinese CPUs/GPUs/NPUs and the Ascend ecosystem, supports RDMA, and supports distributed frameworks such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano.
Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
- apache-kafka
- dataops
- react
- typescript
A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!
- mqtt
- m2m
- scalable
- broker
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
- distributed-computing
- map-reduce
- golang
- distributed-systems
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
A modern high-performance open source message queuing system
Eclipse iceoryx™ - true zero-copy inter-process-communication
- iceoryx
- eclipse
- ipc
- inter-process-communication
EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
- esb
- event-driven
- event-mesh
- event-governance
HiveMQ CE is a Java-based open source MQTT broker that fully supports MQTT 3.x and MQTT 5. It is the foundation of the HiveMQ Enterprise Connectivity and Messaging Platform
The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.
Java client for NATS
An open source, a high-performance hybrid integration platform that allows developers quick integration with any application, data, or system.
- integration
- middleware
- wso2
- esb
🚃 A library for handling mediator patterns and simplified CQRS patterns within an event-driven architecture, inspired by csharp MediatR library.
- cqrs
- golang
- mediator
- mediator-pattern
An opinionated Elixir wrapper around brod, the Erlang Kafka client, that supports encrypted connections to Heroku Kafka out of the box.
- elixir
- kafka