Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
47,133 repositories indexed
1–24 of 150 · Most stars
Apache Kafka - A distributed event streaming platform
A realtime distributed messaging platform
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
High-Performance server for NATS.io, the cloud and edge native messaging system.
- go
- golang
- messaging
- message-bus
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
Apache Pulsar - distributed pub-sub messaging system
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.
Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.
- kafka
- diskless
- automq
- serverless
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Distributed Application Framework for .NET
- masstransit
- rabbitmq
- c-sharp
- azure-service-bus
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern
Python Stream Processing
- kafka-streams
- kafka
- python
- asyncio
Apache Storm
- storm
- apache
- distributed
- streaming
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data