Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
47,133 repositories indexed
1–24 of 29 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
Parallel computing with task scheduling
- dask
- python
- pydata
- numpy
Sarama is a Go library for Apache Kafka.
- kafka
- kafka-client
- go
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
Dashboard and code-driven configuration for Laravel queues.
AI Infra / AI Orchestration / AI Control Plane
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
Self-hosted platform to unify wearable health data through one AI-ready API.
- wearables
- apple-health
- apple-watch
- fitness
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.
- dataprep
- data-science
- datapreparation
- dataconnector
Ruby and Rails efficient Kafka processing framework
- karafka-framework
- kafka
- kafka-client
- kafka-clients
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
visual data prep powered by python
- data
- data-pipelines
- etl
- structured-data
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
- golang
- event-sourcing
- stream-processing
- brokers
🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications
- event-driven
- generics
- go
- golang
Karapace - Your Apache Kafka® essentials in one tool
- kafka
- schema-registry
- rest-proxy
Distribute work across machines using the lparallel API.
Declare AMQP entities like queues, producers, and consumers in a declarative way. Can be used to work with RabbitMQ.
- amqp
- rabbitmq
- rabbit
- golang
Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.
- ruby
- rake
- pwrake
- parallel
Parallelize _anything_ //
- ocaml-library
- parallel-computing
- stream-processing
- multi-processing