Repository Index

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

A unified ecosystem and workspace for large-scale federated learning, bridging the gap between centralized model evolution and decentralized edge intelligence.

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

• distributed-computing
• map-reduce
• golang
• distributed-systems

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

• hpc
• workflow
• workflow-automation
• workflow-management

Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.

• apache-iceberg
• apache-spark
• data-engineering
• delta-lake

Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.

Distributed batch data processing framework

• cassandra
• rabbitmq
• go
• batch-processing

Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.

• ruby
• rake
• pwrake
• parallel

Common Lisp distributed computation framework implemented using Swank Client

Web application for distributed compute analysis of Archive-It web archive collections.