Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• ray
• distributed
• parallel
• machine-learning

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

Parallel computing with task scheduling

• dask
• python
• pydata
• numpy

Build, Manage and Deploy AI/ML Systems

• machine-learning
• model-management
• ai
• ml

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

Data-Centric Pipelines and Data Versioning

• go
• pachyderm
• docker
• analytics

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

• machine-learning
• python
• data-engineering
• distributed-computing

Apache Ignite

• distributed-sql-database
• iot
• osgi
• network-client

A cluster computing framework for processing large-scale geospatial data

• cluster-computing
• spatial-sql
• geospatial
• spatial-analysis

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering

A distributed task scheduler for Dask

• pydata
• dask
• distributed-computing
• python

Cloud-native genomic dataframes and batch computing

• genetics
• vcf
• genomics
• gwas

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

• hpc
• workflow
• workflow-automation
• workflow-management

Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.

• apache-iceberg
• apache-spark
• data-engineering
• delta-lake

Standard API for Distributed Data Structures in R

Distributed batch data processing framework

• cassandra
• rabbitmq
• go
• batch-processing

Tuple MapReduce for Hadoop: Hadoop API made easy