Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
52,643 visible repositories in the index
1–24 of 28 · Most stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Parallel computing with task scheduling
- dask
- python
- pydata
- numpy
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
A unified ecosystem and workspace for large-scale federated learning, bridging the gap between centralized model evolution and decentralized edge intelligence.
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions
- adaptive-learning
- machine-learning
- python
- live-plots
Cloud-native genomic dataframes and batch computing
- genetics
- vcf
- genomics
- gwas
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
The repository host gives no description.
- hacktoberfest
Haskell on Apache Spark.
- apache-spark
- spark
- haskell
- analytics
Phoebus is a distributed framework for large scale graph processing written in Erlang.
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
- apache-iceberg
- apache-spark
- data-engineering
- delta-lake
Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.
Standard API for Distributed Data Structures in R
Distribute work across machines using the lparallel API.
Serverless ML Framework