Run frontier AI locally.
Repository Index
77,353 visible repositories in the index
1–24 of 29 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Parallel computing with task scheduling
- dask
- python
- pydata
- numpy
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
A lightweight data processing framework built on DuckDB and 3FS.
- data-processing
- duckdb
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
- data-analysis
- root-cern
- root
- c-plus-plus
💎多语言实现，高性能生成唯一数字ID。 💎优化的雪花算法（SnowFlake）——雪花漂移算法，在缩短ID长度的同时，具备极高瞬时并发处理能力（50W/0.1s）。 💎原生支持 C#/Java/Go/Rust/C/JavaScript/TypeScript/Python/Pascal 多语言，提供其它适用于其它语言的多线程安全调用动态库（FFI）。💎支持容器环境自动扩容（自动注册 WorkerId ），单机或分布式唯一IdGenerator。💎顶尖优化，超强效能。
- snowflake
- idgenerator
- dotnet
- rust
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
IPython Parallel: Interactive Parallel Computing in Python
Run MapReduce jobs on Hadoop or Amazon Web Services
Distributed self-replicating programs in Python
- python
- devops
- infrastructure
- ansible
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
- spark
- dask
- data-practitioners
- machine-learning
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
A distributed task scheduler for Dask
- pydata
- dask
- distributed-computing
- python
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions
- adaptive-learning
- machine-learning
- python
- live-plots