Ongoing research training transformer models at scale
- large-language-models
- model-para
- transformers
140,194 visible repositories in the index
1–9 of 9 · Most stars
Ongoing research training transformer models at scale
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).
An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
Massively Parallel Deep Reinforcement Learning. 🔥
Ongoing research training transformer language models at scale, including: BERT & GPT-2
The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.
Ongoing research training transformer language models at scale, including: BERT & GPT-2
Training library for Megatron-based models with bidirectional Hugging Face conversion capability