Jlama is a modern LLM inference engine for Java
Repository Index
108,094 visible repositories in the index
145–168 of 244 · Most stars
E2M converts various file types (doc, docx, epub, html, htm, url, pdf, ppt, pptx, mp3, m4a) into Markdown. It’s easy to install, with dedicated parsers and converters, supporting custom configs. E2M offers an all-in-one, flexible, and open-source solution.
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
- k8s
- llm
- openai-api
- autoscaler
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
A single interface to use and evaluate different agent frameworks
Distributed training & inference for multimodal perception: shared embeddings across text, images, video, audio, code & graphs from 64 to 1024 dimensions 🖼️
- huggingface-transformers
- language-vision
- multimodal
- pytorch
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
PDEBench: An Extensive Benchmark for Scientific Machine Learning
The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 200+ tok/s single-request decode with support of FP8 weight ( Join Discord :https://discord.gg/VFqVVySdMS )
Fine-tune the Whisper speech recognition model to support training without timestamp data, training with timestamp data, and training without speech data. Accelerate inference and support Web deployment, Windows desktop deployment, and Android deployment
- asr
- ctranslate2
- huggingface
- whisper
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM
- ggml
- kv-cache
- llama-cpp
- llm-inference
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
Local inference engine optimized for NVIDIA Blackwell
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
- huggingface
- inference
- llm
- mechanistic-interpretability
Read it. See it. Get it. Built at GDG AI Hack Milan 2026 for "Learn Different" track.
Object Detection Metrics. 14 object detection metrics: mean Average Precision (mAP), Average Recall (AR), Spatio-Temporal Tube Average Precision (STT-AP). This project supports different bounding box formats as in COCO, PASCAL, Imagenet, etc.
- metrics
- object-detection
- bounding-boxes
- pascal-voc
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
- prompt-engineering
- llm
- llm-inference
- prompt
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
- anthropic-api
- cuda
- gemma
- gguf
EdgeRock: Strong Reasoning at Constant Memory. Multimodal models with O(N) generation and O(1) persistent history. Inference and evaluation code.
Compare the performance of different LLM that can be deployed locally on consumer hardware. Run yourself with Colab WebUI.