A high-throughput and memory-efficient inference and serving engine for LLMs
Repository Index
47,133 repositories indexed
1–18 of 18 · Most stars
Your GeForce runs AI at half speed. This fixes it for 20/30/40 series cards. Up to 2× FP16 / BF16 / TF32 tensor throughput
Garnet is a remote cache-store from Microsoft Research that offers strong performance (throughput and latency), scalability, storage, recovery, cluster sharding, key migration, and replication features. Garnet can work with existing Redis clients.
- cache
- cache-storage
- cluster
- concurrent
A new kind of Progress Bar, with real-time throughput, ETA, and very cool animations!
- progress
- bar
- progress-bar
- progressbar
Lightweight, fast and reliable key/value storage engine based on Bitcask.
- database
- kv-store
- golang
- data-structures
A constant throughput, correct latency recording variant of wrk
Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
- java
- jvm
- eclipse-openj9
- compiler
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
C library for high-throughput sequencing data formats
- htslib
- bioinformatics
- sam
- bam
Microservices library - scalecube-services is a high throughput, low latency reactive microservices library built to scale. It features: API-Gateways, service-discovery, service-load-balancing, the architecture supports plug-and-play service communication modules and features. built to provide performance and low-latency real-time stream-processing
- microservices
- reactive-streams
- distributed-systems
- service-mesh
Reckless logging. Low-latency, high-throughput, asynchronous logging library for C++.
- logging
- performance
- asynchronous
- logging-library
One database for user-user (follows), user-item (likes), and item-item (related items) interactions — precomputed at write time, served as simple lookups
- database
- low-latency
- real-time
- graphdb
High throughput data ingestion logger to Fluentd, AWS S3 and Treasure Data
- fluentd
- logger
- treasuredata
- aws-s3
MailMergeLib is a mail message client library which provides comfortable mail merge capabilities for text, inline images and attachments, as well as good throughput and fault tolerance for sending mail messages.
Window-Based Hybrid CPU/GPU Stream Processing Engine
- stream-processing
- hybrid
- multicore
- gpu
Simply ACID* key-value database. At the medium or even low latency it tries to provide greater throughput without losing the ACID properties of the database. The database provides the ability to create record headers at own discretion and use them as transactions. The maximum size of stored data is limited by the size of the computer's RAM.
🔬Python package for high-throughput single-cell imaging analysis.
- microscopy-image-analysis
- single-cell-analysis
Header-only C++11 library using OpenCV for high-throughput image-based plant phenotyping
- plant-phenotyping
- opencv
- charuco
- color-correction