Top #inference Tools & Software

vllm

github

A high-throughput and memory-efficient inference and serving engine for LLMs

zod

github

TypeScript-first schema validation with static type inference

ray

github

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

DeepSpeed

github

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

ColossalAI

github

Making large AI models cheaper, faster and more accessible

mediapipe

github

Cross-platform, customizable ML solutions for live and streaming media.

sglang

github

SGLang is a high-performance serving framework for large language models and multimodal models.

llm-action

github

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

shimmy

github

⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.