Top #inference Tools & Software

vllm

github

A high-throughput and memory-efficient inference and serving engine for LLMs

ray

github

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

DeepSpeed

github

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

ColossalAI

github

Making large AI models cheaper, faster and more accessible