Repo

LLM-Inference-Engine

@joezhuo2by Joe Zhuo · Independent

0

A single-GPU LLM inference engine in C++20 and CUDA. It serves TinyLlama-1.1B-Chat-v1.0 in BF16 with FP32 accumulation, matches HuggingFace output token for token, and serves many concurrent requests with a paged KV cache, a paged decode attention kernel, continuous batching, custom fused kernels, sampling on the GPU, and CUDA Graphs.

Built withTypeScriptinferenceinference-enginellmmachine-learningsystems-programming

See what other members are building and what each project is looking for.