What Is an Inference Engine, Anyway? — Charles Frye, Modal

Modal's Charles Frye explains what an inference engine actually does, from batching and KV cache to scheduling, in a talk every builder paying per token should watch.