AI inference, UIC
Auto Tuner for Local LLM's- is your local llm really running at its best?
Well, I noticed a problem: there is no fixed setting for GPUs and models to achieve maximum throughput, as it can be optimized depending on the specific GPU model and engine. So, I tried to build this auto-tuner. Honestly, this is just the start of the main idea I have, and I will keep posting updates :)
Please try it on your machine and let me know how it works for you.
Thanks!
Github: github.com/Aagam-Bothara/rigtune
LLMInference






Comments
0 comments