What tools and projects do you use for your local LLM chats and agents? I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
What tools and projects do you use for your local LLM chats and agents? I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
I’m on an rx7600 too. I have done loads of testing and benchmarking and the best model is orinth1.5 35b a3b. I get around 40tok/s generation from it. I run it with llama.cpp with the rocm driver, and use the iq4_xs unsloth quant. It’s based on qwen3.6 35b a3b, but trained on complex thinking and took calling. It’s miles ahead of anything else I’ve tried.
How big is your system prompt? My 6.5k (which is mostly 5.5k of tool defs) leads to an error message.