qwen3.8-ridge-gguf runs great in ollama on the 9070xt in 4bit kv cache quant at 35t/s.
You can also run deepseek 4.1 flash in strata if you have time. both models are very capable.
the 27b model fits completely on the gpu so ram and cpu makes no difference.
For the deepseek model you need a fast ssd because no matter how big your ram is that model won’t fit there since it is almost 300gb.
I’ve got a ryzen9 5900x and 40gb ram. I don’t know how much of an impact the cpu actually has because i think most of it is evaluated on the gpu but could be that the router that selects the experts is executed on the cpu.
in any case the main impact will probably be vram size, pci-e throughput and ssd read speed (and maybe cpu speed).
What’s a nice free locally hosted PC AI model to set up and use? One that plays well with AMD gpu’s?
Any you like. I’ve had very nice results with llama.cpp w/ vulkan backend.
qwen3.8-ridge-gguf runs great in ollama on the 9070xt in 4bit kv cache quant at 35t/s. You can also run deepseek 4.1 flash in strata if you have time. both models are very capable.
Can you speak in common market specs? How much RAM, CPU, GPU, etc.
the 27b model fits completely on the gpu so ram and cpu makes no difference.
For the deepseek model you need a fast ssd because no matter how big your ram is that model won’t fit there since it is almost 300gb.
I’ve got a ryzen9 5900x and 40gb ram. I don’t know how much of an impact the cpu actually has because i think most of it is evaluated on the gpu but could be that the router that selects the experts is executed on the cpu.
in any case the main impact will probably be vram size, pci-e throughput and ssd read speed (and maybe cpu speed).