He’s getting nervous because modern tech is making the LLMs easier to run at home. MoEs with intelligent routing can be amazing on well equipped gaming machines. On my AMD 7900XT with 64GB RAM (all purchased before the RAMpocalypse) using Strata I can run Qwen 3.8 Flash Next a 90GB 125b parameter model with 256k context size and get between 30 and 40 tokens per second. Strata and probably similarly FreeToken have been absolute game changers.
He’s getting nervous because modern tech is making the LLMs easier to run at home. MoEs with intelligent routing can be amazing on well equipped gaming machines. On my AMD 7900XT with 64GB RAM (all purchased before the RAMpocalypse) using Strata I can run Qwen 3.8 Flash Next a 90GB 125b parameter model with 256k context size and get between 30 and 40 tokens per second. Strata and probably similarly FreeToken have been absolute game changers.
i have a similar build except with a 9070xt, running 64k context window
how well is 256k window working for you?
also which model did you pick? i picked swift 1.5 iq3 xxs
The 256k works great. I think I picked the 2 iq3s.
Edit: I realize I typoed my last post. I have a 7900XT not a 7800XT. That 4GB more of vram might matter.