Back when I had a Wordpress blog before I switched to a static website, when I got these spam comments, I’d just edit the positive sounding ones to remove the spam link and post them. It made me like people were reading my blog and caring.

It means they release the model so you can run it yourself with enough hardware. Kind of like open source. Governments and large companies should be self-hosting this kind of stuff because of security, availability, and IP-leak risks anyway.

With LLMs, there is some risk of hidden malicious actions embedded in the weights that you can see until you ask it a magic question and it says “activate skynet.”

Whether or not the government is doing this because of fears of hidden actions in the weights or just because OpenAI/Anthropic CEOs told the government to give them more money is unclear.

Yes, I run Ollama on a NVIDIA 1080 Ti at home. I run granite4:3b for my Home Assistant assistant for my voice assistant, paperless-ngx, OpenWebUI for chat, and a few other things. For speech to text, I run a custom-built hybrid speech-to-phrase + parakeetTTS (requests that match speech-to-phrase take priority over parakeet.) I run GOModel to route requests to my local Ollama instance, or DeepInfra (cloud-based for larger models), or Tinfoil (cloud-based for encrypted LLM chats.)

As much as I love to use local-everything, only my smarthome+basic chats go local. Trying to fit both the STT and LLM models on the same GPU at the same time while also using this computer as a media console is challenging. I’ve recently started looking at the Hailo m.2 accelerator board to see if I can shift the STT to it.

That doesn’t check out when considering model size. Unless that’s what you’re trying to say that self-hosted models force people to use smaller models. Just from an operations perspective a data center had economies of scale and the ability to batch up multiple independent user’s requests which should reduce the per token kWh along with the fact that your utilization will be higher vs somebody’s GPU at home would be idling.

That said I do believe there are negative environmental impacts from AI like always using giant frontier models for everything instead of smaller fine tuned models.

EDIT: I see a bunch of replies on the other comment, which all raise valid points, but that’s definitely not the way most readers are going to interpret the original post and moves the goal posts.