I keep looking at mac mini’s and wondering if I should run models locally. But they’re always so expensive I don’t really need to run AI models at all it’s just an idle curiosity.
I think I’m going to ignore AI until it either gets cheaper or builds us a warp drive.
Them being a great deal is gone
Only 3 years behind. Getting better, Apple
Thyey aren’t 3 years behind on custom hardware to run local AI models.
E.g nvidia’s DGX Spark came out in late 2025.
Their MacBook pros have been running models for years just fine, but this is more purpose built.
Edit: Also the Studio Ultra’s have been running AI models for awhile as well with their higher memory bandwidth and total memory allowed, but they’re very expensive.
I mean it’s apple and their AI is Siri. They’re lucky if they can even break even.
Call me when something lets me train and run my own 1.4T model locally, in about 120 years.
Those 120 years went by in a flash.
He is the world’s dumbest man afterall
I feel there is a catch here. I forgot mixture of experts was a thing -_-. They only activate like 32B at a time.
But this can be incredibly good, lets see!
Btw “and train” people missed the very very key word.
EDIT: @_@ training qwen 3.8 on under 6 GB. Yes, this seems VERY acceptable! I can see the endless potential here.
Nah, it seems it can finetune a 128 GB model with 6 GB RAM, I don’t see evidence that it can finetune something of 1.4T or more. I was excited for nothing.
Maybe just another two years, but the trajectory is promising and hardware will improve, as well. Strata is good for inference, at least.
I give it a week before someone reverse engineers it and uploads a foss option.
Idk if I hate this kind of corpo project if they’re paying people for their work over more FOSS-slop
There are already a lot of FOSS options that run on Apple Silicon.
Oh sure I’d imagine so. What I meant was whatever magic sauce that makes this better will be assimilated.
Sooooo it’s Ollama but not FOSS?
https://sleepingrobots.com/dreams/stop-using-ollama/
Turns out that ollama is kinda garbage, and their desktop app wasn’t FOSS for a while too.
Also, modelfiles suck
Ooh I didn’t know that! Thank you! I like FOSS, so I’ll try out the stuff referenced in your article!
Harbor is currently in a private preview under Apple’s Claris brand…
Now there’s a name I haven’t heard in a long time.
FileMaker is still going strong
sus
it is called install asahi or buy a proper computer
Dynamic pooling could make this more accessible to lower power macs.
Are there other handlers that can bundle computers into a local Ai instance in a similar manner?
I guess vLLM supports clusters so you could just use that? The setup is more difficult than just plugging in a cable though.
Interesting. I kind of want to buy another Mac to dee if it works either way my M4 Max MBP.
