Price drop caused in majority by open weight models. No wonder Altman wants regulations against open models and to take everybody’s home computer’s RAM.
And nVidia is now selling hardware for local models specifically. For cheap.
There is a clear winner in the AI race and its hardware makers. To me, its almost like Jensen Huang calculated this outcome quitea while ago, and fed into the bubble with the loans to the software companies.
You do, however Nvidia got greedy and started to circular invest in the billions if not trillions into those buying the shovels. Now, when the bubble bursts, Nvidia will be hit massively as well.
Eh somewhat. I think they were careful enough to just have the loans not actually be in their name if you follow it as deeply as Ed Zitron has. But Nvidia is screwed from a different angle as they’ve set up expectations of selling more GPUs to data center builders each quarter in order to justify their share price.
That’s still much better than holding the actual bag though.
Nvidia did a massive share buyback with a portion of the ridiculous profits they made last year. That serves to temper expectations, as they’re signaling that they’re not going to be investing all their profit in yet more production (which is difficult anyway given they don’t have their own fabs and TSMC seems to be running at full capacity).
Likely they’ll be doing another one next year and shareholders will be happy.
If they can refinance their own customers to buy their own products without getting caught in the middle that would be even more scandalous than the circular financing itself but then, the financial system has been completely broken for a good while.
They’re basically operating as a kinda/sorta backstop on the loans by saying they’ll rent them back if everything falls through and be a guaranteed customer…but I think they did so in snaky legalese that lets them ultimately get away without actually renting them. I think it’s through like subsidiaries and shit. Oracle is one of the actual bag holders and I think there are several others. They set up shell companies for each specific data center project and shit. It’s quite the fucking racket and should pretty obviously be illegal from multiple angles.
Which hardware is for running local models that is cheap? I was not looking at GPU market for around two years, because I thought everything got expensive through the roof.
Nothing consumer grade - I’m talking about nVidia DGX.
$6k for the cheapest might seem expensive to a lot of people, but for an engineer with stupid amounts of data and a 1.2kW power draw, it’s insanely affordable from corporate perspectives.
I’d have to look way more into it, but from what we were looking at, DGX spark is made with the intent of scaling from local development to datacenter execution. From memory the Mac Studio was an extremely beefy workstation.
Both serve the purpose better than whatever my laptop’s RTX ADA 2000 can produce lol
The market for AI is never going to be sustainable based on the average home user. It’s about businesses. And when the big AI companies are forced to raise prices to keep investors happy, they’re going to start looking into hosting their own. I don’t know how much we pay, but the company I work for has a couple dozen software engineers with Claude, and a bunch more employees who also get AI they use in their work. If AI prices suddenly quadrupled, I bet the CEO and CTO would be looking very seriously at buying a server to run models locally. And if that happens across the whole economy, it’s going to hurt the big companies hard.
Even worse if they’re getting hit from two angles. If governments and local communities keep making it hard for them to build datacentres, their rising costs are going to make them have an even harder time without raising prices even further, driving more smaller companies into self-hosting. If the pushback on datacentres is strong enough, they may not be able to keep up the practices causing hardware prices to be so high, making it more affordable to self-host, too.
Meanwhile, Nvidea is chuffed either way. They don’t care if OpenAI or CRUDs r Us buys their chips. They get paid either way.
Seriously though, home AI is never going to compete. Few people have PCs in their home any longer and only a tiny subset of those users will fiddle with local models. I’d like to, but I don’t have the compute. :(
And I don’t see that changing. :( And in no case are loads of people going to get into local AI. Lotta nerds around here, but Joe on the street barely knows what AI is, and he certainly doesn’t have a PC any longer.
At best a macbook air/neo or equivalent on windows good enough to browse the average web and the gamer joe will have a gpu with ~8GB vram for most games (aka 3070 or something like that)
Honestly, if these shit heads hadn’t inflated RAM prices with their speculative datacenter building, I could probably afford to run a quantized GLM-5.3 on local hardware right now
Price drop caused in majority by open weight models. No wonder Altman wants regulations against open models and to take everybody’s home computer’s RAM.
People running local models would RUIN their profit margins.
And nVidia is now selling hardware for local models specifically. For cheap.
There is a clear winner in the AI race and its hardware makers. To me, its almost like Jensen Huang calculated this outcome quitea while ago, and fed into the bubble with the loans to the software companies.
During a gold rush, you want to be the guy selling shovels.
You do, however Nvidia got greedy and started to circular invest in the billions if not trillions into those buying the shovels. Now, when the bubble bursts, Nvidia will be hit massively as well.
Eh somewhat. I think they were careful enough to just have the loans not actually be in their name if you follow it as deeply as Ed Zitron has. But Nvidia is screwed from a different angle as they’ve set up expectations of selling more GPUs to data center builders each quarter in order to justify their share price.
That’s still much better than holding the actual bag though.
Nvidia did a massive share buyback with a portion of the ridiculous profits they made last year. That serves to temper expectations, as they’re signaling that they’re not going to be investing all their profit in yet more production (which is difficult anyway given they don’t have their own fabs and TSMC seems to be running at full capacity).
Likely they’ll be doing another one next year and shareholders will be happy.
If they can refinance their own customers to buy their own products without getting caught in the middle that would be even more scandalous than the circular financing itself but then, the financial system has been completely broken for a good while.
They’re basically operating as a kinda/sorta backstop on the loans by saying they’ll rent them back if everything falls through and be a guaranteed customer…but I think they did so in snaky legalese that lets them ultimately get away without actually renting them. I think it’s through like subsidiaries and shit. Oracle is one of the actual bag holders and I think there are several others. They set up shell companies for each specific data center project and shit. It’s quite the fucking racket and should pretty obviously be illegal from multiple angles.
That’s the best way to describe this.
Which hardware is for running local models that is cheap? I was not looking at GPU market for around two years, because I thought everything got expensive through the roof.
Nothing consumer grade - I’m talking about nVidia DGX.
$6k for the cheapest might seem expensive to a lot of people, but for an engineer with stupid amounts of data and a 1.2kW power draw, it’s insanely affordable from corporate perspectives.
Depending on what you’re looking at, a Mac Studio might be the most bang for your buck. They have much higher memory bandwidth than the DGX Spark
I’d have to look way more into it, but from what we were looking at, DGX spark is made with the intent of scaling from local development to datacenter execution. From memory the Mac Studio was an extremely beefy workstation.
Both serve the purpose better than whatever my laptop’s RTX ADA 2000 can produce lol
They don’t have profit margins though
This is the logical endgame IMO so I get why the losers are fighting it.
They can only afford to buy the worlds compute for so long though.
The world at large is nothing like us here on lemmy. Few people even have a PC, let alone interest in local models.
The market for AI is never going to be sustainable based on the average home user. It’s about businesses. And when the big AI companies are forced to raise prices to keep investors happy, they’re going to start looking into hosting their own. I don’t know how much we pay, but the company I work for has a couple dozen software engineers with Claude, and a bunch more employees who also get AI they use in their work. If AI prices suddenly quadrupled, I bet the CEO and CTO would be looking very seriously at buying a server to run models locally. And if that happens across the whole economy, it’s going to hurt the big companies hard.
Even worse if they’re getting hit from two angles. If governments and local communities keep making it hard for them to build datacentres, their rising costs are going to make them have an even harder time without raising prices even further, driving more smaller companies into self-hosting. If the pushback on datacentres is strong enough, they may not be able to keep up the practices causing hardware prices to be so high, making it more affordable to self-host, too.
Meanwhile, Nvidea is chuffed either way. They don’t care if OpenAI or CRUDs r Us buys their chips. They get paid either way.
What profit margins? :)
Seriously though, home AI is never going to compete. Few people have PCs in their home any longer and only a tiny subset of those users will fiddle with local models. I’d like to, but I don’t have the compute. :(
If hardware prices weren’t insane, we would have prosumer level turnkey home AI already.
Now it’s just for hobbyists with too much money.
And I don’t see that changing. :( And in no case are loads of people going to get into local AI. Lotta nerds around here, but Joe on the street barely knows what AI is, and he certainly doesn’t have a PC any longer.
It’d be Apple packaging local AI capabilities to something like an Apple TV Pro or any other manufacturer with enough integration.
Google might do it too, but less private - ad revenue you know.
But with these prices a decent local AI box would be closer to 1k, ain’t nobody buying that.
At best a macbook air/neo or equivalent on windows good enough to browse the average web and the gamer joe will have a gpu with ~8GB vram for most games (aka 3070 or something like that)
Honestly, if these shit heads hadn’t inflated RAM prices with their speculative datacenter building, I could probably afford to run a quantized GLM-5.3 on local hardware right now
That’s a lot of VRAM! What would be your dream build in that scenario?