I don’t know of this counts, but I have a side gig making websites and managing servers. A lot of generic WordPress or HTML sites for small local businesses.
In the last month I’ve seen a huge uptick of people trying to do stuff on the fucking servers themselves.
With the recent PHP CVEs, I patched every server I manage as soon as I got the alert. But that didn’t stop one guy who’s techy enough to have heard about it and know its an issue, but not techy enough to manage his own systems, use Claude to ssh into the server and ask it to run a security scan on all the packages.
Claude suggested updating absolutely everything, deleting packages that were in use, and installing a bunch of unnecessary stuff.
Needless to say he took his site offline, and I spent like 2 hours trying to get it back in shape before canning it and restoring from a backup. It cost him a nice chunk of change since I charge extra for emergency services.
At the end of all that he still said “here’s a list of issues Claude says we should fix”.
I had another guy who wanted to start pirating shows and movies, and was “trying to set up kodi” but for some reason the robot told him to install the whole *arr stack. So naturally he used his business server because all its doing is hosting his website and he’s already paying for it. He ended up installing every arr software there is, he was almost ready to pirate books and music too. Well his server wasn’t prepared for that since all it was doing was hosting a small local websites and only had 4gb ram. All the resources got drained and everything kept crashing, the ai told him to stop nginx since it was using too many resources for him to run his arr stack.
These are 2 pretty intelligent guys, not dumb by any standards. Whose confidence was propped up so high that they trusted fancy autocorrect who told then exactly how to willy nilly take down their own systems.
This was just this week, there was another incident last week and the week before that.
ai told him to stop nginx since it was using too many resources
That is pure, undistilled awesome. We’re approaching “delete system32” kinda awesome.
I’m not that techy, but even I understand you don’t want to delete the backbone of your system (referring to both nginx and system 32). Jfc.
Same thing happened to me, I work on industrial kitchen equipment and have had two calls where a BRAND NEW machine had a big failure and was down. Asked the workers what they did to try and get a heading for what the problem could be. Finally someone said " well I asked chat GPT and it told me to do this…" had to explain that the system is laced together, change 1 thing and you have to change other stuff too otherwise you have a problem…
This was an ICE CREAM MACHINE and they still try to use ai to “hack” it.
I’ve encountered the same thing in a completely unrelated field.
I’m a tax consultant, a CPA, for nearly 20 years now.
About once a week I get a call from either a client or a prospective client and straight away get the vibe they’ve been talking to a bot. It’s tough to explain but I shouldn’t have to waste my time responding to a bot’s assertions before getting to the actual problem at hand.
We try to be pretty cool with charging for our time, because starting an enquiry with an explanation of a 5 minute billing structure sets the wrong tone, and I prefer to just name a price for solving an actual problem.
However, with these calls instead of a client telling me a problem and me explaining the solution, the client proposes several solutions they need help with and I have to undo all of that before I can understand the problem. The tail kinda wags the dog.
For example, one guy emailed me explaining that he wanted a new business structure. The email was slop but ok - we charge heaps for this type of job so I can put up with it. However, after an hour long discussion I had convinced him that he was much better off without the structure. Only an hour of my time, (but still a waste) and a misadventure for him, that could have cost him $5k.
Another guy was on the phone, rattling off a series of questions a bot had given him to evaluate a consultant. One of them was “what can you do to help me grow my business”. He was 20 years old and had 7 clients for whom he mowed their lawns.
I’m still trying to figure out the best policy for these situations. I think for the time being, my plan is to interrupt as soon as I get that bot-vibe, and say that it sounds a bit like they’ve been researching this with a bot, and that I need to start from the start just to make sure nothing is overlooked. Perhaps I’ll add as an aside that correcting the problems caused by advice from bots is very time consuming and costly.
That last paragraph sounds like an ideal approach. Do that thing.
Well, as I said it’s the best I have for the moment. It’s kind of easier said than done though.
After the client has spent n hours with a bot that does everything it can to endear itself to the user, I’m sort of starting in second place with less than zero “equity”.
“Disregard all the time you’ve invested and everything you think you know because it sounded plausible, and listen to me who will charge you lots of money”
We have a similar situation, though it has nothing to do with AI (but it’s gotten worse since some clients do ask AI bots “I would like our LSP to do this” and since the bot will always tell them what they want to hear, they take it as God’s word, while most of it would be unsustainable at best).
So we decided that we will immediately disregard whatever idea they may have, ask what the end-game is for them, and provide our solution or flat-out refuse. If the client chooses to leave because of this, we’re cool with that. In 3 occasions they left for another provider (you know, those that say “yes” to everything only to fond out that it can’t be done) only to come back to us in a couple of moths and find out that “the rates have slightly gone up” from what you had in our previous agreement.
Every single one of them has signed the new agreement without any fuss.
Damn, what a read ! Hope you got compensated fairly to fix their mistakes
I made very fair money from the first guy.
My prices are high but I usually give discounts, I can’t remember the last time I actually charged full price for anything. But this guy was a dick the whole way through, “why didn’t this work? That should’ve done this! The server must be so outdated you have to do a better job”. He tried to be smarter than me by compensating for lack of a decade and a half of experience with a $20 subscription to a chatbot.
The 2nd guy was completely apologetic, basically said “hey I fucked up, charge me whatever you need to but please don’t rake me over the coals”, and was just trying to do something unrelated on his own and figure it out with the resources he had, even if he went about it in a silly way. I charged him an hour to clean up the server which was just removing the stuff he installed, and then showed him how to set up stremio on his devices and kodi on his TV for sports stuff.
The first guy probably won’t renew his contract, which is fine. The second was easy about it all and I would’ve helped him set up that stuff for free anyway if he just asked, as I do for anyone who asks.
Good news: We’re rapidly making FOSS the most secure software ever.
Bad news: We’re doing it in the most painful way possible.
It‘s probably still much less secure than pre-AI FOSS simply because it didn‘t have to deal with that kind of attacks.
The way corporate software will survive is through absurds laws they lobby for and data centers, though. Because if they kept playing by the old rules of consumers owning devices they would lose to FOSS really quickly at this point. With so many vibe coded clones and all. But when the hardware becomes unobtainable for mere citizens most of us will get pushed further and further into the cloud and complete surveillance. I bet that‘s what they‘re betting on.
No, the vulnerabilities were always there, AI is just pointing them out publicly.
I used to work as a professional hacker. Let me tell you something that the news never, ever talks about: If a company doesn’t have a bug bounty program, we kept those vulnerabilities to ourselves.
A government has vast resources to do just that: Keep the vulnerabilities to themselves until they feel like using them. They will not disclose them. They will keep them in their back pocket until the day comes when they need them for an op.
The only time they disclose vulnerabilities is when they think they’re too easy to find and there’s a public interest benefit to getting the problem fixed (e.g. the government is using those products).
What this means is that AI really is making FOSS vastly more secure than it ever was. This is probably pissing off government red teams to no end.
What we’re not hearing about as much is AI finding vulnerabilities in closed source products. Not because it’s not finding them, but because the disclosures are kept private (because companies treat it as bad PR and they’re assholes about it).
You can bet your ass that AIs run by state actors are building up massive war chests of vulnerabilities in closed source products right now. We’ll only hear about them after these products get exploited and maybe not even then because so, so many companies have no idea WTF they’re doing in regards to infosec. Think about it: How many CEOs are right now demanding their teams reduce the amount of FOSS they use because of the recent news? These people are idiots and apparently never heard that old joke about a new employee making a $100,000 mistake… “We just spent $100,000 training them!”
B-b-but Fediverse users assured me this could never happen when AI started decompiling games in an industrial speed because „You don‘t have corporate software on your computer to decompile so it will be fine“ as if that was the point. AI is a brute force machine if nothing else.
No software is safe. Not your supermarkets. Not your bank. Not your hospital. We may soon stop thinking about software as an engineering thing and more of a gardening activity because it will just keep growing when unattended.
I don’t believe for a second a single person ever said that.
Maybe I said it… maybe you LIKED it when I said too.
