The mess is metaphorically a result of a kind of hasty, brainless merger of the mindsets of Internet and server companies with those of companies that construct field buses and factory floors. These two ways of connecting things are not very compatible. And the factory floor corps don’t even understand why security is necessary.

To sum it up, the problem is systemic. It is not just that Danish connected thermostat manufacturer.

GenAI and security seem to be things that are a bit on collision course.

Kara had experienced increasing burnout, much of which she attributes to being instructed to use AI despite her concerns over legality and creative use. She said her team was encouraged to use AI-enabled production tools like Runway, which generates AI videos and images, and ElevenLabs, an AI voice generator, to supplement its creative work. As the producer, she was charged with handling legal clearance at a time when AI-generated video and audio based on a human’s likeness is still a gray area.

Her employer “compelled” the team to incorporate generative AI, then “in the same conversation, warned about the legal dangers”, she said. The result was constant stress.

This is a pattern that I see a lot in tech. None of the managers is taking responsibility. They create double-binds all the time - stuff has to be on time, but with high quality, has to done rapidly, but safe, one has to save work but the product should be secure, and so on. This is not psychologically healthy.

So, a hit-and-run.

I think the legal principle of liability is that a company is fully liable for the damage that their products cause. “It was an automation” or “it was a robot” is not an excuse.

So, maybe the consequence should be the permanent cancellation of the driving license of the responsible persons? If nobody else, the CEO of that company?

There will be trials in our lifetime, and these people will be judged by how many human beings they have killed with their actions. Science leaves no iota of doubt what are the consequences of what they do.

Yeah, it clones the Unix / POSIX API: Programs written for Unix will run on Linux. In the same way, the Photoshop/Office lookalikes are clones, they were written from scratch to yield a result compatible to the original.

Look at the explanations I linked.

Also it’s incredibly telling how Anthropic doesn’t add code to stop suicide ideation in teens (what they are their ilk are being sued for) but can somehow very quickly stop reverse engineering.

This.

I can assure you local models will blow up too.

Why?

The adobe suite was not reverse engineered, in the sense in which the term is applied to binary, closed-source softeware. Rather, it was cloned.

This is technically and often also legally very different. Cloning is a new implementation of a system which reproduces the abstract API of an existing System. Since APIs are not subject to copyright law, this is legally unproblemaric. It is also a creative process (so far). For exsmple, Linux is a clone of AT & T Unix.

Reverse engineering uses techniques like analysing the internal behaviour and decompiling the binary code of an existing product. It is im general not illegal, but it can violate license agreements and is more in a grey area. Laws like the DMCA might also apply if some kind of copy protection is used.

BTW, many Linux drivers are reverse enginnered to some degree.

I don’t get why it is listed in the Arch Linux wiki.

Kafka has written other stories, the author might have had The Trial (Der Process) in mind.

And the supermarket systems still don’t get right the prices for fruits. Which means you have to call another employee to enter the price for you.

Just to get that straight: Legacy code is practically always large code with countless hidden requirements, and with insufficient or no tests and without documentation. But which works somehow. Any change needs to meet numerous unwritten requirements, and it can often only be tested in the real world. If you have N features, you have N*(N-1)/2 interactions between two features, so it is quadratic growth with N. This is why it is very hard and time-consuming to change.

It shouldn’t be that way, but writing large software - especially in commercial environments - is often like climbing a high roof on a ladder, and then throwing away that ladder. Modifications are only incremental, and usually not documented. Incremental modifications are one of the only two working engineering approaches for software we know - and the only one used for commercial software.

The other way - which is used in aviation for example - is to specify, document, and test everything, right from the start.

Now the problem is you can rewrite legacy stuff as much as you like, but it is near impossible to make sure that large changes work. Because in order to do that, you would need to run the tests that do not exist, or create tests from specifications and requirements which do not exist either.

And GenAI tools are not going to change that. The agentic tools kinda work by filling in missing requirements with assumptions and average observed stuff. It can generate reams of code, fast. But while this kinda works for a small phone app, it doesn’t for large complex stuff. If the code does not fit in the head of one person working for months on it, it will never fit in the context window of an LLM.

These companies advertising methods are becoming a bit too much!

and now they can freely make copies of some really important keys^*

* Communications, which could be you logging in into your banking account. The attacker would be able to see your banking password, for example. Or it could replace a software update your laptop is receiving with malware. A manipulated WLAN router in a cafe or an airport could do that.

Oof. Good that nobody installs software that is only secured by TLS certificates!

this is absolutely solved

How? According to the EU CRA, each connected device should have unique access credentials. How do you solve that in practice, in an organization like this?

I agree that using a VPN is much more secure. And together with not allowing remote temperature control, this would have prevented this specific issue.

Still, if you have a VPN in a large supermarket chain, you need to manage the access keys - in a way that when one user of the VPN quits the organization, their access is revoked.

And also, it isnot a good idea that many people share passwords to devices which give administrative rights. So, you should give a unique password to every user, similar as every guest in a hotel gets an NFC card with a unique code.

The best way I know of to do that would be a PKI and smart cards. This also prevents users from using dumb passwords or using the same ones in multiple places.

But my impression is that very few organizations like supermarkets do that.

It gets even more complex with the modern rush to outsource every little thing.

Well, in Europe this could be a condition for rapists to temporarily leave jail. At the end of their sentence.

You are right. But also, authentication in a distributed industrial infrastructure is hard, and it is not really a solved problem. You would need to set up a public key infrastructure, giving each shop clerk, service worker, and manager a smart card, revoking them when they leave, recover lost access credentials, and so on. For every class of devices.

And it gets even more difficult if you want machines to securely talk with each other (think in administering every smart home thing un your house with a smart card…).