• prodaccess@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    7 months ago

    Like all interesting outages, there are probably multiple key action items.

    I’m also curious why deleting a “staging volume” would affect prod. I don’t know Railway, but it seems like a bad architectural design.

    • PolarKraken@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      7 months ago

      Sounds like a responsible strategy to draw back from a lot of this. It’s all so…effervescently remade, the “ecosystem”, every few months.

      For me the takeaway comes from time I spent in some safety-critical parts of engineering and personal hobbies. Ultimately relying on people to make good decisions ~all of the time isn’t enough to prevent disaster, if something like disaster is on the line.

      Systems must be engineered to remove possibilities for accidentally bad, in-the-moment human decisions, where it counts. Thoughtfully. This is the weird same-shape but exactly-opposite doppelganger of that set of best practices.

      When the systems are using ~opaque automations that behave like humans (w.r.t. some decision-making and unreliable expectations of behavior) - and then relying on people making the right calls on top of that ever-shifting set of capabilities - I mean c’mon lol.

      This is gonna happen a lot, while the carrot of go-faster remains dangling so unignorably (because it’s in front of everyone, everyone working anywhere near the stuff). Until we look around and take a broader view. Which will be learned the same way we learned to make safety regulations, but I largely doubt our ability to respond in a similar way.

      The money will eventually respond, of course, but that’s always a poor and late proxy for what ought to be done.


      Sidenote, for aspiring engineers, take heart!

      It will be you who ends up tasked with unburying from all the technical debt incurred, truly. A practice steeped in the ancient wizardly traditions of yore. Spending a career on that and building something better.

      It will be necessary, the work begins roughly a while ago lol but more fully when things settle somewhat. Many large and slow organizations are right now very engaged in simply unburying themselves from the technical debt of a previous hype cycle, AKA now making use of all the data they collected (badly, via go-fast charlatans) during the “Big Data! You’ll be left behind if you don’t collect extreme amounts of data, it’s cheapish and everyone else is doing it!” era.

  • nonentity@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    0
    arrow-down
    1
    ·
    7 months ago

    LLMs can’t ’go rogue’, as that would require innate coherence and intent.

    They’re explosively imprecise, statistically luke-warm grey goo extrusion sphincters of historical sewage.

    Anyone who deploys one without supervision deserves everything it excretes, and anyone impressed by it enough that it resembles intelligence is betraying their limited natural capacity.

  • Retail4068@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    arrow-down
    1
    ·
    7 months ago

    The PocketOS boss puts greater blame on Railway’s architecture than on the deranged AI agent for the database’s irretrievable destruction. Briefly, the cloud provider’s API allows for destructive action without confirmation, it stores backups on the same volume as the source data, and “wiping a volume deletes all backups.” Crane also points out that CLI tokens have blanket permissions across environments.

    I’m sure they totally would have survived a junior hire.

    Pathetic hit piece for ignorant anti AI people.

  • alpha1beta@piefed.social
    link
    fedilink
    English
    arrow-up
    0
    arrow-down
    1
    ·
    7 months ago

    I’ll never happen to me. I ain’t stupid enough to use this shit. I’m not stupid enough to make myself unneeded. But damn, a lot of fucking programmers who I’m sure make 100k+ a year, are stupid fucks working to put themselves on the street by using this shit.