• 1 Post
  • 46 Comments
Joined 3 years ago
Cake day: October 4th, 2023
  • tal@lemmy.todaytoTechnology@lemmy.world•AI Layoffs Tracker — Job Cuts Attributed to AIEnglish
    20·
    2 days ago

    Yeah, I’ve seen a number of articles talking about how if a company isn’t doing well and needs to do layoffs that it’s a better narrative for investors that savings were found via AI. Then it’s not a “company is doing poorly” story, but a “company is forward-thinking” one.

    Like, there are real layoffs that have happened and are induced by AI, but I’d also be pretty confident in saying that it’s not all of the ones where leadership is citing AI.

  • tal@lemmy.todaytoSelfhosted@lemmy.world•[AIT] Experiences with using openclaw as a sex/roleplaying chatbot without GPU?English
    71·
    6 days ago

    I’ve already got a 128GB Framework Desktop that I picked up prior to the memory shortage for AI compute stuff, but if I didn’t…

    In 2028, I expect memory prices to start coming down.

    The GPU hardware itself isn’t really the limiting factor, but rather available memory.

    I just don’t think that I’d be willing to put in massive amounts of effort trying to work around hardware limitations (or spend a bunch of money, which is the other route, outbidding people for the limited supply of memory) to get a local AI compute setup going in 2026 or 2027 when the hardware situation should change substantially in the not-so-distant future.

    Specifically on generating literary erotica, my own experience is that quality is substantially better with larger, dense models. I’ve read that MoE models, like Qwen, are not very good at it, and my limited experimentation with Qwen supports that, though I haven’t tried to aggressively track the latest versions.

    If you look at this thread:

    https://www.reddit.com/r/SillyTavernAI/comments/1vp24rj/qwen_38_27b_is_the_next_stepping_stone_for_local/

    …while the OP in that thread is talking about using Qwen 3.6 for erotic role-play (ERP), other users are not generally all that enthusiastic about Qwen 3.6.

    In general, I think that /r/SillyTavernAI favors /u/TheDrummer’s models—which are not merely abliterated base models, but ERP-oriented finetunes—for ERP.

    EDIT: If you are after generating erotica, my own experience is that TheDrummer’s Cydonia, which is recommended in that thread, is the best smaller (consumer GPU-runnable) model, though it starts to get nonsensical once the history grows large enough. I don’t know if you’d get a tolerable response rate out of that on a CPU-only setup. The best quality I feel I’ve seen has been out of the Llama-3-based AnubisLemonade, but I think that it’s going to be painful to run that on even a very high-end CPU-only setup. On my 128GB, unified memory setup, I use a Q6_K quantitization of that with llama.cpp run with (checks) --direct-io --fit-off --no-mmproj -c0 -ngl 99 --dry-multiplier 0.8 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on and a 128k token context window.

    I’m presently using Vulkan for compute with llama.cpp, though my experience with ComfyUI is that ROCm 10.1 is (finally) stable and effective on Strix Halo hardware, and I’ve been meaning to look into how llama.cpp handles ROCm now. In the past, llama.cpp had ROCm win on prompt processing performance and Vulkan win on inference performance; I’d been using Vulkan because ROCm releases had been having problems on that hardware and because I used a large context window and unchanging-from-prompt-to-prompt history with llama.cpp, so prompt processing time didn’t matter much, since one could rely on the cache to avoid most of the prompt processing, and inference time did matter.

    The quantitization there is done to get response time down rather than because the system is memory-constrained.

    In your case, the performance constraints may be quite different.

    Often discrete GPU-based users (more bandwidth than my system, but much less memory) aim to use a small context window and have the system modify the history for each prompt, so they can’t leverage their K-V cache but can play tricks with things like RAG to keep a small context window and still keep a lot of relevant information, injecting relevant information into the history on each prompt. The problem they run into is generally trying to work around limited memory, not being able to store large models and a lot of history.

    On my unified-memory setup, I use a large context window, leverage my memory, and rely on cache to keep performance acceptable. I’ve got plenty of memory, so I don’t have to worry about large model size or having loads of history, but have less memory bandwidth, so response time is the issue that I work on. So I use very different models and techniques from what’s optimal for a discrete GPU user.

    Not sure what the best route for a CPU-only setup is. I’d expect that to be less memory-constrained than a GPU user, but be compute-constrained.

  • tal@lemmy.todaytoFediverse@lemmy.world•SOLVED: lemmy.cafe downEnglish
    8·
    9 days ago

    I hit archive.org’s Wayback Machine—learned in the process, that apparently use of Anubis breaks the Wayback Machine, which is…a little concerning. I don’t know how many websites out there aren’t being archived now…

    Anyway, from a pre-Anubis page Archive.org page, the admins appear to be:

    Looking at lemmy.world’s view of their user pages (which is still accessible):

    I don’t know whether user pages on remote instances than a user’s home instance are complete; I’m a little suspicious that they might only display comments and posts to communities to which at least one user who subscribes to the community with the post/comment on the home instance. But with lemmy.world the largest instance, my guess is that that’s the most-complete record of comments one might get.

    On lemvotes.org, looking for vote activity (which is probably at least-as-current):

    They might have non-admin accounts and use those for everything, but I’d guess that Illecors is likely going to be the only “active” admin, from that.

    Based on the view of the lemmy.world user page for him, he does follow [email protected] and a number of other major communities, so he might see something you post there if the instance is down and he’s using an account on another home instance to get on. He does not list any non-lemmy.cafe contact information in his user bio.

    The archive.org page I listed above does have:

    Lemmy Cafe Chatroom

    In case there’s a need for a chat-like environment, a SimpleX chat is now available.

    I’m not familiar with SimpleX, and I don’t know if that’s still active or if it’s possible to get to it through Archive.org’s archived page, but it does show a page with a QR code, which you might be able to scan with a mobile client to reach it.