WYGIWYG

  • 0 Posts
  • 47 Comments
Joined 2 years ago
cake
Cake day: September 24th, 2024

help-circle









  • You can host a small LLM model with a phone, or a whisper or a TTS, maybe some light diffusion.

    You cannot do tool use, reasonable coding, get non-hallucination answers on even moderate questions. That stuff starts to get viable around 24G and really needs closer to 200G to be able to do most of that they’re doing.

    the reason they bought up all the ram is to keep someone from buying 200G of vram and diying cards putting them out of business.




  • Pardon the mess, this turned out to be quite a lot of text.

    No vector DB, no paid API, no full articles. Plain Python pulling free RSS feeds, plus Ollama running on my own 2070 Super. RSS already gives you the headline and a short blurb, so there are never megabytes to sift through. Every Ollama call is one story at a time, so the context window isn’t an issue. Llama3.1:8b for summaries and scoring. Gemma4:e4b for clickbate work, nomic-embed-text for headline duplicate vectors.

    Find duplicates:

    Python: strip each headline down to keywords (drop words like “the”, “says”, “update”) and only compare headlines that share an uncommon word. Two headlines match if they share enough keywords.

    Python: sends batches of 128 headlines to Ollama

    Ollama: turns each headline into a vector using nomic-embed-text

    Python: compares the vectors, and anything over the config threshold counts as the same story. The vectors just live in memory for the run, no database needed. If Ollama is down, the keyword match still works on its own.

    Regen short summaries:

    Ollama: Generate a 2 sentence summary from the feed text if available

    Get rid of clickbait:

    Ollama: Detect clickbait from the headline only. Classify this news headline. “informative”: it states the news itself (who did what, what happened, what was found). Dramatic or emotional wording is fine if the fact is stated. “teaser”: it withholds the key fact to make you click: e.g. “The Surprising Reason…”, “The Class That…”, “Why X…”, “How X became…”, “What we know about…”, “Is X about to…?”, “Then things got weird.”

    Unwrap clickbait and rewrite the headline if possible, otherwise drop the story

    Ollama: A clickbait headline hides a key fact. Question it leaves open: {withheld}

    Does the text below explicitly answer that exact question? Mentioning the topi talking around it, or answering a different question does not count.

    “answer”: the answer quoted or closely paraphrased from the text, or “” if the text doesn’t give it. “headline”: if answered, a plain, factual replacement headline (at most 90 cha states that answer, using only facts stated in the text. Keep the text’s certainty: …

    Respond only with JSON: {“answer”: “”, “headline”: “”}

    Headline: {title} Text: {text}

    Score each headline

    Ollama: return a summary and an importance score in the same call:

    SCALE = ("1-3 niche, local or trivial; 4-6 notable; 7-8 major national or inte “9-10 historic events or emergencies affecting millions right now.”)

    Python: rank = importance + a bonus for more outlets carrying it - points for age

    Don’t put US articles in World unless they have non-US countries involved

    Ollama: Which countries is this news story about? List the country where it takes place and every other country that is a real party to it (acting, targeted, negotiating, fighting, trading, sanctioned). Use common English country names, e.g. “United States”, “United Kingdom”. Ignore countries only mentioned in passing or as background. Always name at least one country: if the text doesn’t say wh give the country where it most likely takes place.

    Respond only with JSON: {“countries”: [“…”]}

    Headline: {title} Text: {text}

    Python: if the only country is the US, it’s left out of World

    Everything gets cached per story, so each one only goes through Ollama once.


  • Ollama container, 2070 super, 3-7b, meta and qwen

    I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.

    I also use it for a self hosted mealie to figure out recipe->ingredient.

    I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.






  • To an extent. Perf is often as much about architecture as it is about code. That guy who created Adobe alternatives- all those apps are dog slow; they have to do low-quality preview renders. That’s something Adobe probably spent a few lifetimes siphoning out every last watt of power from, and it’s the reason they didn’t want to move to Linux. The app is still serviceable, but it has to degrade active quality to work at a reasonable pace. To fix that in Rust is going to be an enormous ask. There needs to be hackery and jiggery-pokery, the kind of things that aren’t documented well in training.

    Maybe he can go back and have it try to figure out what Adobe is going or brainstorm ways to do it at full quality. Maybe someone will have to dive into the code, get familiar with it (hoping it’s not too confusing), and either fix the architecture, fuck with the security to let it use more base system calls, or maybe just tell another Claude session how to make it more performant.