fuck yeah monday checkmarks, bitches!
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters CGNAT Carrier-Grade NAT Git Popular version control system, primarily for code NAS Network-Attached Storage NAT Network Address Translation VPS Virtual Private Server (opposed to shared hosting)
[Thread #148 for this comm, first seen 27th Oct 2026, 16:10] [FAQ] [Full list] [Contact] [Source code]
Everybody tests their backups. Some people even manage to test their backups before they need them.
Congrats on becoming one of the 5% of ppl who backup, but actually also test them.
5 %, that many?
There are two kinds of self-hosters: those who backup, and those who will start testing em after their first failed restore.
I tested mine involuntarily. I enabled full disk encryption on my the main drive of my server, ignored all the warnings and then noticed, that the whole drive was formatted.
Luckily, all the important data was in my backups, I lost some compose files and the database of my paperless ngx seems to be broken (although theoretically, I am sure I could recover that data, I just don’t have that time and there is nothing that important on there).
Recently read a comment here: “Think you have a backup? Delete your drives now. If you’re hesitant, you don’t really have a backup.”
As true as that is: I’d be in the “hesitant” category because I only back up my data / containers, not my bare metal system and its config. Do you guys? (Proxmox users will probably do, but what about the rest?)
I could do that, wouldn’t lose a bit, but I hesitate anyways because downloading a few TB over a 300 Mbit/s Internet connection isn’t fun at all and takes a shitload of time
<cries in 100 MBit/s connection>
spoiler
Aber, wie ich sehe, ein Landsmann. Für erschwingliche Glasfaser leben wir einfach im falschen Land.
DSL? Or Cable?
Spoiler
Jep, leider auch lange von deutschem Internet geplagt (bis 2023 nur max. 50 Mbit). Aber seit hier vor 3 Jahren Glasfaser verlegt wurde, hätten wir auch Gigabit verfügbar, würde knapp das 1.5fache kosten (59.99 statt 39.99 im Monat). Für normale Nutzung, selbst mit Up-/Downloads von Docker-/ISO-Images reichen 300 Mbit eigentlich. Wohne halt noch bei meinen Eltern, und mein Vater will nicht mehr bezahlen als jetzt, aber bin durchaus ab und zu am überlegen, die 20€ pro Monat dazuzugeben
DSL.
spoiler
<spuckt sein Müsli über den Tisch> Wo gibt’s 300 Mbit für EUR 40 monatlich? <recherchiert> Ahja, Vodafone. Eigentlich nicht schlecht. Wie sind deine Erfahrungen mit denen? Kriegt man bei denen eine eigene öffentliche IPv4-Adresse ohne CGNAT?
Sitze leider im tiefsten Rentnerviertel; hier kann man froh sein, wenn die Leitungen aus Kupfer sind und nicht noch Fachwerk. Glasfaser sollte verlegt werden, aber dann geriet die Deutsche Glasfaser an den Rand der Insolvenz. Mal sehen, wie das weitergeht. Für mein bisschen Homelab reichen mir 100 MBit (bzw. magere 40 im Upstream) derzeit noch aus. Das könnte aber anders werden, wenn ich mehr Nutzer dazubekomme, mit denen ich mir die schmale Upstream-Bandbreite teilen muss.
The whole config is in git. The data gets backed up nightly by restic and pushed to a local backup site, and then to 2 separate offsite backup locations, fulfilling 3-2-1.
Every month on the first day there’s a restore test.
I actually did that. Or well, I was planning to upgrade the Debian version and instead of doing it in place I replaced the drive (kept the old around for a while just in case) and did a clean install. I didn’t make any new backups or moved configs or anything outside what I already had, the system was back to operations faily quickly. The main reason, I have a markdown file describing every configuration and recovery step needed to get a fresh system to a fully working state in case of a disaster, and my test showed that it worked great. It does take a bit of effort to keep it up-to-date, but as shown, really worth it, and it douples as a small wiki for accumulated knowledge. I do wish to play with Ansible at some point if I have the time, but this works for now.
Nice philosophy.
I’m kinda in the middle ground here.
I backup “everything” from my raspberries, and just \home for laptops but for an actual disaster recovery I’d just do a clean install, get an IP address and then reinstall everything via Ansible, then drag my home data back off the NAS
But… I like that statement. I’ll use that at work.
Is your NAS on site? If it is not backed up: I used to follow a 3-2-1 strategy, 3 separate copies, 2 mediums, 1 off site. I have sorta feel to a 2-2-1, I am wanting to get to a 2-2-2ish.
I was following a 2-2 strategy but then 3 houses went went up in flames down the street.
The NAS is on site, yes. It’s backed up to Hetzner - inc. it’s own
/etc,/home, etc. so the NAS itself is kinda a 2-1 (2 backups, 1 offsite), but it would be a lot of rework to restore and that’s the device I’d need to recover everything else in case of a total disaster (like a house fire).(🤔 better check the Hetzner encryption is noted in my keypass file)
I do need to get it’s config setup in Ansible though so it can be rebuilt automatically, but Ansible for single devices can be a little overkill and difficult to maintain.
Similar for my hypervisor box… damn… there goes some more evenings…
I do not have much that I I cannot just rebuild and retune so I got a little VPS, to backup my irreaplaceables and host my blog all the docker images can be replaced the pi’s can be replaced.
The encription I chose was age, I had just asked Vibe what the best option for me would be, wish I had chosen differently, maybe some day I will sit my butt down for a few days and figure out a new strategy. I wish I had chosen differently because it is coded in Go.
Looking Go up to see if there has been any big issues found with it I see that other major parts of my homelab are also coded, in full or in part, with Go. Most importantly Caddy followed closly with Docker so maybe I will not worry to much about Age being coded in Go! Without Caddy, fuck this girl aint remembering port numbers.
whenever the self hosting mood strikes, i try so much off the wall shit that it’d all be irrevocably broken if i wasn’t able to scrap the whole day and reload last nights back up!
i keep each group of services in their own lxc so at least only a part goes down if so!
Question for you, is there an off site backup of your stuff? If there is not an off site back up it aint backed up.
A little while ago there where 3 houses that went up in flames just down the block had the wind been blowing just a little, so many more houses would have gone up but had it blown just a little to the south I am sure my condo complex would have gone up. When the fire fighters told use to get back in to our places but be prepared to move I had all my devices and a few other things ready to go.
I thought how lucky I was to be home and the wind was not blowing, but I realized I needed an off site backup as well.
How do you check?
rm -rf /(don’t actually do this obv)
Here, you dropped this.
–no-preserve-root
Its OK, root was preserved
You reapply/ reload database and do all the recovery procedure on a clean new installation for everything. Disaster recovery is much a thing together with backup. Depends on your infra how to proceed in concrete steps.
Restore a backup and make sure everything is where you left it.
yep! restore a lxc on the backup machine and shut down the main one for a second
Thats good. So many places I see just accept that backup job completed without errors means backups are ok.
I have a remote backup of my home partition so I don’t loose any documents, videos, etc. How do you test these backups? I can browse them using Pika (GUI for BorgBackup) and I can do an integrity check, is there any more to it? I mean if my drive fails, I just copy those files back into my new drive’s home directory. But I don’t see how a test on that would be beneficial
What are you using to run backups?
proxmox style
apologies if that is not a lot of info!
Proxmox VM snapshots and backups, plus proxsave I assume?
Huge! Feels good, huh?
