Apparently we need to also be 14 to understand the words in that post

Storage is hard to do right :(

If you can get away with it, use a separate NAS that exposes NFS to your other machines. Iscsi with a csi might be an option too.

For databases, it’s usually better to not put their data on shared storage and instead use the databases built in replication (and take backups!).

But if you want to go down the rabbit hole, check out ceph, glusterfs, moosefs, seaweedfs, juicefs, and garagehq.

Most shared file systems aren’t fully posix compliant so things like file locking may not work. This affects databases and sqlite a lot. Glusterfs and moosefs seen to behave the best imo with sqlite db files. Seaweedfs should as well, but I’m still working on testing it.

+1 for Nomad. Ive used k8s a lot and still use it, but i prefer Nomad for home purposes. You dont even need a consul cluster to run it anymore so it’s pretty simple to start.

The second isn’t a bad idea if it’s in combination with the first. Then you have an image you can restore with most of your config and you can just restore the rest from the normal backups.

What I do right now is I have a rclone sidecar container that uploads files in a directory every few seconds, and I also have another init sidecar that runs before the main application and downloads those files (incl sqlite dbs) to the normal disk. This works okay but feels pretty clunky and can still result in stuff getting corrupted because I’m just backing up the db files and not using any sqlite commands to actually back up the db to another file that isn’t in-use first.

How do you handle a job going from one nomad node to another? Or do you pin jobs like grafana to specific hosts?

Thanks! I’ll do some testing over the weekend and see how it goes.

While I’d love to be able to use it for postgres, I figured that wouldn’t work out well so probably won’t try it any time soon. I do have several apps that use sqlite databases though, do you think those would have any issues? e.g. trilium, ntfy, ghost

The main downside to most of the distributed/clustered storage that I’ve tried is they always seem to corrupt sqlite db files due to not supporting locking or some other posix feature. Reading through some older github issues, it looks like that is something the dev of seaweedfs fixed hopefully.

Hey, your stack is pretty similar to mine. One thing I recently started testing is Seaweedfs. I saw it listed in your repo too, how are you liking it so far? And do you use it on all of your nodes?

UptimeKuma is great, I use it for the simple “are my services up?” and is what I pay most attention to.

I still use zabbix for finer grained monitors though like checking raid status, smartctl, disk space, temperatures, etc.

I’ve been trying out librenms with more custom snmp checks too and am considering going that route instead of zabbix in the future

deleted by creator

Bookstack is really nice and user friendly. It’s probably one of my favorites.

Dokuwiki is simple and stores files in plaintext.

I haven’t used wiki.js much but I’ve heard good things about it too.

Another option if you don’t need to share the wiki with anyone would be a note tool like Trilium. It has built in support for stuff like mermaid or excalidraw diagrams.

Don’t forget to setup backups for whatever wiki you do go with, and make sure you can restore them when your wiki is broken ;)

That makes sense, it does sound better to keep it within nixos! I’ve mostly been using nixos to bootstrap servers that run nomad+docker, so beyond the system-level config, I haven’t done a lot with additional software yet.

Make sure your backups are solid and can’t be deleted or altered.

In addition to normal backups, something like zfs snapshots also help and make it easier to restore if needed.

I think I remember seeing a nextcloud plugin that detects mass changes to a lot of files (like ransomware would cause). Maybe something like that would help?

Also enforce good passwords.

Do you have anything exposed to the internet that also has access to either nextcloud or the server it’s running on? If so, lock that down as much as possible too.

Fail2ban or similar would help against brute force attacks.

The VM you’re running nextcloud on should be as isolated as you can comfortably make it. E.g. if you have a camera/iot vlan, don’t let the VM talk to it. Don’t let it initiate outbound connections to any of your devices, etc

You can’t entirely protect against zero day vulnerabilities, but you can do a lot to limit the risk and blast radius.

Iirc crowdsec is like fail2ban but blocks ips reported by other servers, not just ones attacking your server. Kinda like a distributed fail2ban I guess?

I only recently started using nix and NixOS. How’s the update process for nextcloud? Can you use the self updater?

I kinda want to know what a Moonie is, but I’m going to choose to believe it’s someone who believes the Nazis have colonized the dark side of the moon and are growing genetically engineered dinosaurs.

I’m not 100% sure, but wasn’t ssdnodes one of the companies that offers really cheap deals without actually giving you the specs they say?

E.g. they say 64gb ram, but you actually get a VM with memory ballooning enabled and then your account gets suspended if you consistently use that much ram

I’ve tried lot of different apps, but I think I’ve settled on Trilium for now.

It doesn’t have a great mobile experience, but the web app works fine on mobile. The app in general is super customizable and way easier to write scripts / plugins for.

But this doesn’t look like it will be compatible with current ActivityPub, right?

Do you like any of them more than others?

Have you reported the posts and users you’re referring to? Or have you tried reaching out to the admins yet? Without proper reporting, you can’t know if they’re even aware of an issue.