That is good to know! Will keep an eye on memory usage of immich. I really like it, so I’m reluctant to let it go.

Thanks heaps, that’s is very useful. Will connect monitor and keyboard and have a look.

I’ve uninstalled it, it’s an index/search tool. Don’t need it :D

Thanks, it’s so awesome to see so many useful replies here! If you are interested, I found some very weird things in the logs :( https://lemmy.nz/comment/6192604

Yeah think I’ll get a standalone WiFi smart plug, not connected to my Home Assistant, as a kill switch. But you’re right, it’s overkill.

I found some weird things in the logs, this goes beyond my knowledge :( See https://lemmy.nz/comment/6192604

Yeah tracker miner sounds dodgy. I’ve only installed Immich & Frigate on the box, and no dodgy repositories. It’s also auto updating. Will do research how to check for malware, thought that was a Windows only thing :D

Thanks, but a data center is probably overkill for my needs. I’ve got it power loss protected with a UPS, and that’s more than enough for us. Thanks anyway :)

I have a RPI, but of course that one can hang too. I’ll buy a simple WiFi smart plug, standalone, as a kill switch.

Thanks! Yeah it seemed to be an OOM issue, but based on my Kagi qualities it seems like an OS issue. But, it also has an error about the GPU. Normal memory usage is more than fine, so perhaps it was a one time thing. See logs: https://lemmy.nz/comment/6192604

Thanks mate. It was a bit of a rabbit hole, I found stuff about the watchdog package, and you can configure it to use the iTCO_wdt module, but I also read it was blacklisted, and then I just gave up. I posted somewhere else in the thread what lead up to the hang. And, I think I’ll buy a WiFi smartplug so I can remotely reboot everything; assuming the WiFi still works :D

Yes, thanks for that. Good point. I checked the logs, and minutes before it crashed I can see below in the logs. Seems like either a GPU error or out of memory error. I’ve deleted tracker-miner-fs as I don’t need it. It also shows a massive list of processes with their memory usage.

Feb 21 17:27:49 hppd600-g3 kernel: i915 0000:00:02.0: [drm] GPU HANG: ecode 9:0:00000000
Feb 21 17:32:43 hppd600-g3 kernel: 1305621 total pagecache pages
Feb 21 17:32:43 hppd600-g3 kernel: 16258 pages in swap cache
Feb 21 17:32:43 hppd600-g3 kernel: Free swap  = 0kB
Feb 21 17:32:43 hppd600-g3 kernel: Total swap = 1000444kB
Feb 21 17:32:43 hppd600-g3 kernel: 2065206 pages RAM
Feb 21 17:32:43 hppd600-g3 kernel: 0 pages HighMem/MovableOnly
Feb 21 17:32:43 hppd600-g3 kernel: 64196 pages reserved
Feb 21 17:32:43 hppd600-g3 kernel: 0 pages hwpoisoned

Feb 21 17:32:43 hppd600-g3 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-113.slice/user@113.service/background.slice/tracker-miner-fs-3.service,task=t>
Feb 21 17:32:43 hppd600-g3 kernel: Out of memory: Killed process 833 (tracker-miner-f) total-vm:625676kB, anon-rss:3144kB, file-rss:4816kB, shmem-rss:4kB, UID:113 pgtables:280kB oom_score_adj:200
Feb 21 17:32:43 hppd600-g3 kernel: i915 0000:00:02.0: [drm] Resetting rcs0 for stopped heartbeat on rcs0

Yes, thanks for that. Good point. I checked the logs, and minutes before it crashed I can see below in the logs. Seems like either a GPU error or out of memory error. No idea what tracker-miner-f is by the way. It also shows a massive list of processes with their memory usage.

This goes beyond my knowledge :(

Feb 21 17:27:49 hppd600-g3 kernel: i915 0000:00:02.0: [drm] GPU HANG: ecode 9:0:00000000
Feb 21 17:32:43 hppd600-g3 kernel: 1305621 total pagecache pages
Feb 21 17:32:43 hppd600-g3 kernel: 16258 pages in swap cache
Feb 21 17:32:43 hppd600-g3 kernel: Free swap  = 0kB
Feb 21 17:32:43 hppd600-g3 kernel: Total swap = 1000444kB
Feb 21 17:32:43 hppd600-g3 kernel: 2065206 pages RAM
Feb 21 17:32:43 hppd600-g3 kernel: 0 pages HighMem/MovableOnly
Feb 21 17:32:43 hppd600-g3 kernel: 64196 pages reserved
Feb 21 17:32:43 hppd600-g3 kernel: 0 pages hwpoisoned

Feb 21 17:32:43 hppd600-g3 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-113.slice/user@113.service/background.slice/tracker-miner-fs-3.service,task=t>
Feb 21 17:32:43 hppd600-g3 kernel: Out of memory: Killed process 833 (tracker-miner-f) total-vm:625676kB, anon-rss:3144kB, file-rss:4816kB, shmem-rss:4kB, UID:113 pgtables:280kB oom_score_adj:200
Feb 21 17:32:43 hppd600-g3 kernel: i915 0000:00:02.0: [drm] Resetting rcs0 for stopped heartbeat on rcs0

Thanks, I’ve got a HP SFF as well. Not 100% sure how to turn it on though from Ubuntu. There’s a software based version: https://manpages.ubuntu.com/manpages/xenial/en/man8/watchdog.8.html

But I guess that’s not the one using the motherboard watchdog function.

Thanks! That should work.

Yes, very good idea. I’ve got HA on a RPI so that should be easy.

That’s interesting! I’ve converted all my docker run commands to docker compose, as I found that easier to manage. But, I guess you can’t do the dependencies like you have. Also, yours has the advantage it always pulls the latest.

Awesome, yes, definitely will do. After years of using Linux, the whole systemd thing is still a bit of a black box to me. I know how to create /start/stop services etc but that’s about it. Thanks for the prompt replies!

Yeah I used /etc/fstab which are static mounts.

I switched to autofs and that seems to be much better, as it does the mounts “at runtime”, ie when requested.

Thanks! I’ve just set that up. That would seem to solve the solution, right, without reboots?

Sorry, I’m absolutely not a Linux expert :) I use /etc/fstab for the mounts, and to manually re-mount I run “mount -a”.

Welcome in the rabbit hole of selfhosted enthousiast.

Haha yes looks like it. I only have 1 server (to be bought) and a RPI, which should be sufficient. Proxmox seems overkill for that scenario; I’ll just run Docker. Thanks.