Linux for sure. Whatever distro seems cool to you.

My useless advice: Do it in phases as you learn.

• Start off with Yunohost. It is simple to get started and works pretty well. Try different apps to see what you like and what might be worth using for real. Just make sure that you keep in mind this is more of a “proof of concept” for testing things. Unless you plan to purchase another mini pc later.
• When you feel like you have out grown it and want to start learning more about things, you can move to something like Proxmox. This allows you to create virtual machines and play with containers (docker/lxc). If you plan well, you can back up your Yunohost data and configs to another drive, wipe Yunohost install and replace it with Proxmox. Then install a VM running Yunohost and restore your data and configs you previously backed up.
• Then you can start playing with lxc containers and docker containers.
• If you can get a second machine with multiple drives, install TrueNAS or OMV. Use that to store all of your data on NFS drive that you mount from your Proxmox VMs and containers.

Years ago I used to run a linux server with everything installed under Apache virtual directories and fought the constant upgrade cycle. Life got in the way and I gave up on it until the pandemic slowed life down enough for me to start playing again. So I went the Yunhost route on an old Mac Mini. I now have a 3 node Proxmox cluster with Yunhost in a VM (with a dozen apps running on it) and another 15-20 containers running under either lxc or docker. I eventually purchased a cheap NAS device for data storage so that I could make use of the Proxmox fail over capabilities.

If your mini pc has the capability for two drives, install the OS on one and store data on the other (unless/until you get a second pc/NAS).

I wouldn’t do Windows, Linux will give you freedom to use docker for most things that you might want to host. As for which distro use whatever you find nice, there’s not going to be much difference. Some of the things people are suggesting are great for extremely advanced use cases, for just spinning up some services whatever you feel more comfortable would be best.

Personally… Id spin up a Ubuntu desktop instance for Jellyfin and Sonarr. I went through a few different operating systems before I went back to Ubuntu and…it just works. Lots of documentation on installing those services that way. Jellyfin is very happy behind Sonarr… I’d stay away from Windows for any kind of selfhosting because I don’t trust them

proxmox and then you can run either LXC containers directly on it or VMs when necessary. Almost anything linux-based can be in a container on proxmox and has no need for an actual VM. I have a mini cluster of 3 proxmox servers running jellyfin, all and the -arrs, nzbget, nginx, searxng, pihole, ombi, seaweedfs, streammaster, dizquetv, syncthing, nitter, teddit, piped, vaultwarden, changedetector, freshrss, headscale, rimgo, nextcloud, ntfy, quetre, uptime kuma, homeassistant, kavita, scribe, nzbget and a Windows Server VM. The stuff could all run on one of them, but I like to be able to move stuff around since the 3 instances are not all the same. One is an older server that I’ve had forever. One is about 5 years old, but has 32GB of RAM and 60+TB of disk. The new one has 64GB and an SSD that will do about 7GB/s.

I have been on this path (10+ years in)

• windows server
• FreeNAS > TrueNAS
• Proxmox
• Bare metal Debian with all Docker containers

I learned the most with Proxmox (and ZFS), everything in LXCs.

But I enjoy the pure Debian install with Docker containers the most!

Ps: do not, I repeat, do not install a desktop OS. Pps: stay away from Ubuntu ;)

I recommend debian 12 works like charm

Proxmox with a bunch of small dedicated lxc containers (running small Ubuntu lxcs mostly). It works for me anyway, though I don’t use any of the rr programs. My Plex lxc gets its Media files from an openmediavault NFS mount (Running in a VM instead of lxc), and I use smb file sharing for my windows desktop to access the same files. Your mileage may vary, what works for me night not be for someone else.

I highly recommend Ubuntu Desktop for homelab stuff.

I recently set up my self-hosted system and went down this path:
Ubuntu Server - decided against it because terminal only is tedious sometimes
Ubuntu Desktop - easy, popular (so more documentation), comes ready with Docker
Decided to use Yacht for my container manager and have been happy with it so far, super simple compared to other options out there but you still have the control you need

current containers:

• Home Assistant to unify all smart home gadgets
• Nginx Proxy Manager to create secure reverse proxies so some services are available to the internet for remote use
• Photoprism for all my photo / personal video management
• Plex to serve my media
• Dashy a dashboard for all your services? I still need to learn and play with this one
• Firefly III a personal finance tool, hard to configure but seems super useful once everything is set up and imported. still learning this one too
• Minecraft Server

I definitely want to add to this list. I want to get some stuff behind a VPN and sail the open seas to fill out my plex server, I just want to make sure I am being as safe as possible.

Have a look at Proxmox.

I would never want anything else anymore as the base system on a server.

I’d recommend Debian Linux. It’s free, stable, has all the software you’ll need with long term support, ton of online resources and communities to learn from. You can start with or without UI.

I get where the people saying Proxmox are coming from but realistically, this is a mini-computer and the setup doesn’t need to be that sophisticated. Running Linux will go a long way, slapping Docker containers on there for things.

If you’re super new to all of this, check out Yunohost. You can either run it on Debian 11 or use their own OS image. Its a simple UI interface and makes apps deployable with one click. Doesn’t support everything but I have this running beside my docker just because it’s so dang easy to use and maintain.

My vote is also on Proxmox. I also started out with a mini PC about 6 months ago and installed proxmox. It has a few quirks regarding its disk usage you have to get used to, but its UI and setting up of VMs and LXC containers is so simple, you do not need any manual. You can easily build and tear down linux machines and play around with them. You cannot run docker images directly, but I use an LXC ubuntu container to run them, works flawlessly.