Personally I always use containers unless there is a good reason to use a VM, and those reasons do exist. Sometime you want a whole, fully functional OS complete with custom kernel, in that situation a VM is a good idea, sometimes a utility only comes packaged as a VM.

But absent of a good reason, containers are just better in the majority of cases

Unless you have multiple systems, I don’t think k8s will yield much benefit over plain docker.

Containers, unless you have a specific need for a VM.

With a VM you have to reserve resources exclusively. If you give a VM 2gb of ram, then that’s 2gb of ram that you can’t use for other things, even if the guest OS is using less.

With Containers, you only need as many resources as the process inside the container requires at the time.

Why not use both? I have PVE installed on all of my hosts and then use k3s/docker in VMs. If there ever is anything you don’t want to or just can’t deploy as a container (e.g. opnsense, hassio, truenas, windows [for whatever reason you might have]), you can just spin it up as a VM and not worry about adding and maintaining another physical machine

What I did is install proxmox on the bare metal, setup a vm in which I put the containers.

Proxmox itself stays (almost) completely stock. The only changes I’ve made to it were to add the NUT client package so it could gracefully shut down if my NUT server indicates that the UPS is running out of power during an outage.

In your VMs you can do whatever. Setup OMV, or a stock Ubuntu or Debian vm and install your services on the VM or use Docker/Podman. Setup Fedora CoreOS or IoT vms and host all your services in Podman containers.

The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.

You can also spin up new vms as needed to just try to fuck around with new techs or just a new way of setting up your home lab. It gives you a ton of flexibility and makes backing stuff up way easier.

Another great thing you can do is if 3 years down the line you are looking to replace your server hardware with some newer or more powerful stuff you can just add the new device as a node to the cluster. Then you can migrate all your existing VMs over to your new hardware and decommission your old one with very little to no downtime on anything.

You can try casaOS.

https://github.com/IceWhaleTech/CasaOS

Its something like k8s. But it is easy to use and works very well with docker containers.

Container processes are just ordinary linux processes, so they don’t need extra overhead (cpu and ram reservation) to run, which means your machine can run more of them. If you have a machine with 32GB of ram, can probably run 15 VMs with 2GB of ram each where the actual app running inside the VM might only consume about 50% of the VM ram, or you can run them as container and they all would just consume 15GB of ram, leaving you extra to run more containers. I found this to be ideal for self hosting because all apps are your personal apps so interprocess isolation is not as important compared to running in public cloud.

I’d suggest looking into k8s. It’s definitely a bit more complex on the start, but so much more power once you get to the details. VMs you don’t share the base OS layer and the hardware, you have to pre-define the resources you need per app in a more constrained manner, while containers can move freely in their little sandbox to pickup whatever it needs.

It is also much easier to manage replicas, upgrades, scale and a bunch of other things once you are using containers and an orchestrator like Kubernetes. Let me know if you need any help/insights. I’ve been trying to post more videos/answers about things that could be complicated.

VMs are often imperative and can be quite easy and familiar to setup for most people, but can be harder or more time-consuming to reproduce, depending on the type of update or error to be fixed. They have their own kernel and can have window managers and graphical interfaces, and can therefore also be a bit resource heavy.

Containers are declarative and are quite easy to reproduce, but can be harder to setup, as you’ll have to work by trial-and-error from the CLI. They also run on your computers kernel and can be extremely slimmed down.

They are both powerful, depends how you want to maintain and interface with them, how resource efficient you want them to be, and how much you’re willing to learn if necessary.

Why not do both? I run proxmox on my physical hardware, then have guest VMs within proxmox that run k8s.

Advantages of proxmox:

• Proxmox makes it easy to spin up VMs for non self host purposes (say I want to play with NixOS)
• Proxmox snapshots make migrations and configuration changes a bit safer (I recently messed up a postgres 15 migration and was able to roll back in a button press)

You can then just run docker images through Proxmox, but I like k8s (specifically k3s) because:

Advantages of k8s:

• Certmanager means your HTTP services automatically get assigned TLS certs essentially for free (once you’ve set up cert manager for the first time, anyway)
• I find k8s’ YML-based configuration easier to track and manage. I can spin my containers up fresh just from my config, without worrying about stray environment settings I might not have backed up.
• k8s makes it easy for me to reason about which services are exposed internally to each other, and which are exposed on the host outside of my k8s cluster.
• k8s services get persistent DNS and IPs within the cluster, so configuring nodes to talk to each other is very easy.

And yeah, this way I get to learn two technologies rather than one 😁

I have a pretty low power server at home (Pentium G4560), and the previous one was even slower J3160, so I don’t want to unnecessarily hog the CPU with a VM, and the few services I need at home run perfectly fine in containers.

I run pihole, unbound, wireguard, plex, unifi controller in containers, and I run some additional services directly on the host (samba, transmission).

I have a Windows VM on my Windows PC for work, so it’s isolated from my main rig (various VPN clients and work files etc), and if I needed some Linux stuff on my Windows PC I’d also run a VM, but more VMs also mean more updating and patching, which is much easier with containers.

VMs if you have enough RAM and/or need to run something on a non-compatible system (like pfsense on ARM). Containers for everything else.

I, personally, haven’t done a whole lot of VM work but I do run a metric ass-ton of containers. I can spool up servers in docker compose on absolutely dogshit hardware and have it run serviceably. Also, the immutability of the container OS is really nice for moving things around and/or getting them set up quickly.

If it’s relevant to your actual job, learning to use k8s will benefit you more. Generally i’d prefer to keep the bare metal OS as clean as possible to avoid breaking anything during upgrades and such, and keep the containers and normal running apps on separate VMs that can communicate with eachothers, k8s is mostly good if you got a lot of servers and want to manage them all at once through a single “orchestrator”. But for self hosting stuff in your home it’s kinda overkill. But it still can be used to manage things up. So imo go for k8s since it can be used in homeservers, it’s just that it’s kinda like using a nuclear bomb to kill a wasp.

If everything you want to run makes sense to do within k8s it is perfectly reasonable to run k8s on some bare-metal OS. Some things lend themselves to certain ways of running them better than others. E.g. Home Assistant really does not like to run anywhere but a dedicated machine/VM (at least last time I looked into it).

Regardless of k8s it may make sense to run some sort of virtualization layer just to make management easier. One panel you can use to access all of the machines in you k8s cluster from a console level can be pretty nice, and a Proxmox cluster would give you this. You can make a VM on a host that takes up basically all of the available RAM/CPU on it. Proxmox specifically has some built-in niceties with gluster (which I’ve never use, I manage gluster myself on bare metal) which could even be useful inside a k8s cluster for PVCs and the like.

If you are willing to get weird (and experimental) look into Rancher’s Harvester it’s an HCI platform (similar to Proxmox or vSphere) that uses k8s as its base layer and even manages VMs through k8s APIs… I played with it a bit and it was really neat, but opted for bare metal Ubuntu for my lab install (and actually moved from rke2 to k3s to Nomad to docker compose with some custom management/clustering over the course of a few years).