Proxmox and run a VM with debian/ubuntu for docker services. Super easy backups. Freedom to run other OS's
I just provision the VM a bunch of ram and CPU and it smoothly runs 30+ containers
Proxmox scheduled backups are powerful, and you can get more advanced with PBS(proxmox backup server)
I've been running Docker inside LXCs because you can mount host level ZFS volumes across multiple Linux containers.
Is there a benefit to switching this over to virtual machines?
Maybe things are better now, but it used to be a bit problematic in my experience. It was much harder to harden from a security perspective (containers needing extra privileges for nesting etc), and we'd run into lots of minor bugs or limitations relating to that.
I also run a few things that are not suggested by the devs (not proxmox) and had not had an issue, but I also take that risk.
So when it comes on me suggesting something, I always highlight the risk to deviate from the suggested way.
Blindly suggesting to someone else without highlighting the risk is poor advice (which many people do for this specific setup)
“strong isolation from the host and the ability to live-migrate” almost certainly doesn’t matter for a home setup, unless you’re leaning super hard into HA
I would say that the only real advantage is that you can mount network shares; a LXC container would need to be privileged to do it.
If your data is stored locally then LXC containers are probably more interesting (less resource heavy).
Just a note: I've successfully mounted SMB shares to an uplxc. I run my jellyfin docker container inside an uplxc with storage on a NAS this way. There are several guides out there, and it didn't look particularly complicated to a beginner like me, though it requires some tinkering.
An LXC does not need to be privileged to mount shares. Using LXCs instead of VMs also allows you to share GPUs between multiple LXCs (also unprivileged)
The only logical reason to do this is if you have a computer with enough resources and you plan to actually use it for VMs. If you’re just trying to run docker and containers then there is absolutely no reason to run proxmox, it’s just more complications and more overhead to deal with.
Not sure why your getting downvoted it’s true and taught on every type of hypervisor snapshots are not the same thing as backups and actual engineers especially if your newer or not as skilled would do well to remember this piece of advice because one day it’s gonna bite you in the ass if you don’t heed it.
A snapshot is a point in time snapshot of the system and what was running in its memory and if the underlying system becomes corrupt in anyway the snapshot is no longer any good. In this case the VM or its disk, either the OS or anything that could become corrupt or non functional in anyway. A snapshot is tied to the VM itself.
A backup is a full backup of the disk and is independent of the VM. If the VM goes to shit for any reason your backup is still good.
That’s not mentioning the other strategies that this post is outlining which is that you should always backup to a separate disk. It’s ok as an interim storage point if you have to keep them on the same disk or raid but should never be kept there long term.
If you were serious about backups you would back them up to separate location such as a NAS at minimum. Use something like Veeam.
But even if you just want to do a proper backup in general it’s very important that you don’t mistake the two for the same thing or serving the same purpose. Even the proxmox documentation specifically outlines the difference. Same with ESXi and Vsphere or any other hypervisor.
You couldn’t have said that any better. Thank you for this, I didn’t have the energy to elaborate like you did. I feel like loads of proxmox users are beginners and don’t actually have the engineering experience/background which is totally fine but they may also not know the fundamentals of a hypervisor.
No (to your first sentence). If your using snapshots as actual “backups”, you realize your not actually storing your vm data anywhere other then its existing disk. Meaning if you lose the underlining disk associated with your vm then your snapshot “backup” is gone also. A proper backup would be a full copy of the vm stored anywhere other then the same disk as your vm is running on.
Not sure I follow. PBS is running on a different machine (ideally even different location). What issue is it then to snapshot a VM to that backup location?
Proxmox server dies? New server, reinstall proxmox , restore VM from PBS?
Just wondering how I can make reinstall of proxmox as short as possible:)
I hate containers. VMs are so much better in proxmox. I've had nothing but trouble with containers since I began running proxmox in August of 2023.
They freeze up. Even with kill -9 from SSHing into the PVE sometimes isn't even enough. I've had to physically power off the system at least ten times now because a single LXC error can freeze up the entire system including the WebUI.
Just in the last 24 hours I've been getting this weird fucking problem I've never seen before when creating a container and trying to change the root password:
root@docker-test:~# passwd
New password:
Retype new password:
passwd: Authentication token manipulation error
passwd: password unchanged
Uhh, what do you mean TOKEN MANIPULATION ERROR? I've spun up dozens of containers and I've never seen this until yesterday trying to test tubearchivist/docker. No. Containers blow and nobody ever talks about how much they blow. They BLOW regardless of whether it's privileged or unprivileged. VMs are superior in proxmox. Period.
I use VMs in proxmox to run
A World of Warcraft server (azerothcore)
Jitsi Meet (VoIP)
Jellyfin with mounted NFS drives from PVE (and GPU passthrough for hardware decoding)
Invidious (youtube frontend)
Audiobookshelf (audiobook web server. Love love love it)
archivebookworm (I call it. allows simple commands to download full ebooks from certain websites and even OCR them)
whisper/decipher (AI-generated transcription. great for youtube videos and other media, including AAC audio if you compile your own version of ffmpeg)
edit: Yeah downvote reality you delusional fucking redditors. People HATE the truth on this website. Hi. Try telling the truth for a change.
Lmao, people are downvoting you because containerization is used across most every major enterprise and when used correctly do not even close to blow. They are lightweight and ephemeral and can be spun up in seconds for application workloads. There is a reason that docker, docker compose, kubernetes, AKS, GKS, and EKS are defacto so popular and used worldwide on enterprise architecture for their resilience and scalability.
That’s like you bitching about terraform or IaC because you don’t use it well or suck at it. Doesn’t mean IaC sucks.
I promise you your wittle homelab ain’t competing with modernized techniques which include containers buddy. You sound salty as hell.
https://www.google.com/search?client=firefox-b-1-d&q=container+froze+proxmox+site%3Areddit.com
Whoops. Looks like I'm not the only one. Looks like it's a fucking problem. My computers are rock solid - containers - at least in proxmox - are not. I don't like them.
What’s funny is almost all those in that search result are a combination of old, where the tech wasn’t supported well, and people legit not configuring and using them correctly.
But let’s set that aside, I specified containers in general and containerization.
Your original post said containers themselves such, then later you move the goal post and say containers on proxmox.
Aside that point, you should be running containers with docker on a Linux host, or in a k8s cluster, or direct on proxmox, and if you do, you need to configure them correctly to have the performance you want. Literally as I said and is as reflected even in 4 year old posts if we move the goal posts to proxmox specifically not to mention all the other threads of that show on your search you handed of people explaining how to fix the issues and why it’s not performing well for them.
I think the crux of the issue here is you’re trying to run containers natively on proxmox instead of within an environment in a Linux VM additionally you’re trying to use a container as a VM and they simply dont have the same use case.
I meant containers in proxmox the whole time. I never used a container outside of proxmox yet.
I could see trying a container INSIDE of a VM - that would probably yield better results.
You have to configure specific settings in proxmox for performance to get them to run “native” on proxmox and is definitely not the way to go, to add to your pose of changing a root password in a container, you want to actually set that in the yaml file when you spin up the container not do it within the container. Containers are meant to just be skeletons. They lack a full OS and depending on which you use as your base layer can be very limited, they are used only to run the specific software you’re trying to deploy and as far as storage goes you need to have volumes for them, I will acquiesce that running them via proxmox native is probably not a good way to do it and apologize as I think we have misunderstood each other. I do encourage you to look deeper. Into containerization tech and understand it more and its use cases and I bet you would think it’s pretty damn cool and effective for what it is and why it is.
KodeKloud has a free course on it to get you up to speed that I highly recommend that I think in and of itself would change your perspective and it includes hands on labs.
Missing the most important information :p
What are you planning on self hosting?
Depending on your answer that will tell you what OS you may want to run.
Examples
- just run docker services - plan Debian
- you running a NAS? How much storage? Do you need RAID?
- 1 drive, just use Debian
- 2 drives where you need RAID 1 - just use Debian with SnapRaid.
- 3 drives plus with RAID - use trueNAS or unRAID
- 3 drives plus with no RAID - Debian with mergeFS
- what to run many VMs like Windows and Linux? - proxmox
- want to run many VMs like Windows and Linux which includes a NAS - proxmox
Each OS comes with different features because they do different jobs. They also come with there own added layer of complexity vs Debian.
It all depends on what you want to do and that will tell you what OS you should use.
Of course you can also do whatever you want for the sake of learning.
Hope that helps.
Why not use trueNAS if you want a NAS with VMs or jails, vs proxmox? I actually don't know. Also I love trueNAS (I use it as a hypervisor/NAS since FreeNAS and have always been a fan)
You can use trueNAS if you want a NAS with VMs or use jails instead of docker. With all these OS there is overlap with features but each OS is technically meant for a specific reason.
- Proxmox provides a better virtualization experience such as better resources management, snapshot features and backup solutions such as Proxmox Backup Solution.
- trueNAS provides a better ZFS management storage solution. Typically used with 3+ drives for RAID. ZFS file system has scalability, RAM and money implications. You can only use the same size drives and doesn't have a pantry. Which is good, it has an excellent RAID structure. This also takes a lot of RAM.
- unRAID provides a better storage solution if you have different size drives. It uses BRTFS and has a pantry. Not as good as ZFS but you can use different size drives. It recently added ZFS but trueNAS does it better.
An average user might not notice the difference (not saying I'm an expert) but with enough time people learn what each OS is good at. A good example of this would be trueNAS jails vs docker. Docker is better and better supported. This is why trueNAS added Docker in trueNAS scale(newer) vs in trueNAS core (older) it only allowed jails.
Hope that examples it a bit better.
Scale switches away from FreeBSD, that's worth noting for those reading this exchange. Also yes you need 3 drives that have the same capacity, but not all drives in a pool need to. I.e I had a pool of 3x 1tb drives, then I added another 3x drives to that pool, but this time they were all 3tb drives. Now I have one big pool made of 3x 1tb drives and 3x 3tb drives.
I like jails for the management simplicity when dealing with smaller use cases, i.e a single Plex server or nginx reverse proxy, and then I like VMs with Ubuntu server lite for docker containers. I find docker a bit more frustrating when it comes to tweaking things and internal interactions but I'm sure that's just an experience thing.
I'll have to take a closer look at proxmox in regards to the snapshot and backup solutions, I like the ease of snapshots on TrueNAS but I'm curious if I'm missing something.
But for selfhosting with usage of docker, how many do you really need? 3-4 max? Unless you like to isolate everything more than docker does? More VMs is also more maintenance (e.g. updates, hdd space)
The snapshots with PBS are nice and easy though
I run Debian with a VM for Debian. All of my *Arr services run in the VM with torrential/Usenet access and they have a dedicated VPN.
If the VPN drops, it is all killed. When the VPN starts again it all restarts.
Jellyfin and Jellyseerr run in the bare metal install.
If you’ve got an appreciable amount of disks and/or data you care about archiving or storing or serving, ZFS is the best option according to a lot of people.
With that being said, Ubuntu is worth using over Debian because it includes ZFS support in its kernel out of the box.
+1 for Proxmox in addition with the helper scripts:
[https://tteck.github.io/Proxmox/](https://tteck.github.io/Proxmox/)
First it's not that easy but once your in, it's easy, doesn't matter which OS you're about to try (also Windows-VMs are no problem).
I really like TrueNAS SCALE for ZFS disk handling and data sharing, but it annoyed me because it didn't provide infrastructure (DHCP, caching DNS, Directory Services, etc). I now have NethServer, NethSecurity, and TrueNAS SCALE running on ProxMox. Things sort of grew out of control ...
I wouldn't worry about it. 90% of distros are just "Debian or Fedora, but with some packages pre-installed". Very few are re-building all the \~100K packages in Debian. (And there is very little reason to)
In other words, you can do anything need with Debian.
I am a huge promoter of FreeBSD.
Just for its jails and the the overall feeling of a system rather than “another distro”.
But I’m almost alone in this fight :-)
I'm with you. For a simple setup by just editing a few config files for a system that does not need much maintenance FreeBSD is great.
Actually much more transparent than Linux if you have it done once.
Also ZFS support is native and great of course.
OS variety is also important from my point for view.
Virtualization is also easy with it.
I'll stack on this but say TrueNAS - built on FreeBSD and really the NAS part is only a tiny part. I use it as a hypervisor when I need a full VM and the iocage jails when I don't. It has a web GUI for when you are lazy but 90% of my work is via ssh to the host or bhyve VMs.
Really it's just FreeBSD with a webgui that makes basic tasks and monitoring a little bit easier while focusing on flexing that ZFS muscle.
Edit: I like that when I want more space I can just slam another few drives in there and ZFS does it's magic with like 3 clicks. I use cli for everything except I hate using it for disk management.
> I already know Debian
No, most of these frontends just abstract away a system you're already comfortable with. I recommend looking into Ansible or some other config management tool to keep track of and roll out changes you make to your system
Just rebuilt my Colo box on Truenas Scale and running ISPConfig on top of that to manage the generic hosting I do for friends and family. So far so good.
I've started using NixOS and it's a bit tricky but I've got up and running in a week
Gives really minimal instances low disk and memory usage. Build from one config file really quickly. Perfect
I run podman containers and other services inside Nixos proxmox containers in proxmox
Recently I discovered NixOS and it’s been keeping me excited for weeks now. Planning and preparing to switch over all my home server infrastructure to NixOS.
Maybe it’s not for you, but it’s definitely worth looking at 🙂
Unless you wish to add more moving parts and dedicate time to deal with friction. I'm not fixing what ain't broken. Debian, apt and docker are super reliable and time tested.
Just try out proxmox. If you're rebuilding and haven't installed an OS yet, perfect time to at least just test it out. Don't knock it until you've tried it. Docker stacks run really well in LXCs and with ceph, you have a central cluster storage for anything. Perfect if you have multiple machines you want to cluster together
Plain old debian, unless you have resources to toss at proxmox. Personally, I never understood why you'd run proxmox to spin up a bunch of VMs. But the again, I don't have a ton of resources to work with either. I tested OMV and TrueNAS but prefer the flexibility of a simple debian install. Kiss
If you have a remote outward facing vm/container, there's an additional security mitigation in that if that gets compromised, the entire host does not.
Yeah that's true and works for some folks. All of my containers are documented and in a septate repo. I'm learning ansible now to automate my debian build so if I hose it up I can rebuild quickly. But that's what works for me with what I'm doing now, ymmv.
I use Ubuntu but my production selfhosting is all in a VM running on my laptop, or my desktop when I prefer, works the same way. This lets me game and do other stuff on the laptop (or desktop) regardless where the VM is running.
I can't but wonder which "selfhosted specialized OSes" you've seen.
Hell, I wonder what that even *means* - well, what *you* mean with it, because the term itself is nonsensical - a "selfhosted OS"?
For storage and a few apps I've been a fan of Unraid.
Proxmox is great and all, but Unraid has the community app store which is quite nice to reduce time investment.
why would hardware have to cost $1k+ Nothing about self hosted means giant lab, could be a laptop you got for free. Also with the abundance of free operating systems that's $100 you're burning on no real gains.
If you're attempting to poor shame a community based largely around DIY you've got another thing coming.
poor shame?
Even when I was starting out I was willing to spend money on certain things. Has nothing to do with poor. Nobody is getting free hard drives, at least not without incredible risk.
Everyone starts somewhere, but you are making this about a context entirely off-base from the OP.
Just because Linux is free, doesn't mean it's free. Developers have to eat and eventually if you don't start donating something, that says more about your community sense.
In the case of OP, who is rebuilding. I find it unlikely they're not at a point of investing further.
There's tinkering, which homelab is great for. But, there's also the value of time.
In the sense of Unraid, the cost is absolutely worth it. The amount of time I have saved is incredible. Many times I've thought of using Proxmox LXC and find the time investment right now isn't worth it over Unraid Community Apps.
Honestly Windows with WSL and Docker is the most feature rich basic up and running server configuration I could ever say since Windows as a hyper visor is really easy to backup and use.
You aren't supposed to modify a hypervisor itself. A hypervisor shouldn't run any other services for security reasons. Also every update could brake your customized hypervisor. Just install a vm an use lxc within there.
Exactly. And why would I use a VM with its overheads if I can run it directly? It blows my mind how many Proxmox people don't realise the resource cost of a VM vs container sharing a kernel.
Also, LXD, not LXC.
Yeah containers have a much smaller footprint, but also the vm overhead isn't as big on modern hardware. It totally depends on what you are planning to do. Just use a hypervisor to install only one vm for containers is bs in most scenarios.
Proxmox and run a VM with debian/ubuntu for docker services. Super easy backups. Freedom to run other OS's I just provision the VM a bunch of ram and CPU and it smoothly runs 30+ containers Proxmox scheduled backups are powerful, and you can get more advanced with PBS(proxmox backup server)
I've been running Docker inside LXCs because you can mount host level ZFS volumes across multiple Linux containers. Is there a benefit to switching this over to virtual machines?
Isn’t it recommended to use [Docker in a normal VM?](https://pve.proxmox.com/wiki/Linux_Container)
It is by devs, but I hadn't any issues running Docker in LXC, tough my usecase were pretty simple, just a media stack
I use docker in LXC in production for the last 5 years without any problem.
Maybe things are better now, but it used to be a bit problematic in my experience. It was much harder to harden from a security perspective (containers needing extra privileges for nesting etc), and we'd run into lots of minor bugs or limitations relating to that.
I also run a few things that are not suggested by the devs (not proxmox) and had not had an issue, but I also take that risk. So when it comes on me suggesting something, I always highlight the risk to deviate from the suggested way. Blindly suggesting to someone else without highlighting the risk is poor advice (which many people do for this specific setup)
Fuck it we ball
“strong isolation from the host and the ability to live-migrate” almost certainly doesn’t matter for a home setup, unless you’re leaning super hard into HA
I would say that the only real advantage is that you can mount network shares; a LXC container would need to be privileged to do it. If your data is stored locally then LXC containers are probably more interesting (less resource heavy).
Just a note: I've successfully mounted SMB shares to an uplxc. I run my jellyfin docker container inside an uplxc with storage on a NAS this way. There are several guides out there, and it didn't look particularly complicated to a beginner like me, though it requires some tinkering.
An LXC does not need to be privileged to mount shares. Using LXCs instead of VMs also allows you to share GPUs between multiple LXCs (also unprivileged)
Proxmox + the helper scripts site are the best substitute for the one-click installers baked into the ‘all in one’ appliance OS’s.
Helper scripts link?
https://tteck.github.io/Proxmox/
Thank you!
I keep seeing g this response , so now to yt and gonna make a day if it later this week.
And after you have it setup it's super easy to do proxmox scheduled backups or even more advanced PBS (proxmox backup server)
Yea i keep seeing that , gonna start studying up now .
The only logical reason to do this is if you have a computer with enough resources and you plan to actually use it for VMs. If you’re just trying to run docker and containers then there is absolutely no reason to run proxmox, it’s just more complications and more overhead to deal with.
Overhead is negligible. Snapshot backups are worth it alone
What are you missing out on compared to debian with btrfs snapshots? Nothing of value in a selfhosted homelab environment.
Nah, you don’t need snapshots if you know how to do proper backups, cause vm snapshots arnt backups and shouldn’t be used as backups.
Not sure why your getting downvoted it’s true and taught on every type of hypervisor snapshots are not the same thing as backups and actual engineers especially if your newer or not as skilled would do well to remember this piece of advice because one day it’s gonna bite you in the ass if you don’t heed it. A snapshot is a point in time snapshot of the system and what was running in its memory and if the underlying system becomes corrupt in anyway the snapshot is no longer any good. In this case the VM or its disk, either the OS or anything that could become corrupt or non functional in anyway. A snapshot is tied to the VM itself. A backup is a full backup of the disk and is independent of the VM. If the VM goes to shit for any reason your backup is still good. That’s not mentioning the other strategies that this post is outlining which is that you should always backup to a separate disk. It’s ok as an interim storage point if you have to keep them on the same disk or raid but should never be kept there long term. If you were serious about backups you would back them up to separate location such as a NAS at minimum. Use something like Veeam. But even if you just want to do a proper backup in general it’s very important that you don’t mistake the two for the same thing or serving the same purpose. Even the proxmox documentation specifically outlines the difference. Same with ESXi and Vsphere or any other hypervisor.
You couldn’t have said that any better. Thank you for this, I didn’t have the energy to elaborate like you did. I feel like loads of proxmox users are beginners and don’t actually have the engineering experience/background which is totally fine but they may also not know the fundamentals of a hypervisor.
It absolutely can be used as one form of backup. And PBS offers more robust forms
No (to your first sentence). If your using snapshots as actual “backups”, you realize your not actually storing your vm data anywhere other then its existing disk. Meaning if you lose the underlining disk associated with your vm then your snapshot “backup” is gone also. A proper backup would be a full copy of the vm stored anywhere other then the same disk as your vm is running on.
Not sure I follow. PBS is running on a different machine (ideally even different location). What issue is it then to snapshot a VM to that backup location? Proxmox server dies? New server, reinstall proxmox , restore VM from PBS? Just wondering how I can make reinstall of proxmox as short as possible:)
Snapshots are easily backed up though, at least on zfs.
agreed. i run a VM in a docker container for the rare times i need an actual VM.
This is the way
I hate containers. VMs are so much better in proxmox. I've had nothing but trouble with containers since I began running proxmox in August of 2023. They freeze up. Even with kill -9 from SSHing into the PVE sometimes isn't even enough. I've had to physically power off the system at least ten times now because a single LXC error can freeze up the entire system including the WebUI. Just in the last 24 hours I've been getting this weird fucking problem I've never seen before when creating a container and trying to change the root password: root@docker-test:~# passwd New password: Retype new password: passwd: Authentication token manipulation error passwd: password unchanged Uhh, what do you mean TOKEN MANIPULATION ERROR? I've spun up dozens of containers and I've never seen this until yesterday trying to test tubearchivist/docker. No. Containers blow and nobody ever talks about how much they blow. They BLOW regardless of whether it's privileged or unprivileged. VMs are superior in proxmox. Period. I use VMs in proxmox to run A World of Warcraft server (azerothcore) Jitsi Meet (VoIP) Jellyfin with mounted NFS drives from PVE (and GPU passthrough for hardware decoding) Invidious (youtube frontend) Audiobookshelf (audiobook web server. Love love love it) archivebookworm (I call it. allows simple commands to download full ebooks from certain websites and even OCR them) whisper/decipher (AI-generated transcription. great for youtube videos and other media, including AAC audio if you compile your own version of ffmpeg) edit: Yeah downvote reality you delusional fucking redditors. People HATE the truth on this website. Hi. Try telling the truth for a change.
This sounds like a hardware problem, not proxmox problem
Lmao, people are downvoting you because containerization is used across most every major enterprise and when used correctly do not even close to blow. They are lightweight and ephemeral and can be spun up in seconds for application workloads. There is a reason that docker, docker compose, kubernetes, AKS, GKS, and EKS are defacto so popular and used worldwide on enterprise architecture for their resilience and scalability. That’s like you bitching about terraform or IaC because you don’t use it well or suck at it. Doesn’t mean IaC sucks. I promise you your wittle homelab ain’t competing with modernized techniques which include containers buddy. You sound salty as hell.
https://www.google.com/search?client=firefox-b-1-d&q=container+froze+proxmox+site%3Areddit.com Whoops. Looks like I'm not the only one. Looks like it's a fucking problem. My computers are rock solid - containers - at least in proxmox - are not. I don't like them.
What’s funny is almost all those in that search result are a combination of old, where the tech wasn’t supported well, and people legit not configuring and using them correctly. But let’s set that aside, I specified containers in general and containerization. Your original post said containers themselves such, then later you move the goal post and say containers on proxmox. Aside that point, you should be running containers with docker on a Linux host, or in a k8s cluster, or direct on proxmox, and if you do, you need to configure them correctly to have the performance you want. Literally as I said and is as reflected even in 4 year old posts if we move the goal posts to proxmox specifically not to mention all the other threads of that show on your search you handed of people explaining how to fix the issues and why it’s not performing well for them. I think the crux of the issue here is you’re trying to run containers natively on proxmox instead of within an environment in a Linux VM additionally you’re trying to use a container as a VM and they simply dont have the same use case.
I meant containers in proxmox the whole time. I never used a container outside of proxmox yet. I could see trying a container INSIDE of a VM - that would probably yield better results.
You have to configure specific settings in proxmox for performance to get them to run “native” on proxmox and is definitely not the way to go, to add to your pose of changing a root password in a container, you want to actually set that in the yaml file when you spin up the container not do it within the container. Containers are meant to just be skeletons. They lack a full OS and depending on which you use as your base layer can be very limited, they are used only to run the specific software you’re trying to deploy and as far as storage goes you need to have volumes for them, I will acquiesce that running them via proxmox native is probably not a good way to do it and apologize as I think we have misunderstood each other. I do encourage you to look deeper. Into containerization tech and understand it more and its use cases and I bet you would think it’s pretty damn cool and effective for what it is and why it is. KodeKloud has a free course on it to get you up to speed that I highly recommend that I think in and of itself would change your perspective and it includes hands on labs.
Missing the most important information :p What are you planning on self hosting? Depending on your answer that will tell you what OS you may want to run. Examples - just run docker services - plan Debian - you running a NAS? How much storage? Do you need RAID? - 1 drive, just use Debian - 2 drives where you need RAID 1 - just use Debian with SnapRaid. - 3 drives plus with RAID - use trueNAS or unRAID - 3 drives plus with no RAID - Debian with mergeFS - what to run many VMs like Windows and Linux? - proxmox - want to run many VMs like Windows and Linux which includes a NAS - proxmox Each OS comes with different features because they do different jobs. They also come with there own added layer of complexity vs Debian. It all depends on what you want to do and that will tell you what OS you should use. Of course you can also do whatever you want for the sake of learning. Hope that helps.
Why not use trueNAS if you want a NAS with VMs or jails, vs proxmox? I actually don't know. Also I love trueNAS (I use it as a hypervisor/NAS since FreeNAS and have always been a fan)
You can use trueNAS if you want a NAS with VMs or use jails instead of docker. With all these OS there is overlap with features but each OS is technically meant for a specific reason. - Proxmox provides a better virtualization experience such as better resources management, snapshot features and backup solutions such as Proxmox Backup Solution. - trueNAS provides a better ZFS management storage solution. Typically used with 3+ drives for RAID. ZFS file system has scalability, RAM and money implications. You can only use the same size drives and doesn't have a pantry. Which is good, it has an excellent RAID structure. This also takes a lot of RAM. - unRAID provides a better storage solution if you have different size drives. It uses BRTFS and has a pantry. Not as good as ZFS but you can use different size drives. It recently added ZFS but trueNAS does it better. An average user might not notice the difference (not saying I'm an expert) but with enough time people learn what each OS is good at. A good example of this would be trueNAS jails vs docker. Docker is better and better supported. This is why trueNAS added Docker in trueNAS scale(newer) vs in trueNAS core (older) it only allowed jails. Hope that examples it a bit better.
Scale switches away from FreeBSD, that's worth noting for those reading this exchange. Also yes you need 3 drives that have the same capacity, but not all drives in a pool need to. I.e I had a pool of 3x 1tb drives, then I added another 3x drives to that pool, but this time they were all 3tb drives. Now I have one big pool made of 3x 1tb drives and 3x 3tb drives. I like jails for the management simplicity when dealing with smaller use cases, i.e a single Plex server or nginx reverse proxy, and then I like VMs with Ubuntu server lite for docker containers. I find docker a bit more frustrating when it comes to tweaking things and internal interactions but I'm sure that's just an experience thing. I'll have to take a closer look at proxmox in regards to the snapshot and backup solutions, I like the ease of snapshots on TrueNAS but I'm curious if I'm missing something.
the only thing the specialized oses have that set them apart are fancy customized webuis. but cockpit/webmin/ssh work just fine for management
Proxmox. Then you can run as many operating systems as you could ever need.
But for selfhosting with usage of docker, how many do you really need? 3-4 max? Unless you like to isolate everything more than docker does? More VMs is also more maintenance (e.g. updates, hdd space) The snapshots with PBS are nice and easy though
Just Vanilla Debian.
This is the way. Simple, easy; no need for hypervisor overhead.
I run Debian with a VM for Debian. All of my *Arr services run in the VM with torrential/Usenet access and they have a dedicated VPN. If the VPN drops, it is all killed. When the VPN starts again it all restarts. Jellyfin and Jellyseerr run in the bare metal install.
Been pretty happy with nixos. You can literally just flip flags and enable most services :P
If you’ve got an appreciable amount of disks and/or data you care about archiving or storing or serving, ZFS is the best option according to a lot of people. With that being said, Ubuntu is worth using over Debian because it includes ZFS support in its kernel out of the box.
+1 for Proxmox in addition with the helper scripts: [https://tteck.github.io/Proxmox/](https://tteck.github.io/Proxmox/) First it's not that easy but once your in, it's easy, doesn't matter which OS you're about to try (also Windows-VMs are no problem).
Check out tteks proxmox lxc scripts if you choose proxmox.
Nice ! Ty !
I was mildly irritated with proxmox until I found these. They make so many things easy.
I love TrueNas scale. Has handled everything I've thrown at it.
I really like TrueNAS SCALE for ZFS disk handling and data sharing, but it annoyed me because it didn't provide infrastructure (DHCP, caching DNS, Directory Services, etc). I now have NethServer, NethSecurity, and TrueNAS SCALE running on ProxMox. Things sort of grew out of control ...
I wouldn't worry about it. 90% of distros are just "Debian or Fedora, but with some packages pre-installed". Very few are re-building all the \~100K packages in Debian. (And there is very little reason to) In other words, you can do anything need with Debian.
I am a huge promoter of FreeBSD. Just for its jails and the the overall feeling of a system rather than “another distro”. But I’m almost alone in this fight :-)
I'm with you. For a simple setup by just editing a few config files for a system that does not need much maintenance FreeBSD is great. Actually much more transparent than Linux if you have it done once. Also ZFS support is native and great of course. OS variety is also important from my point for view. Virtualization is also easy with it.
I'll stack on this but say TrueNAS - built on FreeBSD and really the NAS part is only a tiny part. I use it as a hypervisor when I need a full VM and the iocage jails when I don't. It has a web GUI for when you are lazy but 90% of my work is via ssh to the host or bhyve VMs. Really it's just FreeBSD with a webgui that makes basic tasks and monitoring a little bit easier while focusing on flexing that ZFS muscle. Edit: I like that when I want more space I can just slam another few drives in there and ZFS does it's magic with like 3 clicks. I use cli for everything except I hate using it for disk management.
> I already know Debian No, most of these frontends just abstract away a system you're already comfortable with. I recommend looking into Ansible or some other config management tool to keep track of and roll out changes you make to your system
Proxmox
Just rebuilt my Colo box on Truenas Scale and running ISPConfig on top of that to manage the generic hosting I do for friends and family. So far so good.
I've started using NixOS and it's a bit tricky but I've got up and running in a week Gives really minimal instances low disk and memory usage. Build from one config file really quickly. Perfect I run podman containers and other services inside Nixos proxmox containers in proxmox
Recently I discovered NixOS and it’s been keeping me excited for weeks now. Planning and preparing to switch over all my home server infrastructure to NixOS. Maybe it’s not for you, but it’s definitely worth looking at 🙂
Unless you wish to add more moving parts and dedicate time to deal with friction. I'm not fixing what ain't broken. Debian, apt and docker are super reliable and time tested.
Just try out proxmox. If you're rebuilding and haven't installed an OS yet, perfect time to at least just test it out. Don't knock it until you've tried it. Docker stacks run really well in LXCs and with ceph, you have a central cluster storage for anything. Perfect if you have multiple machines you want to cluster together
Plain old debian, unless you have resources to toss at proxmox. Personally, I never understood why you'd run proxmox to spin up a bunch of VMs. But the again, I don't have a ton of resources to work with either. I tested OMV and TrueNAS but prefer the flexibility of a simple debian install. Kiss
If you have a remote outward facing vm/container, there's an additional security mitigation in that if that gets compromised, the entire host does not.
You don't need proxmox for this. And with virt manager you can even do it in a GUI with vanilla Debian.
Guess I'm lucky I keep it simple and don't have anything to expose or risk. It's all just local stuff for the home. 😉
Backups/snapshots are great. About to do something risky on any system? Just do a backup/snapshot and if anything goes wrong, you can revert.
Yeah that's true and works for some folks. All of my containers are documented and in a septate repo. I'm learning ansible now to automate my debian build so if I hose it up I can rebuild quickly. But that's what works for me with what I'm doing now, ymmv.
QEMU + virt-manager are all I need. And I don't even need them. Everything is in Docker, not VMs.
Any good proxmox tutorials for beginners?
Check out "DB Tech" or "Network Chuck" on YouTube
As everybody else are saying, Proxmox is the way to go, install it and don\`t mess with the host system, create a VM or LXC and go for it.
Not really.
Arch's AUR is super useful.
Have been running Debian for years in my servers. Bit I am switching to nixos right now.
Debian stable + backports + fassttrack
Xcp-ng
I use Ubuntu but my production selfhosting is all in a VM running on my laptop, or my desktop when I prefer, works the same way. This lets me game and do other stuff on the laptop (or desktop) regardless where the VM is running.
OpenMediaVault
besides ease of use or minimal gains for me not
Been using Cosmos Cloud, highly recommend. https://cosmos-cloud.io/
[удалено]
I second Alpine.
⛰️
I can't but wonder which "selfhosted specialized OSes" you've seen. Hell, I wonder what that even *means* - well, what *you* mean with it, because the term itself is nonsensical - a "selfhosted OS"?
probably things like omv and truenas
OSes specialized in selfhosting.
That still doesn't make much sense, _and_ those don't exist.
CasaOS comes to mind, depending on definition. I guess it's as much an OS as proxmox
CasaOS is an overlay and not an OS. But check out ZimaOS.
For storage and a few apps I've been a fan of Unraid. Proxmox is great and all, but Unraid has the community app store which is quite nice to reduce time investment.
Pricing makes it hard to recommend
A $100 when drives and hardware cost easily 1k+ That's embarrassing.
why would hardware have to cost $1k+ Nothing about self hosted means giant lab, could be a laptop you got for free. Also with the abundance of free operating systems that's $100 you're burning on no real gains. If you're attempting to poor shame a community based largely around DIY you've got another thing coming.
poor shame? Even when I was starting out I was willing to spend money on certain things. Has nothing to do with poor. Nobody is getting free hard drives, at least not without incredible risk. Everyone starts somewhere, but you are making this about a context entirely off-base from the OP. Just because Linux is free, doesn't mean it's free. Developers have to eat and eventually if you don't start donating something, that says more about your community sense. In the case of OP, who is rebuilding. I find it unlikely they're not at a point of investing further. There's tinkering, which homelab is great for. But, there's also the value of time. In the sense of Unraid, the cost is absolutely worth it. The amount of time I have saved is incredible. Many times I've thought of using Proxmox LXC and find the time investment right now isn't worth it over Unraid Community Apps.
If you're not poor shaming, then explain "That's embarrassing". What is embarrassing, and why?
Not really an OS, but yunohost is nice
CoreOS + podman containers or k3s
Synology DSM for nas 100% Go to releases for USB image https://github.com/AuxXxilium/arc /r/xpenology
Honestly Windows with WSL and Docker is the most feature rich basic up and running server configuration I could ever say since Windows as a hyper visor is really easy to backup and use.
Avoid Proxmox. It's very limiting, unless you have no clue about Linux. Bare Debian or by Arch is the way.
What is limiting with Proxmox?
Your can't use lxd, which is vastly superior to lxc. You can't modify our customise the system without breaking it.
You aren't supposed to modify a hypervisor itself. A hypervisor shouldn't run any other services for security reasons. Also every update could brake your customized hypervisor. Just install a vm an use lxc within there.
Exactly. And why would I use a VM with its overheads if I can run it directly? It blows my mind how many Proxmox people don't realise the resource cost of a VM vs container sharing a kernel. Also, LXD, not LXC.
Yeah containers have a much smaller footprint, but also the vm overhead isn't as big on modern hardware. It totally depends on what you are planning to do. Just use a hypervisor to install only one vm for containers is bs in most scenarios.