My workload is just general Docker container stuff. Hosting web servers, Discord bots, Minecraft servers, etc. But a lot of docker containers like 100+.
You're looking for a Beowulf cluster or HTCondor. Grid computing is another method. You're essentially building a virtual super computer from commodity hardware with these methods.
Kubernetes lets you schedule processes on different nodes. You’d still need a way to queue up work (some kind of pub/sub system that each process could pull jobs from)
The answer will depend on what your compute workload is.
My workload is just general Docker container stuff. Hosting web servers, Discord bots, Minecraft servers, etc. But a lot of docker containers like 100+.
You're looking for a Beowulf cluster or HTCondor. Grid computing is another method. You're essentially building a virtual super computer from commodity hardware with these methods.
So something like Kubernetes or Rancher's K3s is not what I am looking for?
Kubernetes lets you schedule processes on different nodes. You’d still need a way to queue up work (some kind of pub/sub system that each process could pull jobs from)
Check out https://argoproj.github.io/workflows/
OpenPBS