Building Your Kubernetes Cluster on Proxmox: The Definitive Guide
Harnessing the power of container orchestration with Kubernetes is a game-changer for deploying and managing applications. But running it on a public cloud can be costly and may not offer the level of control you need. For those seeking a powerful, cost-effective, and self-hosted alternative, combining Kubernetes with Proxmox VE is an exceptional solution.
This guide provides a practical, real-world approach to setting up a robust Kubernetes cluster on Proxmox, perfect for everything from a home lab to a production environment.
Why Proxmox for Your Kubernetes Cluster?
Before diving into the setup, it’s important to understand why Proxmox is such a compelling choice for hosting Kubernetes.
Proxmox VE is a powerful, open-source virtualization platform that combines two virtualization technologies under a single, easy-to-use web interface: KVM for full virtualization and LXC for lightweight OS-level containers. This duality provides incredible flexibility for building your cluster.
Key advantages include:
- Cost-Effectiveness: Proxmox is completely free and open-source, eliminating expensive licensing fees associated with other hypervisors.
- Flexibility: You can run your Kubernetes nodes as either full virtual machines (VMs) or lightweight Linux Containers (LXC), allowing you to tailor your setup for performance or isolation.
- Centralized Management: The Proxmox web GUI provides a single pane of glass for managing your entire infrastructure, from nodes and storage to networking and backups.
- Robust Features: Built-in support for clustering, high availability, and diverse storage solutions like Ceph and ZFS makes it a production-ready platform.
The Core Decision: KVM Virtual Machines vs. LXC Containers
When creating your Kubernetes nodes in Proxmox, your first major decision is whether to use KVM VMs or LXC containers. Each has distinct advantages.
Full Virtualization with KVM VMs
KVM (Kernel-based Virtual Machine) creates fully isolated virtual machines, each with its own dedicated kernel.
- Pros:
- Maximum Security and Isolation: Because each VM has its own kernel, a compromise in one node is completely isolated from the others and the Proxmox host. This is the most secure option.
- Kernel Flexibility: You can run any Linux distribution with any kernel version, independent of the Proxmox host.
- Cons:
- Higher Overhead: KVM VMs consume more RAM and CPU resources due to the overhead of running a full operating system and kernel for each node.
Choose KVM if your priority is security, workload isolation, or if you need to run a specific kernel version for your nodes.
Lightweight Virtualization with LXC Containers
LXC (Linux Containers) are an OS-level virtualization method. Containers share the kernel of the Proxmox host but have their own isolated user space.
- Pros:
- Exceptional Performance: LXC containers are incredibly lightweight and fast, with minimal overhead. They start up almost instantly and consume far fewer resources than VMs.
- Higher Density: You can run significantly more LXC-based Kubernetes nodes on the same hardware compared to KVM-based nodes.
- Cons:
- Shared Kernel Risk: All containers share the host’s kernel. A kernel-level vulnerability or crash on the host could potentially impact all containers.
- Less Isolation: While user space is isolated, it doesn’t offer the same hardened security boundary as a full VM.
Choose LXC if your priority is performance, resource efficiency, and high density, especially in a trusted environment like a development lab.
A Step-by-Step Guide to Deployment
Here is a high-level overview of the steps required to get your cluster running. For this guide, we’ll focus on a KVM-based setup for its robust isolation.
1. Prepare a VM Template
Instead of manually creating each node, it’s highly efficient to create a master VM template.
- Install a minimal cloud-ready Linux distribution (like Ubuntu Server or Debian).
- Run all system updates and install necessary packages like
containerd,kubeadm,kubelet, andkubectl. - Clean the image (
cloud-init clean) to prepare it for cloning. - Convert this VM into a Proxmox template.
2. Clone Nodes from the Template
Using your new template, clone the required number of VMs for your cluster. A typical starting point is one control-plane (master) node and at least two worker nodes. Assign each a static IP address for network stability.
3. Initialize the Control-Plane Node
SSH into your designated control-plane VM and initialize the Kubernetes cluster using the kubeadm command.
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
This command will set up all the necessary control-plane components. At the end of the process, it will output a kubeadm join command. Copy this entire command and save it—you will need it for your worker nodes.
4. Configure kubectl
On the control-plane node, run the commands provided by kubeadm to configure kubectl access for your user account. This allows you to interact with your new cluster.
5. Deploy a Container Network Interface (CNI)
A brand new Kubernetes cluster cannot schedule pods until a network overlay is installed. This CNI plugin allows pods to communicate with each other across different nodes. Calico and Flannel are popular choices.
For example, to install Calico:
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
6. Join the Worker Nodes
Now, SSH into each of your worker nodes. Run the kubeadm join command you saved from Step 3 on each worker. This command will securely connect the worker to the control plane, making it available to receive workloads.
After a few moments, you can run kubectl get nodes on your control-plane node to see all your nodes in a Ready state.
Essential Production Considerations
Moving beyond a basic setup requires thinking about storage, networking, and security.
Persistent Storage:
Containers are ephemeral, but your data needs to be persistent. Proxmox offers excellent integrated storage options.
- Ceph: A highly scalable and resilient distributed storage solution built directly into Proxmox. It’s excellent for providing block storage for Kubernetes Persistent Volumes.
- ZFS: Offers robust local storage with snapshot capabilities.
- NFS: A simple and effective way to provide shared storage across all your nodes.
You can use a storage provisioner like Longhorn or the open-source CephFS provisioner to dynamically create Persistent Volumes for your applications.
Load Balancing:
In a bare-metal environment like Proxmox, you don’t have a cloud provider’s load balancer. To expose your services (like a web server) to the outside world, you need a solution like MetalLB. MetalLB integrates with your local network to provide a reliable load-balancing service for your cluster.
Security Best Practices:
- Network Segmentation: Create a dedicated VLAN for your Kubernetes cluster traffic in your network infrastructure and configure a dedicated virtual bridge for it in Proxmox. This isolates cluster communication.
- Leverage the Proxmox Firewall: Use the built-in Proxmox firewall to control traffic to and from your Kubernetes nodes at the hypervisor level.
- Apply Kubernetes RBAC: Use Kubernetes Role-Based Access Control (RBAC) to enforce the principle of least privilege for users and services operating within the cluster.
- Keep Everything Updated: Regularly update the Proxmox host, the guest operating systems on your nodes, and your Kubernetes version to protect against known vulnerabilities.
By combining the enterprise-grade features of Proxmox with the unparalleled orchestration power of Kubernetes, you can build a highly resilient, scalable, and cost-effective platform to run your modern applications.
Source: https://www.horizoniq.com/blog/kubernetes-on-proxmox/
