A Step-by-Step Guide to Safely Shutting Down a Proxmox VE Cluster with Ceph
Managing a Proxmox VE cluster with a hyper-converged Ceph storage backend provides incredible power and resilience. However, this sophisticated setup requires a methodical approach when it comes to planned maintenance or a full shutdown. Simply turning off the nodes can lead to data integrity issues, prolonged recovery times, and a state of panic for any system administrator.
This guide provides the definitive, safe procedure for shutting down and restarting your entire Proxmox cluster with Ceph, ensuring your data remains safe and your cluster comes back online smoothly.
Why a Proper Shutdown is Critical
Ceph is designed for high availability. When a node (OSD) goes offline, Ceph’s self-healing mechanisms kick in, attempting to rebalance data and maintain the required number of replicas. During a planned full-cluster shutdown, this “healing” process is undesirable. It creates unnecessary network traffic and I/O load as the cluster tries to “fix” a problem that isn’t really there.
By following a specific shutdown sequence, you inform the cluster of your intentions, preventing this counterproductive behavior and ensuring a clean, predictable shutdown and startup.
The Safe Shutdown Procedure
Follow these steps precisely to power down your entire infrastructure without causing issues with Ceph or the Proxmox cluster quorum.
Step 1: Power Down All Guests (VMs and Containers)
Before touching the cluster itself, you must cleanly shut down all running virtual machines and containers. This ensures that applications close properly and no in-memory data is lost.
You can do this manually through the Proxmox web interface or via the command line on any node. If you use High Availability (HA), you may need to disable it for the guests first to prevent the cluster from trying to restart them on other nodes.
Step 2: Prepare the Ceph Cluster for Shutdown
This is the most crucial step. You need to tell Ceph to not rebalance or mark OSDs (Object Storage Daemons) as “out” when nodes begin to power off.
Connect to any Proxmox node via SSH and run the following command:
ceph osd set noout
This noout flag is essential. It temporarily prevents Ceph from re-replicating data when an OSD becomes unavailable. You can verify the flag is active by running ceph -s or ceph health detail. You should see a health warning indicating the noout flag is set. This is expected and confirms you are ready for the next step.
Step 3: Shut Down the Proxmox Nodes
With the guests off and Ceph prepared, you can now shut down the physical Proxmox nodes. It is best practice to shut them down one at a time. While the order is less critical after setting the noout flag, a good habit is to shut down non-primary nodes first, saving the cluster leader for last.
You can identify the leader (master) node by running pvecm status and looking for the node designated as the “Master”.
From each node’s command line, execute:
shutdown -h now
Wait for each node to power down completely before moving to the next.
The Safe Startup Procedure
Restarting the cluster is essentially the reverse process, with a strong emphasis on verifying cluster health at each stage.
Step 1: Power On All Proxmox Nodes
Begin by powering on the physical server hardware. It is highly recommended to start the node you shut down last (the cluster leader) first. This helps the cluster re-establish quorum more quickly and reliably.
After the first node is up, proceed to power on the remaining nodes. You can power them on simultaneously or one by one.
Step 2: Verify Proxmox and Ceph Cluster Status
Once all nodes have booted, give them a few minutes to initialize their services. Then, connect to any node via SSH and verify the health of both the Proxmox cluster and the Ceph storage.
First, check the Proxmox cluster quorum:
pvecm status
Ensure all nodes are listed and that the quorum is established. Next, check the Ceph cluster status:
ceph -s
At this point, you will likely see a HEALTH_WARN status because the noout flag is still set. All OSDs should be up and “in.” If everything looks correct, you can proceed.
Step 3: Unset the Ceph noout Flag
Now that the cluster is fully operational, you can remove the shutdown flag to allow Ceph to resume its normal self-healing functions.
Run the following command to allow data rebalancing and recovery if needed:
ceph osd unset noout
After a few moments, run ceph -s again. The cluster health should return to HEALTH_OK. If any inconsistencies occurred during the shutdown, Ceph will now begin to heal itself safely.
Step 4: Start Your Guests (VMs and Containers)
With the infrastructure healthy, you can now start your virtual machines and containers. If you have HA enabled, the cluster will begin starting the designated guests automatically. Otherwise, you can start them manually through the web interface or command line.
Actionable Security and Stability Tips
- Invest in a UPS: An Uninterruptible Power Supply (UPS) is non-negotiable for any cluster environment. It protects against unexpected power outages, which can cause the exact messy scenario this guide helps you avoid.
- Regular Health Checks: Make it a habit to run
pvecm statusandceph -speriodically to catch any issues before they become critical. - Document Your Configuration: Keep a record of your node IP addresses, the cluster leader, and any custom configurations. This documentation is invaluable during recovery scenarios.
By treating your Proxmox and Ceph cluster with the care it deserves, you ensure maximum uptime, data integrity, and peace of mind. Following this structured shutdown and startup procedure is a cornerstone of professional cluster administration.
Source: https://nolabnoparty.com/procedura-di-spegnimento-di-un-cluster-proxmox-con-ceph/
