Mastering Ceph: Your Guide to Building a Resilient Storage Cluster on Ubuntu 22.04
In today’s data-driven world, the need for scalable, reliable, and cost-effective storage solutions has never been greater. Enter Ceph, a powerful, open-source software-defined storage platform that provides unified object, block, and file storage in a single, distributed cluster. Its self-healing and self-managing capabilities make it a top choice for everything from cloud infrastructure to big data analytics.
This guide provides a comprehensive, step-by-step walkthrough for deploying a modern Ceph storage cluster on Ubuntu 22.04. We will leverage cephadm, the current standard for managing Ceph, to simplify the installation and administration process.
Prerequisites: Preparing Your Ubuntu 22.04 Environment
Before diving into the installation, ensuring your environment is properly configured is crucial for a smooth deployment. Your setup should consist of at least three nodes for a basic, resilient cluster, with one designated as the admin node.
- Operating System: All nodes must be running a fresh installation of Ubuntu 22.04 LTS.
- Networking: All nodes must have static IP addresses and be able to communicate with each other over the network. Passwordless SSH access from the admin node to all other cluster nodes is required for
cephadm. - Time Synchronization: Consistent time across all nodes is critical for Ceph’s stability. Configure a Network Time Protocol (NTP) service like
chronyon all nodes. - Storage Devices: Each storage node should have at least one available, unformatted disk or partition dedicated to Ceph for use as an Object Storage Daemon (OSD). Do not use the OS disk for Ceph storage.
- System User: Create a dedicated user with
sudoprivileges on all nodes. For consistency, use the same username and password across the cluster.
Step 1: Configuring the Admin Node
The admin node will be your primary point of control for the entire cluster. From here, you will run all cephadm and ceph commands.
First, create a dedicated user and set up passwordless SSH. On the admin node, generate an SSH key pair:
ssh-keygen
Next, copy this new public key to every node in the cluster (including the admin node itself) to enable seamless, secure communication.
ssh-copy-id your_user@<node_ip_address>
Finally, install the Ceph command-line tools. Add the Ceph repository and install cephadm:
curl --silent --remote-name --location https://download.ceph.com/keys/release.asc
gpg --no-default-keyring --keyring /usr/share/keyrings/ceph-archive-keyring.gpg --import release.asc
echo "deb [signed-by=/usr/share/keyrings/ceph-archive-keyring.gpg] https://download.ceph.com/debian-quincy/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt update
sudo apt install -y cephadm
Step 2: Bootstrapping Your Ceph Cluster
With the admin node prepared, you can now bootstrap the cluster. This critical step initializes the first Monitor (MON) and Manager (MGR) daemon, which are the brains of the Ceph cluster.
Run the bootstrap command from your admin node, replacing <mon_ip_address> with the IP of your first node:
sudo cephadm bootstrap --mon-ip <mon_ip_address>
This process can take several minutes. Upon successful completion, cephadm will provide you with:
- The Ceph Dashboard URL.
- The dashboard username (
admin) and a generated password. - A command to access the Ceph command-line interface (CLI).
Store the dashboard password securely. You can now check the status of your single-node cluster:
sudo ceph status
Step 3: Expanding the Cluster – Adding Nodes and OSDs
A single-node cluster isn’t very useful. Let’s add the other prepared nodes to expand its capacity and resilience.
Add New Hosts: Use the
ceph orch host addcommand for each additional node you prepared. This command tells the orchestrator about the new node and deploys the necessary agent.sudo ceph orch host add <new_node_hostname> <new_node_ip>Discover Storage Devices: Once the hosts are added, instruct Ceph to scan them for available storage devices.
sudo ceph orch device lsThis command will list all available, unformatted disks that can be used as OSDs.
Create the OSDs: The core of your storage capacity comes from Object Storage Daemons (OSDs), which are processes that manage data on a physical disk. You can add all available devices at once with a single command:
sudo ceph orch apply osd --all-available-devicesCeph will automatically partition, format, and activate each available disk as an OSD, incorporating it into the cluster. After this step, your cluster’s raw capacity will increase. You can monitor the progress with
sudo ceph status.
Step 4: Enabling Core Ceph Services
Your cluster is now operational, but to make it truly useful, you need to enable services like a distributed file system or an S3-compatible object gateway.
Setting up CephFS (File Storage): To create a POSIX-compliant file system that can be mounted by multiple clients, you need to create a metadata pool, a data pool, and the file system service itself.
sudo ceph fs volume create my_filesystemSetting up RADOS Gateway (Object Storage): To provide an S3 and Swift-compatible object storage interface, deploy the RADOS Gateway (RGW) service.
sudo ceph orch apply rgw my_gateway --placement="<num_daemons> <host1> <host2>"This command deploys the RGW service on the specified hosts, allowing applications to interact with your cluster using the popular S3 API.
Accessing the Ceph Dashboard
One of Ceph’s most powerful features is its web-based dashboard. It provides a real-time overview of the cluster’s health, performance metrics, OSD status, pool management, and configuration options.
Navigate to the URL provided during the bootstrap process (usually https://<your_admin_node_ip>:8443/) and log in with the admin user and the password you saved. The dashboard is an invaluable tool for both daily monitoring and advanced administration.
Essential Security Best Practices
Securing your storage cluster is paramount. Always follow these foundational security principles:
- Network Segmentation: Use a dedicated backend network for internal Ceph cluster traffic (replication, rebalancing) and a separate public network for client access. This isolates critical cluster operations from public-facing traffic.
- Firewall Configuration: Implement strict firewall rules on all nodes. Only open the ports necessary for Ceph daemons and client access, restricting all other traffic.
- Principle of Least Privilege: When creating users for block devices or object storage, grant them only the minimum permissions required for their specific pools.
- Regular Updates: Keep your Ubuntu and Ceph packages updated to protect against known vulnerabilities.
By following this guide, you have successfully deployed a robust and highly scalable Ceph storage cluster on Ubuntu 22.04, ready to meet the demands of modern data storage.
Source: https://kifarunix.com/install-and-setup-ceph-storage-cluster-on-ubuntu-2204/
