Proxmox QDevice on Raspberry Pi

26 February, 2021 at 15:00 -0400

Proxmox (the Proxmox Virtual Environment, PVE) is a great open-source virtualization platform. Built on top of the Debian Linux distribution (with a modified Ubuntu kernel), PVE leverages KVM for virtual machines and LXC for containers.

In smaller clusters, as would be typical in a home-lab environment, the presence of an odd number of nodes (3 or more) is not guaranteed. As such, PVE provides a small component, called QDevice, that augments the number of voting members to enhance the cluster resilience.

As the QDevice only provides a voter for quorum and no virtualization support, it can be installed on Raspberry Pi devices.

If the cluster already has an odd number of nodes (N), adding the QDevice will allocate it N-1 votes, therefore making it a single point of failure. Thus, if you already have an odd number of nodes, don’t add the QDevice.

Installation

Raspberry Pi

First thing is to ensure that the Raspberry Pi has a stable IP address. This is often done through a manually configured static IP address or through a DHCP reservation. PVE expects hosts to be always reachable at the same IP address, so ensuring that the Pi has a stable one is important.

First, root login must be allowed. This can be insecure, so a long and complex secure password should be used. Once configured, PVE will use key-based authentication. Thus, as an alternative, the PVE ssh key can be added to the /root/.ssh/authorized_keys file and key-based authentication allowed. In any case, once the system is configured, login can be changed to prohibit-password.

$ # Make sure to configure a secure password for the root user.
$ sudo passwd

$ # Allow root login and restart the SSH daemon.
$ sudo sed -i 's/^.*PermitRootLogin.*$/PermitRootLogin yes/' /etc/ssh/sshd_config
$ sudo systemctl restart ssh

Next, the qnetd daemon must be installed on the Pi. Because raspbian is a debian-based distribution, there is no need to manually start the service. corosync-qnetd will be automatically started and will run as an unprivileged user.

$ sudo apt-get install -y corosync-qnetd

The installation will generate an NSS database with a CA and some certificates that will be used for communication with the cluster nodes.

If you have a firewall in place, ensure that connections to port 5403 are allowed. If using ufw, which the Raspberry Pi Foundation recommends, the command to allow inbound connections is as follow.

$ sudo ufw allow 5403/tcp

Cluster Nodes

Next, the QDevice daemon must be installed on all the existing PVE cluster nodes.

# apt-get install corosync-qdevice -y

Finally, on one of the PVE cluster node, initiate the QDevice setup.

# pvecm qdevice setup <raspberry_pi_ip_address>

Verification

Once the pvecm command returns, the status of the QDevice can be checked by through pvecm status. The last block of output should look similar to the following.

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW IP.ADDR.FIRST.NODE (local)
0x00000002          1    A,V,NMW IP.ADDR.SECOND.NODE
0x00000000          1            Qdevice

Clean-up

Finally, to re-secure the root ssh login on the Pi, the configuration can be changed to disallow password logins.

$ # Allow root login and restart the SSH daemon.
$ sudo sed -i 's/^.*PermitRootLogin.*$/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
$ sudo systemctl restart ssh