Proxmox No Quorum Errors: Corosync Troubleshooting

The symptom arrives in several disguises. The web UI returns cluster not ready - no quorum? (500) when you try to start a guest. Editing anything under /etc/pve fails with a read-only filesystem error even though the disk is healthy and has space. A node reboots itself for no visible reason while the hardware logs stay clean. All three are the same event: the node is no longer part of a quorate cluster, and Proxmox is refusing to let it act as though it were.

This is a diagnostic walk-through built from the Proxmox VE administration guide and the corosync manual pages, not a report of an incident handled here.

What quorum protects

Cluster membership is handled by corosync, and the cluster requires a majority of votes to be quorate. Every node carries one vote by default. Without a majority, the guest and storage configuration held in /etc/pve would be editable on both sides of a network partition, and two nodes could each decide they own the same virtual machine and start it against the same disks.

Proxmox prevents that at the filesystem layer. /etc/pve is not an ordinary directory. It is the Proxmox Cluster file system, described in the documentation as “a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync”. It holds qemu-server/ VM configs, lxc/ container configs, corosync.conf and storage.cfg, and it switches to read-only the moment a node loses quorum.

So the read-only error is not a bug and not a disk fault. It is the safety interlock doing exactly what it exists to do, and fixing the storage will not help. The thing to repair is cluster membership.

The first five commands

Run these on the affected node before changing anything.

pvecm status
corosync-cfgtool -s
journalctl -u corosync -u pve-cluster --since "-1h"
ha-manager status
chronyc tracking

pvecm status is the one that answers the question. It reports the total votes expected, the votes currently present, and whether the partition is quorate. Compare “Expected votes” against “Total votes”: if expected is 3 and total is 1, two nodes are unreachable, and the problem is the network or those nodes rather than this one.

corosync-cfgtool -s shows each configured link and whether it is connected. On a cluster with a single link, a link down here and a quorum failure are the same fault. On a multi-link cluster, one link down and the cluster still quorate is the design working.

The journal for corosync and pve-cluster is where token loss and retransmit messages appear, and the timestamps usually line up with something else on the network: a backup window, a migration, a switch reconverging after a spanning-tree change.

Cause one: the node count cannot produce a majority

A two-node cluster has no majority once either node drops. The survivor holds one vote out of two expected, which is not a majority, so it goes read-only. This is arithmetic rather than a fault, and no amount of network tuning fixes it.

The documented answer is a third vote. Another full node is the cleanest option, and the high-availability chapter asks for “at least three cluster nodes (to get reliable quorum)” before HA is used at all. Where a third node is not practical, corosync’s external vote support provides one: the documentation notes that “for smaller 2-node clusters, the QDevice can be used to provide a 3rd vote”. The QDevice daemon runs on separate infrastructure outside the cluster, which can be a small always-on machine, and exists only to break ties.

The same problem returns at every even node count. Four nodes need three votes to stay quorate, so a four-node cluster survives one failure, exactly like a three-node cluster, while costing an extra machine. Adding a QDevice to an even-numbered cluster is what restores the odd vote total.

Two constraints are worth noting in advance: the QDevice has to be removed before you delete a cluster node, and it must be reachable from every node, so putting it behind the same single switch as the cluster removes most of the benefit.

Cause two: corosync is sharing a link with something loud

This is the most common cause on clusters that were working yesterday. Corosync has an unusual traffic profile. The cluster manager chapter describes it as needing “consistent low latency but not a lot of bandwidth” and recommends a dedicated 1 Gbit NIC precisely to “avoid situations where other services can use up all the available bandwidth”.

The requirement it has to meet is strict: “the Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably”. A backup job, a live migration or Ceph recovery traffic saturating a shared link will push latency past that, corosync declares token loss, and the cluster partitions while every cable is still plugged in and every switch port shows a link.

The tell is correlation. If quorum failures cluster around a scheduled backup window or begin when a guest was migrated, the link is contended and the fix is separation rather than a longer timeout.

Cause three: the timeouts are being hit for real

Corosync’s defaults are documented in corosync.conf(5). The token timeout defaults to 3000 milliseconds, which is the time allowed before token loss is declared and a processor is considered failed. For clusters of three or more nodes, token_coefficient adds 650 milliseconds per node beyond two, so the effective token timeout grows with cluster size. The consensus timeout, which governs how long to wait for agreement before starting a new membership round, defaults to 3600 milliseconds.

Since Proxmox VE 6.0, corosync 3 uses Kronosnet, which “only supports regular UDP unicast”. Multicast is no longer involved, so multicast-related switch configuration is not the answer on any current release, and IGMP snooping is no longer the classic culprit it once was.

Raising timeouts is occasionally correct, for example across a WAN link that genuinely cannot meet the latency target. It is usually the wrong first move, because it converts a fast, visible failure into a slow one and delays HA recovery by the same amount. Fix the path first.

Cause four: something about the node’s identity changed

Corosync binds to configured addresses and identifies nodes by name and node ID. Changes that break it are easy to make by accident:

• A DHCP lease change or a renumbered management network moves a node away from the address in corosync.conf.
• A hostname change without updating /etc/hosts and the cluster configuration on every node.
• An MTU mismatch after enabling jumbo frames on some interfaces but not all, which lets small packets through and drops large ones, producing an intermittent failure that looks like flapping hardware.
• A VLAN or firewall change that blocks the corosync ports between subnets.

Editing corosync.conf has its own trap: it lives in /etc/pve, which is read-only exactly when you most want to fix it. The documented approach is to edit the copy under /etc/pve/corosync.conf while the cluster is quorate and let it replicate, and to always increment the config_version field so that other nodes accept the change. Editing the local /etc/corosync/corosync.conf directly is a recovery measure, not a normal workflow.

Cause five: the clocks disagree

Time skew across nodes produces confusing symptoms: certificate validation problems, log timelines that cannot be correlated, and replication and backup schedules firing at the wrong moment. chronyc tracking on each node takes seconds to check and removes a whole class of misleading evidence. Nodes should agree on a common NTP source.

Getting a single node writable again

When a majority genuinely cannot be restored and you need to bring guests up on a surviving node, pvecm expected 1 lowers the expected vote count so the node becomes quorate by itself. It is documented for exactly this situation, and it is the right tool during a real outage.

It is also the command that creates split-brain if you get it wrong. Lowering expected votes on a node whose peers are actually alive and simply unreachable means two partitions can each start the same guest against the same storage. Before running it, confirm the other nodes are genuinely down rather than isolated, and preferably keep them powered off until membership is restored. The setting is not persistent across a corosync restart, which is a safety property rather than an inconvenience.

Why HA nodes reboot themselves

If the cluster runs HA services, quorum loss does not just make things read-only. The HA stack fences failed nodes with a watchdog timer that the local resource manager and cluster resource manager reset during normal operation. The documentation is direct about what happens when they stop: a node that loses quorum “cannot reset the watchdog”, and if it has active services, or the LRM or CRM process is not scheduled or is killed, “this will trigger a reboot after the watchdog has timed out (this happens after 60 seconds)”.

That is why a hardware investigation into “random reboots” on an HA cluster so often finds nothing. The reboot is deliberate, and it exists so that a node which might still be running a guest cannot keep writing to shared storage while another node starts the same guest elsewhere. The event to investigate is the quorum loss, not the reboot.

Preventing the next one

• Give corosync its own link, and add a second on separate physical infrastructure. Kronosnet supports up to eight links, and the documentation warns that “a single link backed by a bond can be problematic in certain failure scenarios”.
• Keep the vote total odd. Three nodes, or an even count plus a QDevice.
• Keep backup, migration and Ceph traffic off the cluster link. The full separation argument is in planning a Proxmox VE cluster, and the port counts each role implies are in Proxmox VE hardware requirements.
• Verify latency stays under 5 ms under load, not just when the cluster is idle.
• Watch pvecm status and the corosync journal for retransmit messages, which appear well before a cluster actually partitions.

The Proxmox VE HA cluster and Ceph sizer shows how the vote arithmetic changes as node count changes, which is a quick way to check whether a planned cluster can actually survive the failure you expect it to. If you are still choosing a platform, the high-availability and placement differences against vSphere are covered in Proxmox VE vs VMware ESXi.