Proxmox Cluster Not Ready - No Quorum (500): Corosync Fix

cluster not ready - no quorum? (500) means this Proxmox node can no longer see a majority of Corosync votes, so /etc/pve turns read-only and guest starts are refused. Run pvecm status: if Total votes is below Quorum, bring back the missing nodes or the Corosync link. Use pvecm expected 1 only when the other nodes are truly down.

The symptom arrives in several disguises. The web UI returns cluster not ready - no quorum? (500) when you try to start a guest. Editing anything under /etc/pve fails with a read-only filesystem error even though the disk is healthy and has space. A node reboots itself for no visible reason while the hardware logs stay clean. All three are the same event: the node is no longer part of a quorate cluster, and Proxmox is refusing to let it act as though it were.

This is a diagnostic walk-through built from the Proxmox VE administration guide, the corosync manual pages and public Proxmox forum threads, not a report of an incident handled here.

What quorum protects

Cluster membership is handled by corosync, and the cluster requires a majority of votes to be quorate. Every node carries one vote by default. Without a majority, the guest and storage configuration held in /etc/pve would be editable on both sides of a network partition, and two nodes could each decide they own the same virtual machine and start it against the same disks.

Proxmox prevents that at the filesystem layer. /etc/pve is not an ordinary directory. It is the Proxmox Cluster file system, described in the documentation as “a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync”. It holds qemu-server/ VM configs, lxc/ container configs, corosync.conf and storage.cfg, and it switches to read-only the moment a node loses quorum.

So the read-only error is not a bug and not a disk fault. It is the safety interlock doing exactly what it exists to do, and fixing the storage will not help. The thing to repair is cluster membership.

What is Corosync in Proxmox?

Corosync is the cluster engine Proxmox VE uses for membership and quorum. It tracks which nodes can reach each other, counts their votes, and carries the real-time replication of /etc/pve. It needs little bandwidth but steady low latency, and every node must reach every other node on UDP ports 5405 to 5412.

The upstream Corosync project lists among its features “a quorum system that notifies applications when quorum is achieved or lost”, and Proxmox’s cluster filesystem is one of those applications. The project’s latest stable release was 3.1.10 when checked in October 2026. The port range is from the cluster manager chapter.

Two services carry it on every node: corosync for membership, links and vote counting (corosync_votequorum), and pve-cluster for the pmxcfs filesystem behind /etc/pve. They fail differently, so check both.

How to check Corosync status in Proxmox

Run pvecm status for the quorum verdict, corosync-cfgtool -s for per-link connectivity, and systemctl status corosync pve-cluster to confirm both services are running. A healthy node shows Quorate: Yes, Total votes equal to Expected votes, and every peer connected. Anything else tells you which node or link dropped out.

Run the full set on the affected node before changing anything.

systemctl status corosync pve-cluster
pvecm status
corosync-cfgtool -s
journalctl -u corosync -u pve-cluster --since "-1h"
ha-manager status
chronyc tracking

Reading pvecm status

pvecm status is the one that answers the question. It reports the total votes expected, the votes currently present, and whether the partition is quorate. Compare “Expected votes” against “Total votes”: if expected is 3 and total is 1, two nodes are unreachable, and the problem is the network or those nodes rather than this one.

Quorum is the majority threshold. This output, posted to the Proxmox forum from an isolated member of a four-node cluster, shows the failure:

Quorate:          No
Expected votes:   4
Highest expected: 4
Total votes:      1
Quorum:           3 Activity blocked
Flags:

One vote against a threshold of three, so corosync appends Activity blocked and Quorate disappears from the flags. On a QDevice cluster the flags also show Qdevice; if the QDevice is listed as NA (not alive), the documentation says to make sure TCP port 5403 on the qnetd server is reachable.

Reading corosync-cfgtool -s

corosync-cfgtool -s shows each configured link and whether it is connected. On a cluster with a single link, a link down here and a quorum failure are the same fault. On a multi-link cluster, one link down and the cluster still quorate is the design working. The corosync-cfgtool man page gives this three-node example:

LINK ID 0
addr = 192.168.100.80
status:
nodeid 1: localhost
nodeid 2: connected
nodeid 3: connected

localhost is the node you ran it on, and each peer should read connected. A peer that is not connected on any link is the node missing from the vote. corosync-cfgtool -n adds the MTU per link, the quickest way to confirm the jumbo-frame mismatch described under cause four.

What the journal shows

The journal for corosync and pve-cluster is where token loss and retransmit messages appear, and the timestamps usually line up with something else on the network: a backup window, a migration, a switch reconverging after a spanning-tree change.

Fixing “cluster not ready - no quorum? (500)” step by step

First rule out a cold start: after a full power loss, pve-guests “waits for quorum” before starting onboot guests, so the first node up reports no quorum until enough peers join. Otherwise:

1. Confirm it is quorum. pvecm status shows Quorate: No and Activity blocked.
2. Find the missing members. Compare the membership table with the nodelist in /etc/pve/corosync.conf, then run systemctl status corosync pve-cluster and journalctl -b -u corosync on each missing node’s console.
3. Test the path. Run corosync-cfgtool -s on both ends, ping the link addresses, and confirm UDP 5405 to 5412 passes and the MTU matches.
4. Check the configs agree. /etc/pve/corosync.conf and /etc/corosync/corosync.conf should match, with the same config_version everywhere. In a thread titled “pvecm expected 1 does nothing”, Proxmox staff pointed at this drift after manual edits without a corosync restart.
5. Restart and recheck. Run systemctl restart corosync on the corrected node and rerun pvecm status. With a majority back, /etc/pve is writable again.

If a majority cannot come back, use the single-node override below.

Cause one: the node count cannot produce a majority

A two-node cluster has no majority once either node drops. The survivor holds one vote out of two expected, which is not a majority, so it goes read-only. This is arithmetic rather than a fault, and no amount of network tuning fixes it.

The documented answer is a third vote. Another full node is the usual starting point, and the high-availability chapter recommends at least three nodes for reliable quorum. Where a third node is not practical, Corosync’s external vote support provides one. Each Proxmox node runs corosync-qdevice; an independent always-on machine runs corosync-qnetd to arbitrate the extra vote. It adds no guest or storage capacity.

The same problem returns at every even node count. Four nodes need three votes to stay quorate, so a four-node cluster survives one failure, exactly like a three-node cluster, while costing an extra machine. Adding a QDevice to an even-numbered cluster is what restores the odd vote total.

Two constraints are worth noting in advance: the QDevice has to be removed before you delete a cluster node, and it must be reachable from every node, so putting it behind the same single switch as the cluster removes most of the benefit.

Cause two: corosync is sharing a link with something loud

This is the most common cause on clusters that were working yesterday. Corosync has an unusual traffic profile. The cluster manager chapter describes it as needing “consistent low latency but not a lot of bandwidth” and recommends a dedicated 1 Gbit NIC precisely to “avoid situations where other services can use up all the available bandwidth”.

The requirement it has to meet is strict: “the Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably”. A backup job, a live migration or Ceph recovery traffic saturating a shared link will push latency past that, corosync declares token loss, and the cluster partitions while every cable is still plugged in and every switch port shows a link.

The tell is correlation. If quorum failures cluster around a scheduled backup window or begin when a guest was migrated, the link is contended and the fix is separation rather than a longer timeout.

Cause three: the timeouts are being hit for real

Corosync’s defaults are documented in corosync.conf(5). The token timeout defaults to 3000 milliseconds, which is the time allowed before token loss is declared and a processor is considered failed. For clusters of three or more nodes, token_coefficient adds 650 milliseconds per node beyond two, so the effective token timeout grows with cluster size. The consensus timeout, which governs how long to wait for agreement before starting a new membership round, defaults to 3600 milliseconds.

Since Proxmox VE 6.0, corosync 3 uses Kronosnet, which “only supports regular UDP unicast”. Multicast is no longer involved, so multicast-related switch configuration is not the answer on any current release, and IGMP snooping is no longer the classic culprit it once was.

Raising timeouts is occasionally correct, for example across a WAN link that genuinely cannot meet the latency target. It is usually the wrong first move, because it converts a fast, visible failure into a slow one and delays HA recovery by the same amount. Fix the path first.

Cause four: something about the node’s identity changed

Corosync binds to configured addresses and identifies nodes by name and node ID. Changes that break it are easy to make by accident:

• A DHCP lease change or a renumbered management network moves a node away from the address in corosync.conf.
• A hostname change without updating /etc/hosts and the cluster configuration on every node.
• An MTU mismatch after enabling jumbo frames on some interfaces but not all, which lets small packets through and drops large ones, producing an intermittent failure that looks like flapping hardware.
• A VLAN or firewall change that blocks corosync’s UDP ports 5405 to 5412 between nodes, or TCP port 5403 to a QDevice.

Editing corosync.conf has its own trap: it lives in /etc/pve, which is read-only exactly when you most want to fix it. The documented approach is to edit the copy under /etc/pve/corosync.conf while the cluster is quorate and let it replicate, and to always increment the config_version field so that other nodes accept the change. Editing the local /etc/corosync/corosync.conf directly is a recovery measure, not a normal workflow.

Cause five: the clocks disagree

Time skew across nodes produces confusing symptoms: certificate validation problems, log timelines that cannot be correlated, and replication and backup schedules firing at the wrong moment. chronyc tracking on each node takes seconds to check and removes a whole class of misleading evidence. Nodes should agree on a common NTP source.

Getting a single node writable again

When a majority genuinely cannot be restored and you need to bring guests up on a surviving node, pvecm expected 1 lowers the expected vote count so the node becomes quorate by itself. It is documented for exactly this situation, and it is the right tool during a real outage.

It is also the command that creates split-brain if you get it wrong. Lowering expected votes on a node whose peers are actually alive and simply unreachable means two partitions can each start the same guest against the same storage. Before running it, confirm the other nodes are genuinely down rather than isolated, and preferably keep them powered off until membership is restored. The setting is not persistent across a corosync restart, which is a safety property rather than an inconvenience.

The documentation also states a limit: “This is not enough if corosync cannot start anymore.” In that case the repair goes into the local /etc/corosync/corosync.conf, which has to stay identical on every node to avoid split-brain.

Why HA nodes reboot themselves

If the cluster runs HA services, quorum loss does not just make things read-only. The HA stack fences failed nodes with a watchdog timer that the local resource manager and cluster resource manager reset during normal operation. The documentation is direct about what happens when they stop: a node that loses quorum “cannot reset the watchdog”, and if it has active services, or the LRM or CRM process is not scheduled or is killed, “this will trigger a reboot after the watchdog has timed out (this happens after 60 seconds)”.

That is why a hardware investigation into “random reboots” on an HA cluster so often finds nothing. The reboot is deliberate, and it exists so that a node which might still be running a guest cannot keep writing to shared storage while another node starts the same guest elsewhere. The event to investigate is the quorum loss, not the reboot.

Preventing the next one

• Give corosync its own link, and add a second on separate physical infrastructure. Kronosnet supports up to eight links, and the documentation warns that “a single link backed by a bond can be problematic in certain failure scenarios”.
• Keep the vote total odd. Three nodes, or an even count plus a QDevice.
• Keep backup, migration and Ceph traffic off the primary cluster link. The network layout and redundant-link planning are in planning a Proxmox VE cluster.
• Verify latency stays under 5 ms under load, not just when the cluster is idle.
• Watch pvecm status and the corosync journal for retransmit messages, which appear well before a cluster actually partitions.
• Alert when corosync-quorumtool -s exits 2 (not quorate) or corosync-cfgtool -s -b exits 1 (a faulty link); the second catches a dead redundant link while the cluster is still quorate.

The Proxmox VE HA cluster and Ceph sizer shows how the vote arithmetic changes as node count changes. Check storage availability and capacity on surviving nodes separately. If the cluster is receiving existing VMware guests, the VMware vSphere to Proxmox VE migration guide covers import limits, guest configuration, and cutover checks.

FAQ

what ports does corosync use in proxmox

Corosync in Proxmox VE uses UDP ports 5405 to 5412, and every node must be able to reach every other node on them. A QDevice adds one more: the qnetd server listens on port 5403, which must be reachable over TCP from each node. A firewall rule blocking either costs votes while every cable looks healthy.

how do i restart corosync in proxmox

Run systemctl restart corosync on the node, then check systemctl status corosync and journalctl -b -u corosync. Changes to /etc/pve/corosync.conf that a running corosync can integrate take effect as soon as the file changes, so a restart is only needed when they cannot. A restart also clears any pvecm expected override.

why do my vms not start after a proxmox power outage

Guests wait for quorum. After a full outage, the pve-guests service on each node holds back every guest with the onboot flag until the cluster is quorate again, so a node that boots faster than its peers sits idle and returns the no-quorum error. Bring enough nodes online for a majority and the guests follow.