Repository navigation
PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence [8.4] - #318
jaideepkarande wants to merge 3 commits into
Conversation
… presence https://perconadev.atlassian.net/browse/PXC-5300 Problem: With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO hangs forever on all nodes. It completes only once garbd leaves the cluster. Description: NBO end is effective only when every member of the current view has sent an NBO end message (Certification checks that the current view is a subset of the NBO end senders). The view included garbd, which never applies write sets and therefore never sends NBO end, so the condition was never met. Nothing in the configuration change (CC) record identified a member as an arbitrator: garbd's name is configurable and an empty incoming address is shared with data nodes bound to loopback. Resolution: Carry each member's GCS stateless flag in the CC record and leave members that are stateless and have no incoming address (garbd always is both, whatever its --name) out of the NBO end barrier. The flag is copied from the GCS member table when the CC is built, so every node, including IST joiners, decides on the same snapshot of the view. - The flags travel in an optional trailer after the member array; the CC version is unchanged and older readers ignore the trailer. - Upgrade gate: upgraded nodes set a new state message bit (ignored by older nodes). Arbitrators are left out only when every other member has it, so a mixed-version cluster keeps the old behaviour on every node instead of diverging. - A data node with gcs.stateless=yes and no incoming address would be treated as an arbitrator; it logs a warning at startup. Test case added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300 Problem: The NBO end barrier leaves out members that are stateless and have no incoming address, which is how garbd joins. A data node configured the same way (gcs.stateless=yes and no incoming address, e.g. bound to loopback) is also left out, so an NBO could end before that node finished the DDL. The node was only warned at startup. Description: Other members cannot tell such a data node from garbd: both are stateless with an empty incoming address, and names are configurable. Only the data node itself knows it is not an arbitrator. Resolution: The data node refuses to start with that configuration: the provider constructor throws, galera_init() fails and mysqld aborts before joining the cluster. "Stateless with no incoming address" can then only be garbd, whatever its name. Test case added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300 Problem: The View unit tests check which members the NBO end barrier waits for, but not that an NBO actually ends through Certification when garbd is in the view. Description: Tests adapted from a community-contributed patch for this bug. Resolution: - test_certification_nbo_arbitrator: runs NBO begin/end write sets through Certification with a view of two data nodes and garbd (stateless, no incoming address), passed through CC serialization. A duplicate NBO end from one node does not end the NBO; the second data node's end does, without garbd. Controls: with garbd not stateless, or with a data node missing the upgrade bit, garbd's end is still required. - test_view_preserves_public_membership: the view given to the server still lists every member, garbd included. - test_view_membership_changes: a data node leaving, a second garbd joining, and a donor data node that is still waited for. - run_wsinfo() takes an optional CC to set the certification view. Test case added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compatibility & upgradeVerdict: safe for rolling upgrade. Nodes never decide differently in a mixed-version cluster. The fix turns on at the first view change after the last data node is upgraded. garbd does not need to be upgraded. CC record format (
|
| View | Old data nodes | New data nodes | Result |
|---|---|---|---|
| all data nodes new, garbd new or old | — | skip garbd | fixed |
| some data nodes old | wait for all | gate off, wait for all | hangs as today while garbd is present (consistent) |
| no garbd | wait for all | nothing to skip | unchanged |
- Old garbd: it always runs with
gcs.stateless=yes(garb/garb_config.cpp:180) and an empty incoming address (garb/garb_gcs.cpp:19-23), so it is recognized whatever its version or--name. - IST from an old donor: those records have no trailer, so every node waits for all members, matching what the cluster did at the time.
- Downgrade: IST from a new donor to an old joiner could replay an NBO that the cluster ended without garbd. Use SST to downgrade or to add an older node.
Behaviour change
A data node with gcs.stateless=yes and no incoming address now refuses to start (galera/src/replicator_smm.cpp:293-305). Error text:
gcs.stateless is set and the node has no address for incoming client connections, so the cluster would treat it as an arbitrator and would not wait for it to finish NBO operations. Unset gcs.stateless or set wsrep_node_incoming_address.
gcs.stateless cannot be changed at runtime (gcs_group_param_set() returns ENOTSUP, gcs/src/gcs_group.cpp:2442), so the check cannot be skipped after startup.
Scope
- All changes are under
#ifdef PXC. - No wsrep API or replication protocol version change.
galera::View(filtered) is used only by the NBO end barrier (certification.cpp:~912). The view the server receives still lists every member.- The trailer and bit
0x80are PXC-only. They should be coordinated with upstream (NBO phase two hangs indefinitely when garbd is present in the Primary View mariadb-corporation/galera#694) to avoid future collisions.
TestsUnit tests added
* Adapted from a community-contributed patch for this bug (commit 159805f). Results
MTR (in the PXC PR)
Jenkinshttps://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/ (Debug, full MTR) is running. Results will be posted here and on the PXC PR. |
|
PXC PR: percona/percona-xtradb-cluster#2410. It has the submodule bump, the MTR tests, and comments on the root cause, design decisions and the community patch. |
Jenkinshttps://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/ ran Debug, full MTR, on PXC with this Galera head (
Full triage: percona/percona-xtradb-cluster#2410. |
PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence
https://perconadev.atlassian.net/browse/PXC-5300
Problem:
With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO
hangs forever on all nodes. It completes only once garbd leaves the cluster.
Description:
NBO end is effective only when every member of the current view has sent
an NBO end message (Certification checks that the current view is a subset
of the NBO end senders). The view included garbd, which never applies write
sets and therefore never sends NBO end, so the condition was never met.
Nothing in the configuration change (CC) record identified a member as an
arbitrator: garbd's name is configurable and an empty incoming address is
shared with data nodes bound to loopback.
Resolution:
Carry each member's GCS stateless flag in the CC record and leave members
that are stateless and have no incoming address (garbd always is both,
whatever its --name) out of the NBO end barrier. The flag is copied from
the GCS member table when the CC is built, so every node, including IST
joiners, decides on the same snapshot of the view.
version is unchanged and older readers ignore the trailer.
older nodes). Arbitrators are left out only when every other member has
it, so a mixed-version cluster keeps the old behaviour on every node
instead of diverging.
treated as an arbitrator; it logs a warning at startup.
Test case added.
PXC PR: percona/percona-xtradb-cluster (8.4), linked below once open.
Test plan
gcs_testsall pass.certification_checkview_skips_arbitratorfails ("data nodes have ended NBO, arbitrator must be ignored"); MTRgalera_nbo_garbdshows the ALTER still running after 60 s.galera_checkcertification 65/65 pass (defaults_checkfails before and after, from PXC-5208, unrelated).galera_nbo_garbd(garbd--name my_arbitrator),pxc_5300_stateless_data_node,galera_garbd(2 suites) pass, both encryption combinations.🤖 Generated with Claude Code