Skip to content

PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence [8.4] - #318

Draft
jaideepkarande wants to merge 3 commits into
percona:4.x-8.0from
jaideepkarande:PXC-5300-8.4-stateless
Draft

jaideepkarande wants to merge 3 commits into
percona:4.x-8.0from
jaideepkarande:PXC-5300-8.4-stateless

Conversation

@jaideepkarande

@jaideepkarande jaideepkarande commented Nov 13, 2026 •

Copy link
Copy Markdown

PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence

https://perconadev.atlassian.net/browse/PXC-5300

Problem:
With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO
hangs forever on all nodes. It completes only once garbd leaves the cluster.

Description:
NBO end is effective only when every member of the current view has sent
an NBO end message (Certification checks that the current view is a subset
of the NBO end senders). The view included garbd, which never applies write
sets and therefore never sends NBO end, so the condition was never met.
Nothing in the configuration change (CC) record identified a member as an
arbitrator: garbd's name is configurable and an empty incoming address is
shared with data nodes bound to loopback.

Resolution:
Carry each member's GCS stateless flag in the CC record and leave members
that are stateless and have no incoming address (garbd always is both,
whatever its --name) out of the NBO end barrier. The flag is copied from
the GCS member table when the CC is built, so every node, including IST
joiners, decides on the same snapshot of the view.

  • The flags travel in an optional trailer after the member array; the CC
    version is unchanged and older readers ignore the trailer.
  • Upgrade gate: upgraded nodes set a new state message bit (ignored by
    older nodes). Arbitrators are left out only when every other member has
    it, so a mixed-version cluster keeps the old behaviour on every node
    instead of diverging.
  • A data node with gcs.stateless=yes and no incoming address would be
    treated as an arbitrator; it logs a warning at startup.

Test case added.

Note: the follow-up commit 461d5e5 replaces the startup warning above with a startup error: a data node with gcs.stateless=yes and no incoming address refuses to start, so "stateless with no incoming address" can only be garbd. Commit 159805f adds certification tests adapted from a community-contributed patch.

PXC PR: percona/percona-xtradb-cluster (8.4), linked below once open.

Test plan

  • Galera builds; gcs_tests all pass.
  • Without the fix: certification_check view_skips_arbitrator fails ("data nodes have ended NBO, arbitrator must be ignored"); MTR galera_nbo_garbd shows the ALTER still running after 60 s.
  • With the fix: galera_check certification 65/65 pass (defaults_check fails before and after, from PXC-5208, unrelated).
  • With the fix: MTR galera_nbo_garbd (garbd --name my_arbitrator), pxc_5300_stateless_data_node, galera_garbd (2 suites) pass, both encryption combinations.
  • Jenkins (Debug, full MTR): https://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/ (no new failures vs baseline; see Jenkins comment)

🤖 Generated with Claude Code

jaideepkarande and others added 3 commits October 7, 2026 15:06
… presence

https://perconadev.atlassian.net/browse/PXC-5300

Problem:
With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO
hangs forever on all nodes. It completes only once garbd leaves the cluster.

Description:
NBO end is effective only when every member of the current view has sent
an NBO end message (Certification checks that the current view is a subset
of the NBO end senders). The view included garbd, which never applies write
sets and therefore never sends NBO end, so the condition was never met.
Nothing in the configuration change (CC) record identified a member as an
arbitrator: garbd's name is configurable and an empty incoming address is
shared with data nodes bound to loopback.

Resolution:
Carry each member's GCS stateless flag in the CC record and leave members
that are stateless and have no incoming address (garbd always is both,
whatever its --name) out of the NBO end barrier. The flag is copied from
the GCS member table when the CC is built, so every node, including IST
joiners, decides on the same snapshot of the view.

- The flags travel in an optional trailer after the member array; the CC
  version is unchanged and older readers ignore the trailer.
- Upgrade gate: upgraded nodes set a new state message bit (ignored by
  older nodes). Arbitrators are left out only when every other member has
  it, so a mixed-version cluster keeps the old behaviour on every node
  instead of diverging.
- A data node with gcs.stateless=yes and no incoming address would be
  treated as an arbitrator; it logs a warning at startup.

Test case added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300

Problem:
The NBO end barrier leaves out members that are stateless and have no
incoming address, which is how garbd joins. A data node configured the same
way (gcs.stateless=yes and no incoming address, e.g. bound to loopback) is
also left out, so an NBO could end before that node finished the DDL. The
node was only warned at startup.

Description:
Other members cannot tell such a data node from garbd: both are stateless
with an empty incoming address, and names are configurable. Only the data
node itself knows it is not an arbitrator.

Resolution:
The data node refuses to start with that configuration: the provider
constructor throws, galera_init() fails and mysqld aborts before joining
the cluster. "Stateless with no incoming address" can then only be garbd,
whatever its name.

Test case added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300

Problem:
The View unit tests check which members the NBO end barrier waits for,
but not that an NBO actually ends through Certification when garbd is in
the view.

Description:
Tests adapted from a community-contributed patch for this bug.

Resolution:
- test_certification_nbo_arbitrator: runs NBO begin/end write sets through
  Certification with a view of two data nodes and garbd (stateless, no
  incoming address), passed through CC serialization. A duplicate NBO end
  from one node does not end the NBO; the second data node's end does,
  without garbd. Controls: with garbd not stateless, or with a data node
  missing the upgrade bit, garbd's end is still required.
- test_view_preserves_public_membership: the view given to the server
  still lists every member, garbd included.
- test_view_membership_changes: a data node leaving, a second garbd
  joining, and a donor data node that is still waited for.
- run_wsinfo() takes an optional CC to set the certification view.

Test case added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jaideepkarande

Copy link
Copy Markdown
Author

Compatibility & upgrade

Verdict: safe for rolling upgrade. Nodes never decide differently in a mixed-version cluster. The fix turns on at the first view change after the last data node is upgraded. garbd does not need to be upgraded.

CC record format (gcs/src/gcs_act_cchange.cpp)

  • Optional trailer after the member array, before the checksum: 'F', version byte 1, then one flags byte per member (0x01 stateless, 0x02 arb_nbo).
  • The CC version is unchanged, so old readers never see an "Unsupported CC action version" error.
  • Old readers ignore the trailer, and the checksum covers it.
  • A record from an old node has no trailer and is read as "no flags". The old record padding is zeros, so it can't be mistaken for a trailer. Covered by gcs_act_cchange_test no_member_flags_trailer.

State message (gcs/src/gcs_state_msg.hpp)

  • New PXC-only bit GCS_STATE_FPXC_ARB_NBO (0x80). Old nodes ignore unknown bits.

Upgrade gate (galera/src/galera_view.cpp)

Arbitrators are skipped only if every non-arbitrator member has arb_nbo_. Otherwise every node waits for all members, the same as before the fix.

View Old data nodes New data nodes Result
all data nodes new, garbd new or old — skip garbd fixed
some data nodes old wait for all gate off, wait for all hangs as today while garbd is present (consistent)
no garbd wait for all nothing to skip unchanged
  • Old garbd: it always runs with gcs.stateless=yes (garb/garb_config.cpp:180) and an empty incoming address (garb/garb_gcs.cpp:19-23), so it is recognized whatever its version or --name.
  • IST from an old donor: those records have no trailer, so every node waits for all members, matching what the cluster did at the time.
  • Downgrade: IST from a new donor to an old joiner could replay an NBO that the cluster ended without garbd. Use SST to downgrade or to add an older node.

Behaviour change

A data node with gcs.stateless=yes and no incoming address now refuses to start (galera/src/replicator_smm.cpp:293-305). Error text:

gcs.stateless is set and the node has no address for incoming client connections, so the cluster would treat it as an arbitrator and would not wait for it to finish NBO operations. Unset gcs.stateless or set wsrep_node_incoming_address.

gcs.stateless cannot be changed at runtime (gcs_group_param_set() returns ENOTSUP, gcs/src/gcs_group.cpp:2442), so the check cannot be skipped after startup.

Scope

@jaideepkarande

Copy link
Copy Markdown
Author

Tests

Unit tests added

gcs/src/unit_tests/gcs_act_cchange_test.cpp

  • Stateless/arb_nbo flags survive a write/read round trip.
  • no_member_flags_trailer: a record without a trailer (old writer) reads as "no flags".

galera/tests/certification_check.cpp (all #ifdef PXC)

Test Checks
test_view_skips_arbitrator Two data nodes and a stateless garbd with no address (renamed my_arbitrator): the NBO end set of the data nodes is enough
test_view_keeps_data_nodes A stateless data node with an address, and a non-stateless node named garb with no address: both are waited for
test_view_mixed_version One data node without arb_nbo: garbd is waited for (gate off)
test_view_all_arbitrators Every member is an arbitrator: fall back to waiting for all
test_certification_nbo_arbitrator * NBO begin/end write sets through Certification with a view passed through CC serialization. A duplicate NBO end from node 1 does not end the NBO; node 2's end does, without garbd. Controls: with garbd not stateless, or with a data node lacking the upgrade bit, garbd's end is still required.
test_view_preserves_public_membership * The view given to the server still has all 3 members
test_view_membership_changes * A data node leaving, a second garbd joining, and a data node in DONOR state that is still waited for

* Adapted from a community-contributed patch for this bug (commit 159805f).

Results

Run Before fix (galera 13ff9ed) With fix (159805f)
galera_check certification certification_check.cpp:1524:F:view_skips_arbitrator: data nodes have ended NBO, arbitrator must be ignored Checks: 65, Failures: 0, Errors: 0
gcs_tests — Total test failed: 0
  • The certification_nbo case has 2 checks and view_skips_arbitrator has 6, which confirms the new tests ran.
  • defaults_check fails both before and after the fix with Provider has extra parameter: repl.force_sst_after_inconsistency = no. It comes from PXC-5208 and is unrelated.

MTR (in the PXC PR)

  • galera_3nodes_nbo.galera_nbo_garbd: garbd started with --name my_arbitrator, then an NBO ALTER runs.
    • Before the fix: NBO DDL completed with garbd in the cluster: 0. The ALTER stayed in closing tables until garbd was killed.
    • With the fix: 1.
  • galera.pxc_5300_stateless_data_node: startup is refused when the node is stateless with no address, and allowed with an address.
  • Candidate run (enc_off/enc_on): galera.pxc_5300_stateless_data_node, galera.galera_garbd, galera_3nodes.galera_garbd, galera_3nodes_nbo.galera_nbo_garbd all pass: 9/9 including shutdown_report.
  • Earlier, on the same Galera rule with the upgrade gate: the galera_3nodes_nbo suite passed 33/33, and the IST/garbd tests that aren't disabled passed.
  • Before the startup error existed: a stateless data node with no address was skipped (alter_ended 0 → 1). The startup error closes that gap.

Jenkins

https://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/ (Debug, full MTR) is running. Results will be posted here and on the PXC PR.

@jaideepkarande

Copy link
Copy Markdown
Author

PXC PR: percona/percona-xtradb-cluster#2410. It has the submodule bump, the MTR tests, and comments on the root cause, design decisions and the community patch.

@jaideepkarande

Copy link
Copy Markdown
Author

Jenkins

https://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/ ran Debug, full MTR, on PXC with this Galera head (159805f7).

  • 16736 tests: 10894 passed, 6 failed.
  • None of the failures are new. All six (5× shutdown_report and galera_sst_allowed_methods) also fail in baseline runs without this change.
  • The PXC-5300 tests (galera_nbo_garbd, pxc_5300_stateless_data_node) and all garbd tests pass.

Full triage: percona/percona-xtradb-cluster#2410.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant