Skip to content

PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence [8.4] - #2410

Draft
jaideepkarande wants to merge 3 commits into
percona:8.4from
jaideepkarande:PXC-5300-8.4-stateless
Draft

jaideepkarande wants to merge 3 commits into
percona:8.4from
jaideepkarande:PXC-5300-8.4-stateless

Conversation

@jaideepkarande

@jaideepkarande jaideepkarande commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence

https://perconadev.atlassian.net/browse/PXC-5300

Problem:
With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO
hangs forever on all nodes. It completes only once garbd leaves the cluster.

Description:
NBO end is effective only when every member of the current view has sent
an NBO end message. The view included garbd, which never applies write sets
and therefore never sends NBO end, so the condition was never met.

Resolution:
Update the galera submodule with the fix: each member's GCS stateless flag
is carried in the configuration change record, and members that are
stateless and have no incoming address (garbd, whatever its --name) are
left out of the NBO end barrier. Arbitrators are left out only when all
other members run a version with this change, so mixed-version clusters
keep the old behaviour on every node.

Test case added.

Note: a data node with gcs.stateless=yes and no incoming address now refuses to start (it would look like garbd). Galera PR: percona/galera#318, which must be merged first; the submodule pointer here references its head.

Test plan

  • Debug build succeeds (mysqld + libgalera_smm).
  • Without the fix: galera_3nodes_nbo.galera_nbo_garbd records NBO DDL completed with garbd in the cluster: 0; the ALTER stays in closing tables until garbd is killed.
  • With the fix: galera_nbo_garbd (garbd --name my_arbitrator), galera.pxc_5300_stateless_data_node, galera.galera_garbd, galera_3nodes.galera_garbd pass, enc_off/enc_on (9/9 incl. shutdown_report).
  • With the fix: Galera unit tests pass (galera_check certification 65/65, gcs_tests).
  • Jenkins (Debug, full MTR): https://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/ (no new failures vs baseline; see Jenkins comment)

🤖 Generated with Claude Code

jaideepkarande and others added 3 commits October 7, 2026 15:06
… presence

https://perconadev.atlassian.net/browse/PXC-5300

Problem:
With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO
hangs forever on all nodes. It completes only once garbd leaves the cluster.

Description:
NBO end is effective only when every member of the current view has sent
an NBO end message. The view included garbd, which never applies write sets
and therefore never sends NBO end, so the condition was never met.

Resolution:
Update the galera submodule with the fix: each member's GCS stateless flag
is carried in the configuration change record, and members that are
stateless and have no incoming address (garbd, whatever its --name) are
left out of the NBO end barrier. Arbitrators are left out only when all
other members run a version with this change, so mixed-version clusters
keep the old behaviour on every node.

Test case added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300

Problem:
A data node with gcs.stateless=yes and no incoming address looks like the
arbitrator to the rest of the cluster and is left out of the NBO end
barrier, so an NBO could end before that node finished the DDL.

Description:
Only the data node itself can tell that it is not an arbitrator.

Resolution:
Update the galera submodule: such a data node refuses to start, before
joining the cluster. A stateless data node with an incoming address still
starts.

Test case added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300

Galera now has unit tests that run NBO begin/end write sets through
Certification with garbd in the view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jaideepkarande

Copy link
Copy Markdown
Contributor Author

Root cause: where NBO hangs

An NBO DDL runs in two phases:

sequenceDiagram
  participant N1 as node_1 (ALTER)
  participant N2 as node_2
  participant G as garbd
  N1->>N1: Phase 1: NBO begin replicated in TOI, keys into nbo_index_, monitors released
  N1->>N1: ALTER runs locally (N2 too)
  N1->>N1: Phase 2: wait_nbo_end() sends NBO end
  N2->>N1: NBO end
  Note over N1: certification.cpp:912<br/>current_view_.subset_of(ended_set())<br/>view = {N1, N2, garbd}, ended = {N1, N2}
  Note over G: never applies write sets,<br/>never sends NBO end
  Note over N1: barrier never met, so wait forever
Loading
  • The waiter sits in galera::ReplicatorSMM::wait_nbo_end() (galera/src/replicator_smm.cpp:1779) until Certification marks the NBO ended. On the server, the ALTER shows in closing tables (captured in the processlist while the test hung).
  • current_view_ is a galera::View built from wsrep_view_info_t, which lists every member, garbd included (galera/src/galera_view.cpp).
  • lock_wait_timeout does not help. wsrep-lib poll_enter_toi checks its deadline only after enter_toi returns, and in this case it never returns.
  • Stopping garbd unblocks the DDL: the view change aborts the waiters, they resend NBO end, and the new 2-member view is a subset of the ended set.

Not captured: gdb stacks. Attaching was blocked (kernel.yama.ptrace_scope = 1), so the location above comes from reading the code and from the processlist.

Fix

Build the barrier's galera::View from the configuration change record (View(const gcs_act_cchange&)). Leave out members that are stateless and have no incoming address, but only if every other member supports this change (upgrade gate). All four cert_.adjust_position() callers pass View(conf). The view the server receives (wsrep_cluster_size, pxc_cluster_view, quorum, flow control) is unchanged.

@jaideepkarande

Copy link
Copy Markdown
Contributor Author

Design decisions: how garbd is identified, and approaches that were rejected

Rule: a member is an arbitrator if, in the configuration change (CC) record of that view, it is stateless and has no incoming client address. A data node that would match refuses to start.

Principles

  1. Identical decision on every node. If nodes end the NBO at different write sets, the NBO keys leave the certification index at different points and later writes to the table certify differently. The result is silent divergence.
  2. Use the CC record, not live cluster state. The CC record is built at the view change, ordered with the write sets, stored in gcache and replayed by IST. The live GCS member table (gcs_group_t::nodes[]) may already reflect the next view, so nodes could get different answers.
  3. No names. garbd's --name is configurable, and names are not unique.
  4. Fail safe. Failing to recognize garbd keeps today's hang. Wrongly skipping a data node ends the NBO while that node is still running the DDL, which is worse.
  5. No wsrep API or replication protocol version change for an 8.4-only fix.

Can a setting change break the rule?

Setting Effect Evidence
garbd --name none MTR galera_nbo_garbd uses --name my_arbitrator
garbd --options gcs.stateless=no overridden; garbd appends gcs.stateless=yes last garb/garb_config.cpp:180
garbd incoming address not configurable, always "" garb/garb_gcs.cpp:19-23
Data node named garb, or bound to loopback (no address) waited for (not stateless) unit test_view_keeps_data_nodes
Data node gcs.stateless=yes with an address waited for unit test; MTR pxc_5300_stateless_data_node
Data node gcs.stateless=yes without an address refuses to start MTR pxc_5300_stateless_data_node
gcs.stateless set at runtime rejected (ENOTSUP) gcs/src/gcs_group.cpp:2442
One data node on an older version gate off; everyone waits for garbd (consistent) unit test_view_mixed_version

Approaches rejected

Approach Why rejected
Reject NBO when garbd is present Removes a feature instead of fixing the bug
Timeout or abort in wait_nbo_end() The DDL has already run locally and the end decision is made in total order. Giving up on one node leaves nodes inconsistent.
Skip members with an empty incoming address A data node bound to loopback also has ""
Name garb + empty address (implemented first) A renamed garbd still hangs; a data node named garb is skipped
Empty address + cached seqno -1 A data node joining by SST also reports -1
Stateless flag only Since upstream 6739948b, a data node may set gcs.stateless=yes (to avoid being chosen as donor). Shown in MTR: such a node was skipped (alter_ended 0 → 1).
Look members up in gcs_group_t::nodes[] or group_node_is_stateful() That is the live state, not the state at the view change (principle 2)
wsrep_cap_t, or a new field in wsrep_member_info_t Needs a protocol or wsrep API change
Bump the CC record version Old readers fail with "Unsupported CC action version". An optional trailer is used instead.
Mixed-version hazard covered by a release note only Upgraded and old nodes would end the NBO at different points. Replaced by the upgrade gate.
garbd sends NBO end itself Needs a new send path in garbd, and gains nothing: NBO end is not a success/failure vote, and stateless members' votes are not counted (group_count_votes())
Dedicated "I am an arbitrator" state bit, compiled only into garbd Implemented on branch PXC-5300-8.4-arbitrator. No setting can fake it, but an older garbd doesn't send it and would keep hanging until garbd is upgraded too. Not chosen.
Make garbd refuse to join when a stateless data node with no address is present garbd can't tell such a node from another garbd either. The data node itself enforces the rule.

@jaideepkarande

Copy link
Copy Markdown
Contributor Author

Tests

MTR tests added

galera_3nodes_nbo.galera_nbo_garbd

  • Shuts down node_3 and starts garbd on its ports with --name my_arbitrator, giving two data nodes plus a renamed arbitrator.
  • Runs ALTER TABLE t1 ALGORITHM=INPLACE, LOCK=SHARED, ADD KEY with wsrep_OSU_method=NBO on node_1.
  • Polls performance_schema.events_statements_current for up to 60 s until the ALTER has ended.
  • Then kills garbd and checks the table and DML on both nodes.

galera.pxc_5300_stateless_data_node

  • Starts node_2 with gcs.stateless=yes and no incoming address, and checks that startup is refused with the new error.
  • Then starts it with an address and checks that it joins with gcs.stateless=yes honoured (stateless_data_node = 1).

Results

Before the fix (PXC 468ebbb / galera 13ff9ed6) With the fix
galera_nbo_garbd (enc_off, enc_on) NBO DDL completed with garbd in the cluster: 0. Processlist: ALTER TABLE t1 ... in closing tables until garbd was killed. ... : 1
pxc_5300_stateless_data_node — (new behaviour) pass
Candidate run: the two tests above + galera.galera_garbd + galera_3nodes.galera_garbd, enc_off/enc_on Completed: All 9 tests were successful.

Earlier runs on the same Galera rule with the upgrade gate (before the startup error was added):

  • galera_3nodes_nbo full suite: 33/33 pass.
  • IST/garbd set: the tests that aren't disabled pass (galera_ist_progress, galera_ist_xtrabackup-v2, galera_wan_restart_ist, pxc_malformed_ist, pxc_5208_force_sst_after_ist_failure, galera_3nodes.galera_ist_only, galera_garbd ×2, galera_garbd_backup_basic, galera_garbd_sst_failure). The failures in that run were tests listed in collections/disabled.def; they ran only because they were named explicitly.

Gap that was found and closed: with only a startup warning, a data node with gcs.stateless=yes and no address was skipped. node_1's ALTER ended while node_2's NBO was held (alter_ended 0 → 1). The startup error closes that gap.

Galera unit tests

Details are in percona/galera#318: galera_check certification 65/65, and gcs_tests all pass. Three certification tests were adapted from a community-contributed patch (see the comment below).

Known, unrelated

  • A debug garbd asserts on SIGTERM (double close at garb/garb_gcs.cpp:170) and leaves a core. This happens without the fix too.
  • galera_check defaults_check has failed since PXC-5208 (repl.force_sst_after_inconsistency).

Not re-run

The local build directory was removed after these runs, so MTR was not re-run on the final head (03c4d8b98b1). Since those runs, the head only adds Galera unit tests. Jenkins #82 covers the final head.

@jaideepkarande

Copy link
Copy Markdown
Contributor Author

Compatibility & upgrade

Verdict: safe for rolling upgrade. Full details are in percona/galera#318.

  • Mixed versions: garbd is skipped only when every data node in the view runs this version (state message bit 0x80). Until then every node waits for all members, the same as today, so nodes never diverge.
  • garbd: it does not need to be upgraded. Every garbd version runs stateless with no incoming address.
  • CC record: flags travel in an optional trailer, and the record version is unchanged. Old nodes ignore the trailer, and old records read as "no flags".
  • Downgrade, or adding an older node: use SST. IST from a new donor could replay an NBO that the cluster ended without garbd.
  • Behaviour change (release note): a data node with gcs.stateless=yes must have a client address (wsrep_node_incoming_address), otherwise it refuses to start:

    gcs.stateless is set and the node has no address for incoming client connections, so the cluster would treat it as an arbitrator and would not wait for it to finish NBO operations. Unset gcs.stateless or set wsrep_node_incoming_address.

  • Release note: NBO with garbd works once all data nodes are upgraded. A mixed cluster keeps the old behaviour (hangs until garbd leaves).
  • Scope: 8.4 only. No server code change; only the Galera submodule pointer and MTR tests change. The Galera changes are under #ifdef PXC. No wsrep API or replication protocol version change.

@jaideepkarande

Copy link
Copy Markdown
Contributor Author

Community-contributed patch

A community patch was offered for this bug. It leaves out every stateless member, records the flag in an NBOR trailer in the CC record, and includes certification tests. We compared it with this PR. Its tests were adapted into percona/galera#318, but its approach was not taken.

Issue in the patch Severity How this PR handles it
A data node with gcs.stateless=yes (allowed since upstream 6739948b) is skipped, so the NBO can end before that node finishes the DDL High Arbitrator = stateless and no incoming address; a data node that matches refuses to start
No mixed-version gate: upgraded and old nodes end the NBO at different points High State bit 0x80 gate
Trailer decode throws on unexpected data, which can break reading CC records from other versions Medium Unknown or absent trailer reads as "no flags"
No fallback when every member is stateless Medium Falls back to waiting for all members
Changes not under #ifdef PXC Low All fork changes guarded
Its trailer format is incompatible with this one Low One trailer format ('F', version 1)

Tests adapted (Galera commit 159805f7):

  • test_certification_nbo_arbitrator: NBO begin/end write sets through Certification, including a duplicate NBO end. A mixed-version control was added.
  • test_view_preserves_public_membership
  • test_view_membership_changes, extended with a data node in DONOR state.

Thanks to the contributor; the certification-level tests made the coverage stronger.

@jaideepkarande

Copy link
Copy Markdown
Contributor Author

Jenkins

Run: https://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/

  • Debug, ubuntu:noble, full MTR with --big-test.
  • Branch PXC-5300-8.4-stateless-ci at 8319f63e290: this PR's head 03c4d8b98b1 plus a CI-only .gitmodules change that points the Galera submodule at the fork.

Result: UNSTABLE. 16736 tests: 10894 passed, 6 failed, 5836 skipped. None of the failures are new. Every one also appears in baseline runs without this change.

Baselines: pxc-8.x-pipeline-parallel-mtr #953 (PXC-5291-8.4, same parameters, 11 failures) and #954 (PXC-5291-9.7).

Failing test Cause Baseline
shutdown_report ×5 (galera_3nodes, galera_3nodes_nbo, galera-small, galera-big, rpl_gtid) A node aborts on replicator_str.cpp:1510 Assertion 'ist_seqno >= cc_seqno' after node_3's state transfer at cluster start fails with "Not in primary component" Same 5 suites fail in #953. The assertion appears 6× there, in inconsistency_shutdown, galera_toi_vote, GCF-361, galera_nbo_kill_slave_sst and galera_nbo_master_majority_success.
galera.galera_sst_allowed_methods enc_off (enc_on passes) restart_mysqld: shutdown_server failed with error 2 after a garbd SST request (galera_sst_allowed_methods_garbd.inc:26) Same error in #954

This PR's tests:

  • galera_3nodes_nbo.galera_nbo_garbd: enc_on passes. enc_off failed on its first attempt, then passed on both retries. The first attempt was the first test after the cluster started and hit the startup assertion above (on mysqld.2); NBO is not involved.
  • galera.pxc_5300_stateless_data_node: passes, enc_off and enc_on.
  • Other garbd tests pass: galera.galera_garbd, galera_3nodes.galera_garbd, galera_garbd_backup_basic, galera_garbd_backup_ext, galera_garbd_sst_failure, galera_garbd_invalid_donor_name, galera_garbd_invalid_param_value.
  • Unit tests (junit_UNIT_TESTS): no failures.

Verdict: no regressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant