Repository navigation
PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence [8.4] - #2410
jaideepkarande wants to merge 3 commits into
Conversation
… presence https://perconadev.atlassian.net/browse/PXC-5300 Problem: With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO hangs forever on all nodes. It completes only once garbd leaves the cluster. Description: NBO end is effective only when every member of the current view has sent an NBO end message. The view included garbd, which never applies write sets and therefore never sends NBO end, so the condition was never met. Resolution: Update the galera submodule with the fix: each member's GCS stateless flag is carried in the configuration change record, and members that are stateless and have no incoming address (garbd, whatever its --name) are left out of the NBO end barrier. Arbitrators are left out only when all other members run a version with this change, so mixed-version clusters keep the old behaviour on every node. Test case added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300 Problem: A data node with gcs.stateless=yes and no incoming address looks like the arbitrator to the rest of the cluster and is left out of the NBO end barrier, so an NBO could end before that node finished the DDL. Description: Only the data node itself can tell that it is not an arbitrator. Resolution: Update the galera submodule: such a data node refuses to start, before joining the cluster. A stateless data node with an incoming address still starts. Test case added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PXC-5300 Galera now has unit tests that run NBO begin/end write sets through Certification with garbd in the view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root cause: where NBO hangsAn NBO DDL runs in two phases: sequenceDiagram
participant N1 as node_1 (ALTER)
participant N2 as node_2
participant G as garbd
N1->>N1: Phase 1: NBO begin replicated in TOI, keys into nbo_index_, monitors released
N1->>N1: ALTER runs locally (N2 too)
N1->>N1: Phase 2: wait_nbo_end() sends NBO end
N2->>N1: NBO end
Note over N1: certification.cpp:912<br/>current_view_.subset_of(ended_set())<br/>view = {N1, N2, garbd}, ended = {N1, N2}
Note over G: never applies write sets,<br/>never sends NBO end
Note over N1: barrier never met, so wait forever
Not captured: gdb stacks. Attaching was blocked ( FixBuild the barrier's |
Design decisions: how garbd is identified, and approaches that were rejectedRule: a member is an arbitrator if, in the configuration change (CC) record of that view, it is stateless and has no incoming client address. A data node that would match refuses to start. Principles
Can a setting change break the rule?
Approaches rejected
|
TestsMTR tests added
Results
Earlier runs on the same Galera rule with the upgrade gate (before the startup error was added):
Gap that was found and closed: with only a startup warning, a data node with Galera unit testsDetails are in percona/galera#318: Known, unrelated
Not re-runThe local build directory was removed after these runs, so MTR was not re-run on the final head ( |
Compatibility & upgradeVerdict: safe for rolling upgrade. Full details are in percona/galera#318.
|
Community-contributed patchA community patch was offered for this bug. It leaves out every stateless member, records the flag in an
Tests adapted (Galera commit
Thanks to the contributor; the certification-level tests made the coverage stronger. |
JenkinsRun: https://pxc.cd.percona.com/job/pxc-8.x-pipeline-parallel-mtr-AI/82/
Result: UNSTABLE. 16736 tests: 10894 passed, 6 failed, 5836 skipped. None of the failures are new. Every one also appears in baseline runs without this change. Baselines:
This PR's tests:
Verdict: no regressions. |
PXC-5300: NBO method gets completely stuck with an arbitrator (garbd) presence
https://perconadev.atlassian.net/browse/PXC-5300
Problem:
With garbd in the primary component, any DDL run with wsrep_OSU_method=NBO
hangs forever on all nodes. It completes only once garbd leaves the cluster.
Description:
NBO end is effective only when every member of the current view has sent
an NBO end message. The view included garbd, which never applies write sets
and therefore never sends NBO end, so the condition was never met.
Resolution:
Update the galera submodule with the fix: each member's GCS stateless flag
is carried in the configuration change record, and members that are
stateless and have no incoming address (garbd, whatever its --name) are
left out of the NBO end barrier. Arbitrators are left out only when all
other members run a version with this change, so mixed-version clusters
keep the old behaviour on every node.
Test case added.
Test plan
galera_3nodes_nbo.galera_nbo_garbdrecordsNBO DDL completed with garbd in the cluster: 0; the ALTER stays inclosing tablesuntil garbd is killed.galera_nbo_garbd(garbd--name my_arbitrator),galera.pxc_5300_stateless_data_node,galera.galera_garbd,galera_3nodes.galera_garbdpass, enc_off/enc_on (9/9 incl.shutdown_report).galera_checkcertification 65/65,gcs_tests).🤖 Generated with Claude Code