Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions pxc-galera-operations/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Operating Percona XtraDB Cluster (PXC) - the Galera-based synchron

# Percona XtraDB Cluster (PXC) / Galera Operations

*Last updated: 2026-06-02*
*Last updated: 2026-11-10*

PXC is Percona Server for MySQL plus Galera: synchronous, virtually-multi-master replication where every node holds the full dataset and commits are certified cluster-wide. A transaction's write-set is broadcast to all nodes and certified before commit; if it conflicts with a concurrent transaction on any node, the committing node rolls back. A write confirmed anywhere is durable everywhere, or it never commits. The cost: commit latency grows with inter-node round-trips, and same-row writes from multiple nodes raise the certification-conflict rate. PXC is for **high availability and read scaling with zero divergence** - throughput is capped by the slowest node, so it is **not** a write-scaling solution. Most "it behaves weirdly" surprises come from Galera semantics, not MySQL.

Expand All @@ -24,7 +24,7 @@ Open ports: **3306** (SQL), **4567** (group comm), **4444** (SST), **4568** (IST

| Likely agent answer | Closer to reality |
|---|---|
| Creating tables without a primary key | Galera requires a PRIMARY KEY on **every** table for write-set ordering and parallel apply. With `pxc_strict_mode=ENFORCING` (default), DML on PK-less tables is rejected. `wsrep_certify_nonPK` (on) can auto-generate one, but the documented rule is: **all tables must have a primary key.** |
| Creating tables without a primary key | Galera requires a PRIMARY KEY on **every** table for write-set ordering and parallel apply. With `pxc_strict_mode=ENFORCING` (default), DML on PK-less tables is rejected. From **9.7.2-2**, ENFORCING/MASTER also sets `sql_require_primary_key=ON` globally: `CREATE`/`ALTER` of a PK-less table fails, and setting it `OFF` in those modes is an error (8.4.11-11 shipped this too, but later 8.4 releases revert to the 8.4.10 behavior, so don't count on it in 8.4). `wsrep_certify_nonPK` (on) can auto-generate one, but the documented rule is: **all tables must have a primary key.** |
| Treating `COMMIT` as guaranteed once the DML statements succeed | Certification is optimistic and resolved at **commit time**: `COMMIT` can return `ERROR 1213` (certification conflict, not a local deadlock). The app must catch it and **retry the whole transaction.** |
| Assuming MyISAM (or any non-InnoDB) data replicates | Only **InnoDB** is replicated. `wsrep_replicate_myisam` is **OFF** by default, so MyISAM DML is node-local -> silent divergence (DDL still replicates). MyISAM-in-cluster is experimental and discouraged. |
| Expecting writes to all nodes to scale throughput | Throughput is capped by the slowest node; concurrent same-row writes across nodes just raise the 1213 conflict rate. PXC scales reads and gives HA, not write throughput. |
Expand All @@ -41,7 +41,7 @@ Open ports: **3306** (SQL), **4567** (group comm), **4444** (SST), **4568** (IST
| HAProxy `option mysql-check` / raw TCP health check | Use `option httpchk` against `clustercheck` (port 9200) so `JOINING`/`DONOR` nodes are excluded from rotation. |
| `pc.bootstrap=true` on a survivor during a network partition | Only when the other side is **confirmed down**. During a mere partition this creates two clusters and an eventual emergency shutdown + full SST. |
| Different SSL certs per node | All nodes (and `garbd`) must share **identical** CA/cert/key files or they won't join. |
| Using XtraBackup-level SST encryption (`[xtrabackup] encrypt=`) on PXC 8.4 | **Removed in 8.4** - PXC errors if it detects it. Use `pxc-encrypt-cluster-traffic` (on by default) for SST/IST/replication TLS. |
| Using XtraBackup-level SST encryption (`[xtrabackup] encrypt=`) on PXC 8.4 | **Removed in 8.4** - PXC errors if it detects it. Use `[sst] encrypt=4` with `ssl-ca`/`ssl-cert`/`ssl-key` (defaults to 4 when `pxc-encrypt-cluster-traffic=ON`, the default). Clone SST uses the MySQL `ssl-*` settings instead. |

## Bootstrapping essentials

Expand All @@ -68,7 +68,7 @@ Full config and the bootstrap/join command sequence are in `references/operation

## pxc_strict_mode

Values: `DISABLED`, `PERMISSIVE`, `ENFORCING` (default), `MASTER`. Default is `ENFORCING`, except a standalone/bootstrapping node defaults to `DISABLED`. ENFORCING blocks unsupported operations outright (no-PK DML, non-InnoDB writes, `innodb_autoinc_lock_mode != 2`, `LOCK TABLES`/`GET_LOCK()`, wrong `binlog_format`, Group Replication); PERMISSIVE logs and continues. Note: validation happens on the **originating** node - an op let through on a DISABLED node is not re-checked on ENFORCING peers.
Values: `DISABLED`, `PERMISSIVE`, `ENFORCING` (default), `MASTER`. Default is `ENFORCING`, except a standalone/bootstrapping node defaults to `DISABLED`. ENFORCING blocks unsupported operations outright (no-PK DML - and from 9.7.2-2 no-PK `CREATE`/`ALTER` via `sql_require_primary_key=ON` - non-InnoDB writes, `innodb_autoinc_lock_mode != 2`, `LOCK TABLES`/`GET_LOCK()`, wrong `binlog_format`, Group Replication); PERMISSIVE logs and continues. Lowering the mode to `PERMISSIVE`/`DISABLED` does **not** reset `sql_require_primary_key` - it stays `ON` until you set it explicitly; existing sessions keep their session value until they reconnect. Note: validation happens on the **originating** node - an op let through on a DISABLED node is not re-checked on ENFORCING peers.

## PXC vs async vs Group Replication

Expand Down Expand Up @@ -99,14 +99,14 @@ search_percona_docs(query="PXC <your question verbatim>",

## Key gotchas

- **Primary key on every table; only InnoDB replicates.** No-PK DML and MyISAM writes don't replicate / get rejected.
- **Primary key on every table; only InnoDB replicates.** No-PK DML and MyISAM writes don't replicate / get rejected; 9.7.2-2+ also rejects no-PK `CREATE`/`ALTER` under ENFORCING/MASTER (8.4.11-11 only; later 8.4 releases revert this).
- **`COMMIT` can fail with error 1213** (certification conflict) - apps must retry the transaction.
- **Never bootstrap into a running cluster.** It forks the cluster.
- **`mysqld --wsrep-recover`** is how you pick the bootstrap node after an unclean full outage - not `safe_to_bootstrap`.
- **Odd node counts** (3/5) or an arbitrator. Two-node clusters can't tolerate a failure.
- **TOI DDL stalls the cluster** - use RSU/NBO or pt-online-schema-change for long ALTERs.
- **PXC is HA + read scaling, not write scaling** - throughput is bounded by the slowest node.
- **8.4 dropped XtraBackup-SST encryption** - rely on `pxc-encrypt-cluster-traffic`.
- **8.4 dropped XtraBackup-SST encryption** - use `[sst] encrypt=4` + `ssl-*` (implied by `pxc-encrypt-cluster-traffic=ON`).

## Sources

Expand All @@ -118,5 +118,8 @@ search_percona_docs(query="PXC <your question verbatim>",
- [Online schema upgrade (TOI/RSU/NBO)](https://docs.percona.com/percona-xtradb-cluster/8.4/online-schema-upgrade.html)
- [Monitoring the cluster](https://docs.percona.com/percona-xtradb-cluster/8.4/monitoring.html)
- [Load balancing with ProxySQL](https://docs.percona.com/percona-xtradb-cluster/8.4/load-balance-proxysql.html) · [HAProxy](https://docs.percona.com/percona-xtradb-cluster/8.4/haproxy.html)
- [Galera Arbitrator (garbd)](https://docs.percona.com/percona-xtradb-cluster/8.4/garbd-howto.html) · [Encrypt traffic](https://docs.percona.com/percona-xtradb-cluster/8.4/encrypt-traffic.html)
- [Galera Arbitrator (garbd)](https://docs.percona.com/percona-xtradb-cluster/8.4/garbd-howto.html) · [Encrypt traffic](https://docs.percona.com/percona-xtradb-cluster/8.4/encrypt-traffic.html) · [XtraBackup SST options](https://docs.percona.com/percona-xtradb-cluster/8.4/xtrabackup-sst.html)
- [PXC 8.4.11-11 release notes](https://docs.percona.com/percona-xtradb-cluster/8.4/release-notes/8.4.11-11.html)
- [PXC 9.7 strict mode (sql_require_primary_key rules)](https://docs.percona.com/percona-xtradb-cluster/9.7/strict-mode.html)
- [PXC-5341: revert of the 8.4.11 strict-mode PK change on 8.4](https://github.com/percona/percona-xtradb-cluster/pull/2379)
- [wsrep system variables](https://docs.percona.com/percona-xtradb-cluster/8.0/wsrep-system-index/) · [wsrep provider options](https://docs.percona.com/percona-xtradb-cluster/8.0/wsrep-provider-index/)
11 changes: 7 additions & 4 deletions pxc-galera-operations/references/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,17 +47,18 @@ systemctl start mysql@bootstrap.service # bootstrap ONLY that node

## Clone SST (PXC 8.4)

PXC 8.4 (8.4.4-x) adds MySQL Clone-plugin-based SST. Set `wsrep_sst_method=clone` on the joiner; `clone` must be in `wsrep_sst_allowed_methods` on both donor and joiner (8.4 default is `xtrabackup-v2, clone`). Requires the `CLONE_ADMIN` privilege and the NetCat package; default port 4444. (Clone is also available on the 8.0 line from 8.0.41+.) Both variables are read-only / config-file-only.
PXC 8.4 (8.4.4-x) adds MySQL Clone-plugin-based SST. Set `wsrep_sst_method=clone` on the joiner; `clone` must be in `wsrep_sst_allowed_methods` on both donor and joiner (8.4 default is `xtrabackup-v2, clone`). Requires the `CLONE_ADMIN` privilege and the NetCat package; default port 4444. (On the 8.0 line, Clone SST exists from 8.0.41+ but is **tech preview** - test it before relying on it in production.) Clone SST encrypts with the MySQL `ssl-ca`/`ssl-cert`/`ssl-key` settings (or `clone_ssl_*` on the joiner); keep the certs outside the datadir. Both variables are read-only / config-file-only.

## Key variables

| Variable | Default | Purpose |
|---|---|---|
| `wsrep_cluster_address` | - | `gcomm://` peer list; empty = bootstrap |
| `wsrep_sst_method` | `xtrabackup-v2` | SST method (`clone` in 8.4.4+ / 8.0.41+) |
| `pxc_strict_mode` | `ENFORCING` | PK/engine/locking validation (`DISABLED`/`PERMISSIVE`/`ENFORCING`/`MASTER`) |
| `wsrep_sst_method` | `xtrabackup-v2` | SST method (`clone` in 8.4.4+; 8.0.41+ as tech preview) |
| `pxc_strict_mode` | `ENFORCING` | PK/engine/locking validation (`DISABLED`/`PERMISSIVE`/`ENFORCING`/`MASTER`); 9.7.2-2+ ENFORCING/MASTER also forces `sql_require_primary_key=ON` and rejects setting it `OFF` (8.4.11-11 had this; later 8.4 releases revert to 8.4.10 behavior); lowering the mode later does not reset it |
| `wsrep_OSU_method` | `TOI` | DDL replication mode (`TOI`/`RSU`/`NBO`) |
| `wsrep_provider_options` | - | e.g. `gcache.size=1G;pc.weight=2` |
| `repl.force_sst_after_inconsistency` (provider option) | off (runtime-settable) | Added in 8.4.11-11 and 9.7.2-2. A node voted out of the cluster for a data inconsistency removes `grastate.dat` on shutdown, so its next start does a full SST instead of IST. The 8.4 notes also cover a joiner that fails to apply a write-set during IST |
| `innodb_autoinc_lock_mode` | must be `2` | Required by Galera |
| `wsrep_applier_threads` | `1` | Parallel apply (tune to `wsrep_cert_deps_distance`); replaces deprecated `wsrep_slave_threads` |
| `wsrep_sync_wait` | `0` | Causal-read consistency level |
Expand Down Expand Up @@ -95,7 +96,7 @@ GALERA_OPTIONS="socket.ssl_cipher=AES128-SHA256;socket.ssl_key=...;socket.ssl_ce

## Encryption & large transactions

- **Traffic encryption** is ON by default (`pxc-encrypt-cluster-traffic=ON`, covers SST/IST/replication). All nodes must use **identical** cert files; changing it needs a full cluster stop. PXC 8.4 **removed** XtraBackup-level SST encryption (`[xtrabackup] encrypt=`) - rely on `pxc-encrypt-cluster-traffic`.
- **Traffic encryption** is ON by default (`pxc-encrypt-cluster-traffic=ON`, covers SST/IST/replication). All nodes must use **identical** cert files; changing it needs a full cluster stop. PXC 8.4 **removed** XtraBackup-level SST encryption (`[xtrabackup] encrypt=`). The supported SST method is `[sst] encrypt=4` with `ssl-ca`/`ssl-cert`/`ssl-key`; it defaults to 4 when `pxc-encrypt-cluster-traffic=ON` and reverts to 0 if that is OFF. `transferfmt=nc` does not support `encrypt=4`.
- **Huge transactions** - by default the whole write-set is buffered in memory (`wsrep_max_ws_size` 2 GB). Fragment a big batch with streaming replication instead:
```sql
SET SESSION wsrep_trx_fragment_unit='rows';
Expand All @@ -105,3 +106,5 @@ GALERA_OPTIONS="socket.ssl_cipher=AES128-SHA256;socket.ssl_key=...;socket.ssl_ce
## Upgrades

Rolling **8.0 -> 8.4**, one node at a time: remove 8.0 packages (keep datadir), install 8.4, start (auto-upgrade + SST/IST rejoin). Keep writes on 8.0 nodes during the mixed window and avoid 8.4-only DDL until done. PXC 8.4 switches the keyring plugin to a component and defaults to `caching_sha2_password` (needs ProxySQL >= 2.6.2).

**8.4 -> 9.7:** target **9.7.2 or newer** - 8.4.11 -> 9.7.1 is unsupported.
Loading