diff --git a/pxc-galera-operations/SKILL.md b/pxc-galera-operations/SKILL.md index 6e349c0..610e7cf 100644 --- a/pxc-galera-operations/SKILL.md +++ b/pxc-galera-operations/SKILL.md @@ -5,7 +5,7 @@ description: "Operating Percona XtraDB Cluster (PXC) - the Galera-based synchron # Percona XtraDB Cluster (PXC) / Galera Operations -*Last updated: 2026-06-02* +*Last updated: 2026-11-10* PXC is Percona Server for MySQL plus Galera: synchronous, virtually-multi-master replication where every node holds the full dataset and commits are certified cluster-wide. A transaction's write-set is broadcast to all nodes and certified before commit; if it conflicts with a concurrent transaction on any node, the committing node rolls back. A write confirmed anywhere is durable everywhere, or it never commits. The cost: commit latency grows with inter-node round-trips, and same-row writes from multiple nodes raise the certification-conflict rate. PXC is for **high availability and read scaling with zero divergence** - throughput is capped by the slowest node, so it is **not** a write-scaling solution. Most "it behaves weirdly" surprises come from Galera semantics, not MySQL. @@ -24,7 +24,7 @@ Open ports: **3306** (SQL), **4567** (group comm), **4444** (SST), **4568** (IST | Likely agent answer | Closer to reality | |---|---| -| Creating tables without a primary key | Galera requires a PRIMARY KEY on **every** table for write-set ordering and parallel apply. With `pxc_strict_mode=ENFORCING` (default), DML on PK-less tables is rejected. `wsrep_certify_nonPK` (on) can auto-generate one, but the documented rule is: **all tables must have a primary key.** | +| Creating tables without a primary key | Galera requires a PRIMARY KEY on **every** table for write-set ordering and parallel apply. With `pxc_strict_mode=ENFORCING` (default), DML on PK-less tables is rejected. From **9.7.2-2**, ENFORCING/MASTER also sets `sql_require_primary_key=ON` globally: `CREATE`/`ALTER` of a PK-less table fails, and setting it `OFF` in those modes is an error (8.4.11-11 shipped this too, but later 8.4 releases revert to the 8.4.10 behavior, so don't count on it in 8.4). `wsrep_certify_nonPK` (on) can auto-generate one, but the documented rule is: **all tables must have a primary key.** | | Treating `COMMIT` as guaranteed once the DML statements succeed | Certification is optimistic and resolved at **commit time**: `COMMIT` can return `ERROR 1213` (certification conflict, not a local deadlock). The app must catch it and **retry the whole transaction.** | | Assuming MyISAM (or any non-InnoDB) data replicates | Only **InnoDB** is replicated. `wsrep_replicate_myisam` is **OFF** by default, so MyISAM DML is node-local -> silent divergence (DDL still replicates). MyISAM-in-cluster is experimental and discouraged. | | Expecting writes to all nodes to scale throughput | Throughput is capped by the slowest node; concurrent same-row writes across nodes just raise the 1213 conflict rate. PXC scales reads and gives HA, not write throughput. | @@ -41,7 +41,7 @@ Open ports: **3306** (SQL), **4567** (group comm), **4444** (SST), **4568** (IST | HAProxy `option mysql-check` / raw TCP health check | Use `option httpchk` against `clustercheck` (port 9200) so `JOINING`/`DONOR` nodes are excluded from rotation. | | `pc.bootstrap=true` on a survivor during a network partition | Only when the other side is **confirmed down**. During a mere partition this creates two clusters and an eventual emergency shutdown + full SST. | | Different SSL certs per node | All nodes (and `garbd`) must share **identical** CA/cert/key files or they won't join. | -| Using XtraBackup-level SST encryption (`[xtrabackup] encrypt=`) on PXC 8.4 | **Removed in 8.4** - PXC errors if it detects it. Use `pxc-encrypt-cluster-traffic` (on by default) for SST/IST/replication TLS. | +| Using XtraBackup-level SST encryption (`[xtrabackup] encrypt=`) on PXC 8.4 | **Removed in 8.4** - PXC errors if it detects it. Use `[sst] encrypt=4` with `ssl-ca`/`ssl-cert`/`ssl-key` (defaults to 4 when `pxc-encrypt-cluster-traffic=ON`, the default). Clone SST uses the MySQL `ssl-*` settings instead. | ## Bootstrapping essentials @@ -68,7 +68,7 @@ Full config and the bootstrap/join command sequence are in `references/operation ## pxc_strict_mode -Values: `DISABLED`, `PERMISSIVE`, `ENFORCING` (default), `MASTER`. Default is `ENFORCING`, except a standalone/bootstrapping node defaults to `DISABLED`. ENFORCING blocks unsupported operations outright (no-PK DML, non-InnoDB writes, `innodb_autoinc_lock_mode != 2`, `LOCK TABLES`/`GET_LOCK()`, wrong `binlog_format`, Group Replication); PERMISSIVE logs and continues. Note: validation happens on the **originating** node - an op let through on a DISABLED node is not re-checked on ENFORCING peers. +Values: `DISABLED`, `PERMISSIVE`, `ENFORCING` (default), `MASTER`. Default is `ENFORCING`, except a standalone/bootstrapping node defaults to `DISABLED`. ENFORCING blocks unsupported operations outright (no-PK DML - and from 9.7.2-2 no-PK `CREATE`/`ALTER` via `sql_require_primary_key=ON` - non-InnoDB writes, `innodb_autoinc_lock_mode != 2`, `LOCK TABLES`/`GET_LOCK()`, wrong `binlog_format`, Group Replication); PERMISSIVE logs and continues. Lowering the mode to `PERMISSIVE`/`DISABLED` does **not** reset `sql_require_primary_key` - it stays `ON` until you set it explicitly; existing sessions keep their session value until they reconnect. Note: validation happens on the **originating** node - an op let through on a DISABLED node is not re-checked on ENFORCING peers. ## PXC vs async vs Group Replication @@ -99,14 +99,14 @@ search_percona_docs(query="PXC ", ## Key gotchas -- **Primary key on every table; only InnoDB replicates.** No-PK DML and MyISAM writes don't replicate / get rejected. +- **Primary key on every table; only InnoDB replicates.** No-PK DML and MyISAM writes don't replicate / get rejected; 9.7.2-2+ also rejects no-PK `CREATE`/`ALTER` under ENFORCING/MASTER (8.4.11-11 only; later 8.4 releases revert this). - **`COMMIT` can fail with error 1213** (certification conflict) - apps must retry the transaction. - **Never bootstrap into a running cluster.** It forks the cluster. - **`mysqld --wsrep-recover`** is how you pick the bootstrap node after an unclean full outage - not `safe_to_bootstrap`. - **Odd node counts** (3/5) or an arbitrator. Two-node clusters can't tolerate a failure. - **TOI DDL stalls the cluster** - use RSU/NBO or pt-online-schema-change for long ALTERs. - **PXC is HA + read scaling, not write scaling** - throughput is bounded by the slowest node. -- **8.4 dropped XtraBackup-SST encryption** - rely on `pxc-encrypt-cluster-traffic`. +- **8.4 dropped XtraBackup-SST encryption** - use `[sst] encrypt=4` + `ssl-*` (implied by `pxc-encrypt-cluster-traffic=ON`). ## Sources @@ -118,5 +118,8 @@ search_percona_docs(query="PXC ", - [Online schema upgrade (TOI/RSU/NBO)](https://docs.percona.com/percona-xtradb-cluster/8.4/online-schema-upgrade.html) - [Monitoring the cluster](https://docs.percona.com/percona-xtradb-cluster/8.4/monitoring.html) - [Load balancing with ProxySQL](https://docs.percona.com/percona-xtradb-cluster/8.4/load-balance-proxysql.html) · [HAProxy](https://docs.percona.com/percona-xtradb-cluster/8.4/haproxy.html) -- [Galera Arbitrator (garbd)](https://docs.percona.com/percona-xtradb-cluster/8.4/garbd-howto.html) · [Encrypt traffic](https://docs.percona.com/percona-xtradb-cluster/8.4/encrypt-traffic.html) +- [Galera Arbitrator (garbd)](https://docs.percona.com/percona-xtradb-cluster/8.4/garbd-howto.html) · [Encrypt traffic](https://docs.percona.com/percona-xtradb-cluster/8.4/encrypt-traffic.html) · [XtraBackup SST options](https://docs.percona.com/percona-xtradb-cluster/8.4/xtrabackup-sst.html) +- [PXC 8.4.11-11 release notes](https://docs.percona.com/percona-xtradb-cluster/8.4/release-notes/8.4.11-11.html) +- [PXC 9.7 strict mode (sql_require_primary_key rules)](https://docs.percona.com/percona-xtradb-cluster/9.7/strict-mode.html) +- [PXC-5341: revert of the 8.4.11 strict-mode PK change on 8.4](https://github.com/percona/percona-xtradb-cluster/pull/2379) - [wsrep system variables](https://docs.percona.com/percona-xtradb-cluster/8.0/wsrep-system-index/) · [wsrep provider options](https://docs.percona.com/percona-xtradb-cluster/8.0/wsrep-provider-index/) diff --git a/pxc-galera-operations/references/operations.md b/pxc-galera-operations/references/operations.md index ba3bd9f..7745b44 100644 --- a/pxc-galera-operations/references/operations.md +++ b/pxc-galera-operations/references/operations.md @@ -47,17 +47,18 @@ systemctl start mysql@bootstrap.service # bootstrap ONLY that node ## Clone SST (PXC 8.4) -PXC 8.4 (8.4.4-x) adds MySQL Clone-plugin-based SST. Set `wsrep_sst_method=clone` on the joiner; `clone` must be in `wsrep_sst_allowed_methods` on both donor and joiner (8.4 default is `xtrabackup-v2, clone`). Requires the `CLONE_ADMIN` privilege and the NetCat package; default port 4444. (Clone is also available on the 8.0 line from 8.0.41+.) Both variables are read-only / config-file-only. +PXC 8.4 (8.4.4-x) adds MySQL Clone-plugin-based SST. Set `wsrep_sst_method=clone` on the joiner; `clone` must be in `wsrep_sst_allowed_methods` on both donor and joiner (8.4 default is `xtrabackup-v2, clone`). Requires the `CLONE_ADMIN` privilege and the NetCat package; default port 4444. (On the 8.0 line, Clone SST exists from 8.0.41+ but is **tech preview** - test it before relying on it in production.) Clone SST encrypts with the MySQL `ssl-ca`/`ssl-cert`/`ssl-key` settings (or `clone_ssl_*` on the joiner); keep the certs outside the datadir. Both variables are read-only / config-file-only. ## Key variables | Variable | Default | Purpose | |---|---|---| | `wsrep_cluster_address` | - | `gcomm://` peer list; empty = bootstrap | -| `wsrep_sst_method` | `xtrabackup-v2` | SST method (`clone` in 8.4.4+ / 8.0.41+) | -| `pxc_strict_mode` | `ENFORCING` | PK/engine/locking validation (`DISABLED`/`PERMISSIVE`/`ENFORCING`/`MASTER`) | +| `wsrep_sst_method` | `xtrabackup-v2` | SST method (`clone` in 8.4.4+; 8.0.41+ as tech preview) | +| `pxc_strict_mode` | `ENFORCING` | PK/engine/locking validation (`DISABLED`/`PERMISSIVE`/`ENFORCING`/`MASTER`); 9.7.2-2+ ENFORCING/MASTER also forces `sql_require_primary_key=ON` and rejects setting it `OFF` (8.4.11-11 had this; later 8.4 releases revert to 8.4.10 behavior); lowering the mode later does not reset it | | `wsrep_OSU_method` | `TOI` | DDL replication mode (`TOI`/`RSU`/`NBO`) | | `wsrep_provider_options` | - | e.g. `gcache.size=1G;pc.weight=2` | +| `repl.force_sst_after_inconsistency` (provider option) | off (runtime-settable) | Added in 8.4.11-11 and 9.7.2-2. A node voted out of the cluster for a data inconsistency removes `grastate.dat` on shutdown, so its next start does a full SST instead of IST. The 8.4 notes also cover a joiner that fails to apply a write-set during IST | | `innodb_autoinc_lock_mode` | must be `2` | Required by Galera | | `wsrep_applier_threads` | `1` | Parallel apply (tune to `wsrep_cert_deps_distance`); replaces deprecated `wsrep_slave_threads` | | `wsrep_sync_wait` | `0` | Causal-read consistency level | @@ -95,7 +96,7 @@ GALERA_OPTIONS="socket.ssl_cipher=AES128-SHA256;socket.ssl_key=...;socket.ssl_ce ## Encryption & large transactions -- **Traffic encryption** is ON by default (`pxc-encrypt-cluster-traffic=ON`, covers SST/IST/replication). All nodes must use **identical** cert files; changing it needs a full cluster stop. PXC 8.4 **removed** XtraBackup-level SST encryption (`[xtrabackup] encrypt=`) - rely on `pxc-encrypt-cluster-traffic`. +- **Traffic encryption** is ON by default (`pxc-encrypt-cluster-traffic=ON`, covers SST/IST/replication). All nodes must use **identical** cert files; changing it needs a full cluster stop. PXC 8.4 **removed** XtraBackup-level SST encryption (`[xtrabackup] encrypt=`). The supported SST method is `[sst] encrypt=4` with `ssl-ca`/`ssl-cert`/`ssl-key`; it defaults to 4 when `pxc-encrypt-cluster-traffic=ON` and reverts to 0 if that is OFF. `transferfmt=nc` does not support `encrypt=4`. - **Huge transactions** - by default the whole write-set is buffered in memory (`wsrep_max_ws_size` 2 GB). Fragment a big batch with streaming replication instead: ```sql SET SESSION wsrep_trx_fragment_unit='rows'; @@ -105,3 +106,5 @@ GALERA_OPTIONS="socket.ssl_cipher=AES128-SHA256;socket.ssl_key=...;socket.ssl_ce ## Upgrades Rolling **8.0 -> 8.4**, one node at a time: remove 8.0 packages (keep datadir), install 8.4, start (auto-upgrade + SST/IST rejoin). Keep writes on 8.0 nodes during the mixed window and avoid 8.4-only DDL until done. PXC 8.4 switches the keyring plugin to a component and defaults to `caching_sha2_password` (needs ProxySQL >= 2.6.2). + +**8.4 -> 9.7:** target **9.7.2 or newer** - 8.4.11 -> 9.7.1 is unsupported.