Repository navigation
Check the final physical plan in the planner and name the rule that broke it - #26461
Open
jayzhan211 wants to merge 2 commits into
Open
jayzhan211 wants to merge 2 commits into
jayzhan211 wants to merge 2 commits into
Conversation
github-actions
Bot
added
documentation
jayzhan211
marked this pull request as ready for review
October 18, 2026 03:14
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26461 +/- ##
==========================================
- Coverage 82.91% 82.91% -0.01%
==========================================
Files 1155 1155
Lines 458091 458146 +55
Branches 458091 458146 +55
==========================================
+ Hits 379821 379859 +38
- Misses 55231 55244 +13
- Partials 23039 23043 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Thank you, I have a quick look and I think its overall beneficial. Will have a in-depth review after this PR getting some other +1s. Specifically, I think the benefits is prevent internal tests or simple extensions from building invalid optimizer pipeline; but for more advanced extension (that overrides sanity check), we might need additional API to set a custom sanity checker. |
alamb
reviewed
Oct 18, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
I quickly looked at this and it seems like a good idea. I will try and find more time to review later on.
My only concern is plan speed -- I will run some planning benchmarks to check
| | 19 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. | | ||
| | 20 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. | | ||
| | 21 | `SanityCheckPlan` | - | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. | | ||
| After the last rule, including any rule added by the user, the physical planner |
Contributor
There was a problem hiding this comment.
I feel like this document is outdated and redundand and hard to find -- maybe as a follow on PR we can fold it into the documentation in #26351
Contributor
Author
There was a problem hiding this comment.
Agreed. #26351 has merged, so a follow-up PR can move this list next to the contract docs.
This PR is now rebased on it. I also removed SanityCheckPlan from the contract's list of rules that must keep a fixed order, and added a sentence saying the planner checks the plan after the last rule.
Contributor
Author
There was a problem hiding this comment.
TIL about this document 🤯
+1 😆
| // given query plan; i.e. it only acts as a final | ||
| // gatekeeping rule. | ||
| Arc::new(SanityCheckPlan::new()), | ||
| // There is no `SanityCheckPlan` rule at the end of this list. The |
Contributor
There was a problem hiding this comment.
how about we deprecate SanityCheckplan rule so it is clearer we intend to remove it
Contributor
Author
There was a problem hiding this comment.
Done. SanityCheckPlan is now deprecated since 56.0.0, and the note points to check_plan_tree_sanity for code that runs optimizer rules without the planner. Tests that only used the rule to validate a plan now call check_plan_tree_sanity directly, and the upgrade guide says the rule is deprecated.
I deprecated it rather than removing it because several downstream projects construct it in their own rule lists, for example greptimedb, cube, sail and openobserve. They should see a deprecation warning before they hit a compile error.
Contributor
|
run benchmark sql_planner |
1 similar comment
Contributor
|
run benchmark sql_planner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing planner-final-plan-check (2689cbb) to 290ad15 (merge-base) diff Run configurationrun benchmark sql_plannerResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing planner-final-plan-check (2689cbb) to 290ad15 (merge-base) diff Run configurationrun benchmark sql_plannerResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing planner-final-plan-check (2689cbb) to 290ad15 (merge-base) diff Run configurationrun benchmark sql_plannerCPU Details (lscpu)Details
Resource Usagesql_planner — base (merge-base)
sql_planner — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing planner-final-plan-check (2689cbb) to 290ad15 (merge-base) diff Run configurationrun benchmark sql_plannerCPU Details (lscpu)Details
Resource Usagesql_planner — base (merge-base)
sql_planner — branch
File an issue against this benchmark runner |
jayzhan211
force-pushed
the
planner-final-plan-check
branch
from
October 19, 2026 13:53
2689cbb to
184d94f
Compare
Contributor
Author
|
@alamb Thanks for running these. The second run is the quieter one. In it, The change should not add planning work. Before, the planner walked the final plan twice: once in |
Contributor
Author
|
@2010YOUY01 Thanks. To see who would need that API, I looked at how downstream projects use the rule today:
The last group is the one your comment is about. If the check rejects a correct plan, for example one whose ordering holds but cannot be proven from the plan's properties, those projects can no longer skip the check. I'd suggest adding a way to skip or replace the check in a follow-up, once someone needs it, rather than guessing at the shape of the API now. I'm happy to add it here instead if you'd prefer it to ship in the same release. |
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
jayzhan211
force-pushed
the
planner-final-plan-check
branch
from
October 21, 2026 12:20
184d94f to
ff53db5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related: #26171 (option 3), #26351, #26360, #25688.
Rationale for this change
SanityCheckPlanis the last rule inPhysicalOptimizer::new(), so rules appended withSessionStateBuilder::with_physical_optimizer_rulerun after it and are never checked. Onmain, an appended rule that removes a window's sort silently returns wrongrow_number()values. Custom pipelines have to keep the rule last by hand, and a custom list without it runs plans whose requirements do not hold. When the check does fail, the error only saysSanityCheckPlan, so finding the rule that broke the plan has needed a manual bisect each time (for example #26391, #17077, #25921).@alamb suggested this change in the review of #26107: remove
SanityCheckPlanas an optimizer pass and always run it after the optimizer is done (comment).#26171 agreed (option 3) that a custom pipeline must keep the plan valid, while optimization coverage is not guaranteed. This PR makes that validity part checked for every pipeline. The review on #26360 asked for the same shape: one rule,
EnsureRequirements, makes the plan valid, and every later rule keeps it valid.What changes are included in this PR?
SanityCheckPlanleaves the default rule list.DefaultPhysicalPlanner::optimize_physical_planruns the same check (unbounded inputs, ordering, distribution,check_invariants(Executable)) after the last rule, including appended rules. It replaces the planner's ownInvariantChecker(Executable)walk, whichSanityCheckPlanrepeated.The planner keeps the plan after each rule (one
Arcper rule). When the ordering or distribution requirements fail, the error names the rule that broke them:or, when the plan was already invalid as
EnsureRequirementsproduced it:Plans before
EnsureRequirementsneed not satisfy the requirements, so only plans from its last run onwards are searched. The rule is recognized by its name; a separate enforcement phase (feat: run physical requirement enforcement first, as a PhysicalAnalyzer phase #25688) would give a cleaner boundary. Errors about unbounded inputs keep the plainSanityCheckPlancontext, so existing messages do not change.sanity_checkerexposescheck_plan_tree_sanity,check_input_requirementsandinput_requirements_hold, which the planner uses.SanityCheckPlanis deprecated. Tests that used it to validate a plan callcheck_plan_tree_sanityinstead.The upgrade guide, the optimizer rule reference and
explain.sltare updated.What is the testing strategy for this PR?
sanity_checker.rs: a rule appended after the default rules that removes sorts is reported by name, both with the default list and with a custom list; a list withoutEnsureRequirementsreports that no rule produced a valid plan; the default list passes.memory_limit/mod.rs: the two sort merge join tests ran aSortMergeJoinExecover unsorted streams through a custom rule list withoutEnsureRequirements. They now sort the generated data by the join keys and declare that order, so the plan is valid and the tests still measure the join's memory use.Are there any user-facing changes?
with_physical_optimizer_rule, or by a custom rule list, are now checked. A plan whose requirements do not hold, such as a sort merge join over unsorted input, fails to plan instead of running and possibly returning wrong results.PhysicalOptimizer::new()no longer containsSanityCheckPlan, andEXPLAIN VERBOSEno longer printsphysical_plan after SanityCheckPlan. Code that looks the rule up in the default list no longer finds it.SanityCheckPlanitself is deprecated; code that runs optimizer rules without the planner can callsanity_checker::check_plan_tree_sanityon the final plan.This changes the public behavior of the default rule list, so it needs the
api changelabel.