Skip to content

fix: Make number arithmetic, number comparison and JSONPath exact and bounded - #109

Merged
b41sh merged 3 commits into
databendlabs:mainfrom
b41sh:fix/number-jsonpath
Nov 17, 2026
Merged

b41sh merged 3 commits into
databendlabs:mainfrom
b41sh:fix/number-jsonpath

Conversation

@b41sh

@b41sh b41sh commented Nov 16, 2026

Copy link
Copy Markdown
Member

Part 2 of 3 stacked PRs. Continue PR #108 to harden number and JSONPath

Changes

  • Integer arithmetic is computed in i128, so mixed Int64/UInt64 operands no longer wrap: u64::MAX + (-1) returned -2. neg reports overflow.
  • Decimal-to-f64 conversion rounds once and returns the nearest float.
  • Number comparison is transitive. Integers and decimals compare exactly, and a float compares as its shortest decimal, so the float 0.1 still equals the decimal 0.1. Before, 2^53 + 1 equaled the float 2^53, which equals 2^53.
  • JSONPath parsing limits parenthesis depth to 64 and &&/|| operators to 256 (RecursionLimitExceeded). Evaluation is limited to 1,000,000 work units and 32 MiB of extra copies (JsonPathTooComplex). Recursive selection (.**) counts levels in usize and applies the nesting limit from fix: Harden JSONB encoding and decoding against malformed input #108.
  • JSONPath float literals with a fraction or exponent parse. Slices outside the array select nothing. Display output of JSONPath and key paths parses back to the same path.

Behavior changes

  • Comparison results change where the old order was inconsistent. For example, the float 2^64 equals 18446744073709552000, the text it prints as.
  • JSONPath can return the new errors above.

Performance

Measured against main (7b52608) with criterion and the benches in the Databend release profile (opt-level = "s", codegen-units = 1, lto = "thin"). Each side ran twice, and the faster median is compared.

Operation canada citm_catalog twitter small
select_by_path +1% +2% +1% +1%
select_array_by_path($.**) +1% +4% +3% +3%
partial_cmp +2% 0% 0% 0%
contains - 0% 0% 0%
convert_to_comparable -10% -3% -1% 0%
from_raw_jsonb -3% -3% -2% -1%
parse_json_path case Change
simple +5% (0.70 µs to 0.73 µs)
filter_1_op -13%
filter_8_ops -18%
filter_64_ops -31%
filter_256_ops -21%
name_with_&& (a name with 50 &&) +48% (0.54 µs to 0.79 µs)

The other operations in read and array (to_string, to_value, from_slice, get_by_name, object_each, strip_nulls, the array set operations) are within 5%, except small/strip_nulls at +5%.

Tests

Unit Integration Doc
default features 37 154 83
--no-default-features --features databend 34 152 83

The new number_order_is_exact_and_transitive test fails without the comparison fix. fmt and clippy (-D warnings) pass with both feature sets.

Numbers:
- Compute integer arithmetic in `i128`, so mixed `Int64`/`UInt64`
  operands no longer wrap (`u64::MAX + (-1)` returned `-2`), and report
  overflow from `neg`.
- Convert decimals to the nearest `f64` with one rounding, through an
  exact division, Eisel-Lemire, or parsing `<digits>e-<scale>`.

JSONPath:
- Limit parenthesis depth to 64 and `&&`/`||` operators to 256 when
  parsing (`Error::RecursionLimitExceeded`).
- Bound one evaluation to 1,000,000 work units and to copying 32 MiB
  beyond the document size (`Error::JsonPathTooComplex`), and limit
  container nesting while selecting.
- Parse float literals with a fraction, an exponent, or `inf`/`nan`.
- Make slices outside the array select nothing instead of looping.
- Quote and escape names in `Display` for JSONPath and key paths, so the
  output parses back to the same path.
@b41sh
b41sh requested review from sundy-li and a balanced review from Copilot November 16, 2026 12:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Comment thread src/constants.rs Resolved

@sundy-li sundy-li left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 6a34490 fixes the large-array case: $[*] and $[*] ? (@ == 2) now work on 900k and 1.1M int arrays.

Follow-up, not blocking: the work limit is now 1M plus document size times path node count. A long filter or unrelated padding in the document raises the limit for the .** steps before it. On a 304 KB document of 500 nested arrays with 100 ints each, $.**.**.** fails in 67 ms. $.**.**.** ? (@ == 1 || …) with 256 || aborts on a 4 GB allocation. Scaling by selection steps only and capping the work queue length would bound this.

@b41sh
b41sh merged commit 03f2281 into databendlabs:main Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants