Repository navigation
fix: Make number arithmetic, number comparison and JSONPath exact and bounded - #109
Conversation
Numbers: - Compute integer arithmetic in `i128`, so mixed `Int64`/`UInt64` operands no longer wrap (`u64::MAX + (-1)` returned `-2`), and report overflow from `neg`. - Convert decimals to the nearest `f64` with one rounding, through an exact division, Eisel-Lemire, or parsing `<digits>e-<scale>`. JSONPath: - Limit parenthesis depth to 64 and `&&`/`||` operators to 256 when parsing (`Error::RecursionLimitExceeded`). - Bound one evaluation to 1,000,000 work units and to copying 32 MiB beyond the document size (`Error::JsonPathTooComplex`), and limit container nesting while selecting. - Parse float literals with a fraction, an exponent, or `inf`/`nan`. - Make slices outside the array select nothing instead of looping. - Quote and escape names in `Display` for JSONPath and key paths, so the output parses back to the same path.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
sundy-li
left a comment
There was a problem hiding this comment.
LGTM. 6a34490 fixes the large-array case: $[*] and $[*] ? (@ == 2) now work on 900k and 1.1M int arrays.
Follow-up, not blocking: the work limit is now 1M plus document size times path node count. A long filter or unrelated padding in the document raises the limit for the .** steps before it. On a 304 KB document of 500 nested arrays with 100 ints each, $.**.**.** fails in 67 ms. $.**.**.** ? (@ == 1 || …) with 256 || aborts on a 4 GB allocation. Scaling by selection steps only and capping the work queue length would bound this.
Part 2 of 3 stacked PRs. Continue PR #108 to harden number and JSONPath
Changes
i128, so mixedInt64/UInt64operands no longer wrap:u64::MAX + (-1)returned-2.negreports overflow.f64conversion rounds once and returns the nearest float.0.1still equals the decimal0.1. Before,2^53 + 1equaled the float2^53, which equals2^53.&&/||operators to 256 (RecursionLimitExceeded). Evaluation is limited to 1,000,000 work units and 32 MiB of extra copies (JsonPathTooComplex). Recursive selection (.**) counts levels inusizeand applies the nesting limit from fix: Harden JSONB encoding and decoding against malformed input #108.Displayoutput of JSONPath and key paths parses back to the same path.Behavior changes
2^64equals18446744073709552000, the text it prints as.Performance
Measured against
main(7b52608) with criterion and the benches in the Databend release profile (opt-level = "s",codegen-units = 1,lto = "thin"). Each side ran twice, and the faster median is compared.select_by_pathselect_array_by_path($.**)partial_cmpcontainsconvert_to_comparablefrom_raw_jsonbparse_json_pathcasesimplefilter_1_opfilter_8_opsfilter_64_opsfilter_256_opsname_with_&&(a name with 50&&)The other operations in
readandarray(to_string,to_value,from_slice,get_by_name,object_each,strip_nulls, the array set operations) are within 5%, exceptsmall/strip_nullsat +5%.Tests
--no-default-features --features databendThe new
number_order_is_exact_and_transitivetest fails without the comparison fix. fmt and clippy (-D warnings) pass with both feature sets.