Skip to content

fix: Harden JSONB encoding and decoding against malformed input - #108

Merged
b41sh merged 4 commits into
databendlabs:mainfrom
b41sh:fix-invalid-jsonb
Nov 16, 2026
Merged

b41sh merged 4 commits into
databendlabs:mainfrom
b41sh:fix-invalid-jsonb

Conversation

@b41sh

@b41sh b41sh commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Part 1 of 3 stacked PRs. This PR fixes undefined behavior, panics, hangs and wrong results that fuzzing found in jsonb.

Changes

  • Decoding checks headers, JEntrys and payloads against the data. Malformed bytes return InvalidJsonb, InvalidEOF or InvalidUtf8 instead of panicking or reading past the end.
  • Strings are validated as UTF-8, with an ASCII fast path.
  • Nesting is limited to 512 levels in the text parser and in every recursive JSONB walk (RecursionLimitExceeded).
  • Allocations are sized from the input, not from the lengths it claims.
  • Value::to_vec and write_to_vec return Result and fail with ValueTooLarge when a length does not fit the JSONB length fields.
  • to_string writes the same JSON as serde_json through JsonWriter. Comparison and comparable encoding walk undecoded entries.
  • convert_to_comparable orders values nested more than 255 levels correctly. Values nested less than 255 levels encode exactly as on main.
  • serialize_bytes writes a binary value, which deserialize_bytes and deserialize_byte_buf read back as bytes. Both used to be broken: serialize_bytes was todo!(), and the byte deserializers returned binary values as hexadecimal text.
  • deserialize_ignored_any reads the ignored value and checks it, instead of skipping its bytes unread.
  • from_slice returns RecursionLimitExceeded for JSONB nested too deeply, instead of retrying the bytes as JSON text and reporting a syntax error.
  • src/functions/array.rs is reorganized around element_counts and retain_by_membership; this is a clean-up and does not change behavior.

Breaking changes

  • Value::to_vec and Value::write_to_vec return Result.
  • Malformed input that used to be accepted is now an error.
  • Value's Display writes JSON: NaN and the infinities print as null instead of NaN/inf, and strings are escaped like serde_json instead of with {:?}, which wrote non-JSON escapes such as \0 and \u{7f}.
  • deserialize_bytes and deserialize_byte_buf give a string as its UTF-8 bytes, as in serde_json, instead of as a string.

Performance

Measured with criterion and the benches in the Databend release profile (opt-level = "s", codegen-units = 1, lto = "thin"). Each side ran twice, and the faster median is compared.

Operation canada citm_catalog twitter small
partial_cmp -39% -23% -19% -18%
to_string -9% -32% -24% -26%
contains - -18% -4% -14%
from_slice -18% -12% -2% -9%
convert_to_comparable -8% -5% -3% +1%
to_value +7% +7% +10% +10%
strip_nulls +4% +10% +14% +9%
select_by_path -3% +16% +44% +15%
select_array_by_path($.**) +3% +16% +30% +13%
array_distinct / array_overlap -1% / +1% +30% / +31% -1% / 0% +7% / 0%
get_by_name +11% +2% 0% +25%

Of the 105 benchmarks, 17 are more than 5% faster, 32 are more than 5% slower, and 56 are within 5%. Building, encoding and JSONPath parsing are within about 5%. canada/contains is not measured, because one call takes about 36 s.

Tests

New tests: malformed.rs, invalid_utf8.rs, json_text.rs and allocation.rs.

Unit Integration Doc
default features 36 132 83
--no-default-features --features databend 33 129 83

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Malformed JSONB wrappers and long JSONPath expression chains can still overflow the stack, alongside unresolved empty-key formatting, error-propagation, and copy-budget defects.

5 open issues
What changed in this PR

This PR hardens the JSONB library against malformed input and resource exhaustion, corrects numeric and path-handling behavior, and adds fuzzing infrastructure.

Changes:

  • Validates UTF-8 and binary lengths, adds recursion and JSONPath resource limits, and makes value encoding fallible.
  • Corrects arithmetic, decimal conversion, JSON escaping, path round trips, and comparable encoding.
  • Adds five fuzz targets, reference models, seeds, and regression tests.
File Description
tests/​it/​testdata/​json_path.txt Updates escaped-field formatting expectation.
tests/​it/​parser.rs Tests JSON display and negative hexadecimal boundaries.
tests/​it/​number.rs Tests numeric corrections and negative-zero encoding.
tests/​it/​main.rs Registers new regression modules.
tests/​it/​keypath_parser.rs Tests empty-key paths.
tests/​it/​jsonpath_parser.rs Tests literals, parentheses, and empty strings.
tests/​it/​invalid_utf8.rs Adds malformed-string regression and property tests.
tests/​it/​functions.rs Updates encoding calls and path tests.
tests/​it/​encode.rs Updates fallible encoding tests and tests binary serialization.
tests/​it/​decode.rs Adapts corrupted-input setup to fallible encoding.
tests/​allocation.rs Checks allocations from malformed length fields.
src/​value.rs Adds JSON escaping and fallible encoding APIs.
src/​util.rs Adds buffer rollback on failed writes.
src/​raw.rs Tracks depth during recursive comparisons.
src/​parser.rs Limits nesting and fixes hexadecimal negation.
src/​owned.rs Propagates encoding failures.
src/​number.rs Corrects arithmetic, conversion, and decimal formatting.
src/​keypath.rs Escapes formatted key-path names.
src/​jsonpath/​selector.rs Adds evaluation and copy budgets.
src/​jsonpath/​path.rs Fixes index arithmetic and path formatting.
src/​jsonpath/​parser.rs Expands literal parsing and checks parenthesis depth.
src/​jsonpath/​mod.rs Exposes shared path-formatting helpers internally.
src/​jaq.rs Limits recursive JSONB-to-jaq conversion.
src/​functions/​scalar.rs Updates UTF-8 and fallible-encoding examples.
src/​functions/​path.rs Hardens index arithmetic and recursive traversal.
src/​functions/​operator.rs Limits recursion and updates comparable encoding.
src/​functions/​array.rs Avoids overflow in insertion indices.
src/​extension.rs Hardens date, timestamp, and interval formatting.
src/​error.rs Adds resource-limit errors.
src/​core/​item.rs Propagates comparison depth.
src/​core/​databend/​util.rs Validates payloads, headers, and decoded strings.
src/​core/​databend/​ser.rs Hardens serialization and implements binary-byte serialization.
src/​core/​databend/​jentry.rs Validates entry lengths and container counts.
src/​core/​databend/​iterator.rs Validates strings and stops iteration after errors.
src/​core/​databend/​de.rs Bounds payload reads and recursive decoding.
src/​core/​databend/​builder.rs Propagates length-validation failures.
src/​constants.rs Defines nesting and JSONPath resource limits.
fuzz/​src/​model.rs Adds reference implementations for fuzz comparisons.
fuzz/​src/​bin/​write_seeds.rs Generates representative and boundary-value seeds.
fuzz/​src/​bin/​raw_jsonb.rs Adds the binary JSONB fuzz entry point.
fuzz/​src/​bin/​number.rs Adds the numeric fuzz entry point.
fuzz/​src/​bin/​json_text.rs Adds the JSON-text fuzz entry point.
fuzz/​src/​bin/​json_path.rs Adds the path fuzz entry point.
fuzz/​src/​bin/​binary_ops.rs Adds the binary-operation fuzz entry point.
fuzz/​README.md Documents fuzz targets and usage.
fuzz/​Cargo.toml Configures the standalone fuzz package.
fuzz/​.gitignore Excludes generated fuzz artifacts.
Cargo.toml Enables warnings for undocumented unsafe blocks.
benches/​strip_nulls.rs Adapts benchmarks to fallible encoding.
benches/​get_path.rs Adapts benchmark setup to fallible encoding.

Review effort: Balanced

Comment thread src/core/databend/util.rs Resolved
Comment thread src/jsonpath/parser.rs Resolved
Comment thread src/core/databend/iterator.rs Resolved
Comment thread src/jsonpath/selector.rs Resolved
Comment thread src/keypath.rs Resolved
@sundy-li

Copy link
Copy Markdown
Member

Review (sol-loop, round 1 of up to 3) — head b74a5a1

Reviewed with gpt-6.1-sol against base acf90042. Checks on the PR head: cargo clippy --all-targets --all-features clean, cargo test --all-features green (34 + 1 + 135 + 83 tests, doctests).

Verdict: CHANGES. Three findings, all reproduced locally. Fixes are applied in the working tree and will be pushed to this branch; they are not in b74a5a1 yet.

S1 [blocker] Scalar-container wrappers bypass the nesting limit

src/core/databend/de.rs (decode_jsonb, deserialize_any) and src/core/databend/util.rs (value_with_depth) follow a CONTAINER_TAG JEntry under a SCALAR_CONTAINER_TAG header by recursing with the same depth, so enter_container is never charged. 100 000 nested scalar headers each holding a container JEntry overflow a 2 MiB stack in to_value, from_slice and from_raw_jsonb.

Fix: reject a container JEntry under a scalar header in all three decoders. Valid JSONB never emits one, and the serializer, JsonbItem::from_raw_jsonb and jsonb_item_type already refuse it. Regression test: tests/it/malformed.rs::scalar_headers_reject_container_jentries.

S2 [blocker] Paren counting does not bound the logical expression tree

expr_or / expr_and in src/jsonpath/parser.rs fold a || b || … into a left-deep tree, one level per operator. $ ? ( + 10 000 @ == 1 terms joined by || + ) passes the paren pre-scan with depth 1, and evaluating it (Selector::eval_filter_expr) overflows a 2 MiB stack before the item budget is reached. Display and Drop recurse the same way, so a post-parse check is not enough: the tree must not be built.

Fix: the pre-parse text scan (now check_expr_depth) also counts && / || outside string literals and rejects more than MAX_JSON_PATH_LOGICAL_OPS = MAX_NESTING_DEPTH - MAX_JSON_PATH_DEPTH = 448, keeping tree depth within the same 512 budget as data. Regression test: tests/it/malformed.rs::json_path_enforces_logical_chain_limit (449 terms accepted and evaluated, 450 and 10 000 rejected, && too, operators inside quotes ignored). The existing 120-term round-trip test still passes.

S3 [should] Filter literals are copied outside the byte budget

ExprValue::Value(Box<PathValue>) was cloned once per item in convert_expr_val and again per comparison in eval_compare / compare_value. An escaped string literal is Cow::Owned, so $[*] ? (@ == "<65 535 × a>\u0061") against 1 000 nulls allocated ~198 MB for a 4 KB document while returning Ok(0), despite the 32 MiB extra-copy limit.

Fix: ExprValue::Value now borrows &'a PathValue<'a> from the path, eval_compare iterates borrowed slices and compare_value takes references, so the comparison path makes no copies. Regression test: tests/allocation.rs::filter_literals_are_not_copied_per_item (total allocation 198 MB before, < 2 MB after).

Notes for the description (not findings)

  • Value::to_vec() now returns Result<Vec<u8>>: public API break, needs a release note.
  • convert_to_comparable encoding changed (depth prefix is now a big-endian u16; -0.0 encodes as 0). Must not be compared against persisted keys from older versions.
  • Mixed Int64/UInt64 arithmetic now returns exact values; result variants differ in a few cases (-1 + 5u is UInt64(4), was Int64(4)).

Round 2 (changed code and the three ids only) is in progress; I'll follow up here with its verdict.

@sundy-li

Copy link
Copy Markdown
Member

Review follow-up (sol-loop rounds 2–3) — verdict: APPROVE

Rounds 2 and 3 with gpt-6.1-sol reviewed only the changed code and the open ids.

  • S1, S2, S3: resolved (round 2).
  • S4 [should], new in round 2, fixed: my first fix for S2 counted && in a text pre-scan, but & is not a delimiter in raw_string, so $.a&&a&&… is a valid unquoted name and 449 of them were wrongly rejected with RecursionLimitExceeded; the quoted form also broke Display round-tripping. Counting in the text can't be exact without tokenizing, so the operator count now lives in the parser: expr_and/expr_or fold through fold_logical_chain, which tracks &&/|| in a thread-local counter per parse_json_path call and fails before building a chain once the total exceeds MAX_JSON_PATH_LOGICAL_OPS (448). The paren pre-scan is back to parens only. Regressions in json_path_enforces_logical_chain_limit: $. + "a&&"×1000, quoted and unquoted, parse and round-trip; 449 terms are accepted and evaluated, 450 and 10 000 are rejected.
  • Round 3: S4 resolved, no new findings. APPROVE.

Final state of the fixes (not yet pushed; 7 files, +221/−59 over b74a5a1):
src/constants.rs, src/core/databend/de.rs, src/core/databend/util.rs, src/jsonpath/parser.rs, src/jsonpath/selector.rs, tests/allocation.rs, tests/it/malformed.rs.
Checks: cargo clippy --all-targets --all-features clean, cargo fmt --check clean, cargo test --all-features green (34 + 2 + 137 + 83 tests, doctests).

Reviewed with sol-loop: 3 rounds with gpt-6.1-sol. Sol verdict: approved.

@b41sh

b41sh commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The 512-level recursion limit can still allow stack overflow on a default-sized debug thread stack, and the enlarged test stack does not validate safe rejection under normal conditions.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Deserialize binary extensions as bytes, not hexadecimal strings

src/​core/​databend/​ser.rs:261

The new binary serialization cannot round-trip through the Serde deserializer: deserialize_bytes and deserialize_byte_buf in de.rs still call visit_string(self.read_string()?), which formats binary extensions as hexadecimal text. Byte-only visitors therefore fail, while visitors accepting strings receive the hexadecimal characters instead of the original bytes. Please decode binary extensions through visit_bytes/visit_byte_buf in those methods and add a Serde byte-buffer round-trip test.

5 resolved since last review

Review effort: Balanced

Comment thread src/constants.rs Resolved
Comment thread src/jsonpath/parser.rs Resolved

@sundy-li sundy-li left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 561960b. S1 and S3 are fixed (same approach as in my local fix), CI is green. One item is still open, so I'm holding the approval for now:

S4 [should] src/jsonpath/parser.rs check_nesting: counting && in the text rejects valid field names. raw_string treats | as a delimiter but not &, so a&& is a legal unquoted name. On this head:

let name = "a&&".repeat(300);
parse_json_path(format!("$.{name}").as_bytes());      // Err(RecursionLimitExceeded), no expression in the path
let p = parse_json_path(format!("$.\"{name}\"").as_bytes()).unwrap();
parse_json_path(p.to_string().as_bytes());           // Err: Display writes the name unquoted, round-trip breaks

Sol flagged the same bug in my first attempt at S2; a text scan can't tell .a&& (name) from 1.&& (number literal followed by an operator) without tokenizing. The fix that passed round 3 counts the operators where the parser folds them instead, so the parenthesis pre-scan stays parens-only:

thread_local! {
    /// `&&` and `||` folded so far by the current `parse_json_path` call.
    static LOGICAL_OPS: Cell<usize> = const { Cell::new(0) };
}

pub fn parse_json_path(input: &[u8]) -> Result<JsonPath<'_>, Error> {
    check_paren_depth(input)?;
    LOGICAL_OPS.set(0);
    // nom reports the limit as a plain Error, and a failed optional rule leaves
    // its text unparsed, so both failure paths consult the count.
    let limit_hit = || if LOGICAL_OPS.get() > MAX_JSON_PATH_LOGICAL_OPS {
        Error::RecursionLimitExceeded
    } else {
        Error::InvalidJsonPath
    };
    match json_path(input) {
        Ok((rest, json_path)) => {
            if !rest.is_empty() { return Err(limit_hit()); }
            Ok(json_path)
        }
        Err(nom::Err::Error(_) | nom::Err::Failure(_)) => Err(limit_hit()),
        Err(nom::Err::Incomplete(_)) => unreachable!(),
    }
}

/// Folds `exprs` into a left-deep chain of `op`, failing before building it
/// once the whole path has more than MAX_JSON_PATH_LOGICAL_OPS operators.
fn fold_logical_chain(op: BinaryOperator, exprs: Vec<Expr<'_>>) -> Result<Expr<'_>, ErrorKind> {
    let ops = LOGICAL_OPS.get() + exprs.len().saturating_sub(1);
    LOGICAL_OPS.set(ops);
    if ops > MAX_JSON_PATH_LOGICAL_OPS {
        return Err(ErrorKind::TooLarge);
    }
    let mut exprs = exprs.into_iter();
    let first = exprs.next().ok_or(ErrorKind::SeparatedList)?;
    Ok(exprs.fold(first, |left, right| Expr::BinaryOp {
        op: op.clone(), left: Box::new(left), right: Box::new(right),
    }))
}

// expr_and / expr_or: map_res(separated_list1(...), |exprs| fold_logical_chain(BinaryOperator::And, exprs))

Regression to add to json_path_enforces_logical_chain_limit: $. + "a&&"×1000, quoted and unquoted, parses and Display round-trips; operator chains over the limit still return RecursionLimitExceeded.

Happy to approve as soon as this lands.

@b41sh
b41sh force-pushed the fix-invalid-jsonb branch 2 times, most recently from 1f0d434 to 5d48bb6 Compare November 13, 2026 08:45
Decoding:
- Check every header, JEntry and payload against the data length, and
  reject unknown JEntry types, container entries under scalar headers,
  and non-string object keys instead of panicking or reading past the end.
- Validate strings as UTF-8, with an ASCII fast path.
- Limit nesting to 512 levels in the text parser and in every recursive
  walk over JSONB (`Error::RecursionLimitExceeded`).
- Do not size allocations from claimed lengths.
- Iterators stop after the first error, and indexing reports errors in
  the elements it skips.
- Format and compare decimals with any scale without panicking.

Encoding:
- `Value::to_vec` and `Value::write_to_vec` return `Result`, and fail
  with `Error::ValueTooLarge` for values the length fields cannot hold;
  a failed write leaves the buffer unchanged.

Performance:
- Walk undecoded entries (`RawEntry`, `Entries`) for comparison,
  comparable encoding, `to_string` and serialization.
- Write JSON text directly (`JsonWriter`), matching serde_json's output.
@b41sh
b41sh force-pushed the fix-invalid-jsonb branch from 5d48bb6 to b1fa5cb Compare November 16, 2026 02:07
Inline the small functions on the hot paths that stopped being inlined
under `opt-level = "s"` once JEntry decoding returned `Result`:
`read_jentry`, `read_u32`, `slice`, `JEntry::decode_jentry`,
`RawEntry::into_item`, the iterators' `next` and `read_next`, and
`push_comparable_tag`. Key lookups called `read_jentry` for every key,
which doubled `select_by_path` on twitter.json.

Check null and boolean JEntry lengths with one shift and mask instead of
three comparisons.

Write comparable-encoding depths below 255 as one byte again, and deeper
ones as 255 followed by a big-endian `u16`. The `u16` for every depth made
the output 6% to 14% larger and `convert_to_comparable` up to 37% slower,
and changed the encoding of every value. Now values nested less than 255
levels encode exactly as before, and deeper ones still order correctly.
@b41sh

b41sh commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@codex review

@b41sh b41sh changed the title fix: Harden JSONB decoding, number arithmetic and JSONPath fix: Harden JSONB encoding and decoding against malformed input Nov 16, 2026
@b41sh
b41sh requested review from sundy-li and a balanced review from Copilot November 16, 2026 06:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Binary round-trips fail, malformed values can compare as equal, ignored inputs bypass validation, and recursive JSONPath traversal remains vulnerable to counter overflow and stack exhaustion.

4 open issues
2 resolved since last review

Review effort: Balanced

Comment thread src/core/databend/entry.rs Resolved
Comment thread src/core/databend/ser.rs Resolved
Comment thread tests/it/malformed.rs Resolved
Comment thread src/core/databend/de.rs Resolved
@sundy-li

Copy link
Copy Markdown
Member

Review — head f12831f (part 1 of 3)

Reviewed the split-out part 1 against acf9004. Checks on this head: cargo fmt --check clean, cargo clippy --all-targets --all-features clean, cargo test --all-features green (36 + 1 + 131 + 83, doctests).

What I verified beyond the test suite:

  • to_string / to_pretty_string produce the same bytes as serde_json::to_string(&raw) / to_string_pretty for floats (1e21, 1e-7, 5e-324, -0.0, 1.7976931348623157e308), control-char / non-ASCII escapes, and empty nested containers.
  • convert_to_comparable byte order agrees with partial_cmp across the one-byte / three-byte depth boundary (depths 253–256, 300, and sibling arrays at depth 254/255).
  • deserialize_ignored_any skips container-valued unknown struct fields and IgnoredAny array elements, and the following elements still decode.
  • Stack use at MAX_NESTING_DEPTH: in release every recursive API (to_value, from_slice, from_raw_jsonb::<serde_json::Value>, serde / JsonWriter output, partial_cmp, contains, strip_nulls, convert_to_comparable, jaq, text parse, Value Drop/Display) handles 512 nested arrays and objects on a 512 KiB thread stack. In a debug build on a 2 MiB stack, to_value, from_slice, from_raw_jsonb, partial_cmp, contains, jaq and Value Drop overflow at 512 levels (to_string, serde output, strip_nulls, convert_to_comparable and the text parser are fine). Databend runs tokio threads at 20 MiB and std threads at 5 MiB, so this is not a blocker, but see the doc nit below.

Verdict: one should fix, rest are nits. Happy to approve once the bytes round trip is handled or explicitly deferred.

S1 [should] serialize_bytes and deserialize_bytes / deserialize_byte_buf disagree

src/core/databend/ser.rs now encodes serialize_bytes as a Binary extension (good, it was todo!()), but src/core/databend/de.rs:525-537 still implement deserialize_bytes / deserialize_byte_buf as visitor.visit_string(self.read_string()?), and read_str formats an extension payload with Display, i.e. hex text. So a serde_bytes-style type does not round-trip:

// Bytes(Vec<u8>) with serialize_bytes / deserialize_byte_buf + visit_bytes/visit_byte_buf
let owned = jsonb::to_owned_jsonb(&Bytes(vec![1, 2, 3])).unwrap();
jsonb::from_raw_jsonb::<Bytes>(&owned.as_raw())
// Err("invalid type: string \"010203\", expected bytes")

Suggested fix: in deserialize_bytes / deserialize_byte_buf, read the scalar JEntry; if it is EXTENSION_TAG and ExtensionValue::decode gives Binary(b), call visitor.visit_borrowed_bytes(b) (the payload borrows 'de), otherwise keep the current string path. Add a round-trip test next to test_serialize_bytes_as_binary in tests/it/encode.rs. Copilot flagged the same thing on the previous head.

Nits

  • src/constants.rs doc on MAX_NESTING_DEPTH: "about 1.5 KiB of stack per level ... fits tokio's default 2 MiB" only holds for optimized builds; debug to_value needs roughly 4–9 KiB per level and overflows a 2 MiB thread at 512 (the tests use a 64 MiB stack for that reason). Worth stating "in release builds; debug builds need a larger stack" so nobody reads it as a guarantee.
  • src/value.rs Display: Value::Number(Float64(NaN|±inf)) now prints null (was NaN/inf), and strings are JSON-escaped instead of {:?} (fixes \u{7f}, \0-style non-JSON output). Both are correct, but they are user-visible changes of Value::to_string() and should be listed under "Breaking changes" in the description.
  • src/parser.rs from_slice: a decode failure of any kind falls back to parse_value(buf), so a valid JSONB nested deeper than 512 surfaces as a text Syntax error rather than RecursionLimitExceeded. Not new behavior in spirit, but the error is misleading now that the limit exists; consider only falling back when the error is not RecursionLimitExceeded.
  • src/core/databend/json_writer.rs relies on serde_json >= 1.0.147 (zmij) for the float output to match Serialize; Cargo.toml still says serde_json = "1.0". Bumping the minimum makes the "same JSON as serde_json" claim hold for downstream resolvers too.
  • src/functions/array.rs: the element_counts / retain_by_membership rewrite is a clean-up rather than hardening. Behavior is unchanged as far as I can tell (array_distinct on a scalar, array_overlap early exit), just noting it for anyone bisecting later.
  • PR description: "the benches from #PR3" should point at the actual PR number once it exists.

`deserialize_bytes` and `deserialize_byte_buf` read binary values as
their hexadecimal text, so bytes written by `serialize_bytes` came back as
the string "010203" instead of `[1, 2, 3]`, which byte-only visitors
rejected. They now give binary values as bytes, and strings as their
UTF-8 bytes, as in serde_json.

`deserialize_ignored_any` skipped the value's bytes unread. At the top
level it accepted any input, even an empty one, and inside containers it
accepted invalid values and bypassed the nesting limit. Ignored values are
now read like any other.
`from_slice` retried any JSONB decoding error as JSON text, so JSONB
nested deeper than 512 levels failed with a misleading syntax error. It
now returns `RecursionLimitExceeded` directly.

Also state in the `MAX_NESTING_DEPTH` docs that the stack estimate holds
for release builds only; debug builds overflow a 2 MiB stack at 512
levels.

@sundy-li sundy-li left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed e6d23d7 (the branch was rewritten since my last comments).

  • S1 (scalar header wrapping a container JEntry) is fixed in every decoder: to_value, from_slice, serde Serialize, from_raw_jsonb, jaq and partial_cmp all return an error on 100 000 nested wrappers instead of overflowing the stack. Verified locally.
  • The JSONPath changes (parser depth/operator limits, selector budgets, number literals, Display round-tripping) are no longer part of this PR, so S2, S3 and S4 are out of scope here. S2 (a flat || chain of ~10 000 terms overflows the stack in Selector::eval_filter_expr) and S3 (owned string literals cloned per compared item) also exist on main, so they are worth a follow-up PR; the parser-fold counter from my previous comment handles S2 without rejecting a&& field names.
  • convert_to_comparable now keeps the one-byte depth prefix for depths below 255 and escapes deeper ones with a 0xFF marker, so existing sort keys stay byte-compatible. Good.
  • Local checks on this head: cargo clippy --all-targets --all-features clean, cargo test --all-features green (36 + 1 + 133 + 83 tests, doctests). CI build is green.

Approving.

@b41sh
b41sh merged commit 7b52608 into databendlabs:main Nov 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants