You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: Harden JSONB encoding and decoding against malformed input - #108
Part 1 of 3 stacked PRs. This PR fixes undefined behavior, panics, hangs and wrong results that fuzzing found in jsonb.
Changes
Decoding checks headers, JEntrys and payloads against the data. Malformed bytes return InvalidJsonb, InvalidEOF or InvalidUtf8 instead of panicking or reading past the end.
Strings are validated as UTF-8, with an ASCII fast path.
Nesting is limited to 512 levels in the text parser and in every recursive JSONB walk (RecursionLimitExceeded).
Allocations are sized from the input, not from the lengths it claims.
Value::to_vec and write_to_vec return Result and fail with ValueTooLarge when a length does not fit the JSONB length fields.
to_string writes the same JSON as serde_json through JsonWriter. Comparison and comparable encoding walk undecoded entries.
convert_to_comparable orders values nested more than 255 levels correctly. Values nested less than 255 levels encode exactly as on main.
serialize_bytes writes a binary value, which deserialize_bytes and deserialize_byte_buf read back as bytes. Both used to be broken: serialize_bytes was todo!(), and the byte deserializers returned binary values as hexadecimal text.
deserialize_ignored_any reads the ignored value and checks it, instead of skipping its bytes unread.
from_slice returns RecursionLimitExceeded for JSONB nested too deeply, instead of retrying the bytes as JSON text and reporting a syntax error.
src/functions/array.rs is reorganized around element_counts and retain_by_membership; this is a clean-up and does not change behavior.
Breaking changes
Value::to_vec and Value::write_to_vec return Result.
Malformed input that used to be accepted is now an error.
Value's Display writes JSON: NaN and the infinities print as null instead of NaN/inf, and strings are escaped like serde_json instead of with {:?}, which wrote non-JSON escapes such as \0 and \u{7f}.
deserialize_bytes and deserialize_byte_buf give a string as its UTF-8 bytes, as in serde_json, instead of as a string.
Performance
Measured with criterion and the benches in the Databend release profile (opt-level = "s", codegen-units = 1, lto = "thin"). Each side ran twice, and the faster median is compared.
Operation
canada
citm_catalog
twitter
small
partial_cmp
-39%
-23%
-19%
-18%
to_string
-9%
-32%
-24%
-26%
contains
-
-18%
-4%
-14%
from_slice
-18%
-12%
-2%
-9%
convert_to_comparable
-8%
-5%
-3%
+1%
to_value
+7%
+7%
+10%
+10%
strip_nulls
+4%
+10%
+14%
+9%
select_by_path
-3%
+16%
+44%
+15%
select_array_by_path($.**)
+3%
+16%
+30%
+13%
array_distinct / array_overlap
-1% / +1%
+30% / +31%
-1% / 0%
+7% / 0%
get_by_name
+11%
+2%
0%
+25%
Of the 105 benchmarks, 17 are more than 5% faster, 32 are more than 5% slower, and 56 are within 5%. Building, encoding and JSONPath parsing are within about 5%. canada/contains is not measured, because one call takes about 36 s.
Tests
New tests: malformed.rs, invalid_utf8.rs, json_text.rs and allocation.rs.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Malformed JSONB wrappers and long JSONPath expression chains can still overflow the stack, alongside unresolved empty-key formatting, error-propagation, and copy-budget defects.
This PR hardens the JSONB library against malformed input and resource exhaustion, corrects numeric and path-handling behavior, and adds fuzzing infrastructure.
Changes:
Validates UTF-8 and binary lengths, adds recursion and JSONPath resource limits, and makes value encoding fallible.
Review (sol-loop, round 1 of up to 3) — head b74a5a1
Reviewed with gpt-6.1-sol against base acf90042. Checks on the PR head: cargo clippy --all-targets --all-features clean, cargo test --all-features green (34 + 1 + 135 + 83 tests, doctests).
Verdict: CHANGES. Three findings, all reproduced locally. Fixes are applied in the working tree and will be pushed to this branch; they are not in b74a5a1 yet.
S1 [blocker] Scalar-container wrappers bypass the nesting limit
src/core/databend/de.rs (decode_jsonb, deserialize_any) and src/core/databend/util.rs (value_with_depth) follow a CONTAINER_TAG JEntry under a SCALAR_CONTAINER_TAG header by recursing with the same depth, so enter_container is never charged. 100 000 nested scalar headers each holding a container JEntry overflow a 2 MiB stack in to_value, from_slice and from_raw_jsonb.
Fix: reject a container JEntry under a scalar header in all three decoders. Valid JSONB never emits one, and the serializer, JsonbItem::from_raw_jsonb and jsonb_item_type already refuse it. Regression test: tests/it/malformed.rs::scalar_headers_reject_container_jentries.
S2 [blocker] Paren counting does not bound the logical expression tree
expr_or / expr_and in src/jsonpath/parser.rs fold a || b || … into a left-deep tree, one level per operator. $ ? ( + 10 000 @ == 1 terms joined by || + ) passes the paren pre-scan with depth 1, and evaluating it (Selector::eval_filter_expr) overflows a 2 MiB stack before the item budget is reached. Display and Drop recurse the same way, so a post-parse check is not enough: the tree must not be built.
Fix: the pre-parse text scan (now check_expr_depth) also counts && / || outside string literals and rejects more than MAX_JSON_PATH_LOGICAL_OPS = MAX_NESTING_DEPTH - MAX_JSON_PATH_DEPTH = 448, keeping tree depth within the same 512 budget as data. Regression test: tests/it/malformed.rs::json_path_enforces_logical_chain_limit (449 terms accepted and evaluated, 450 and 10 000 rejected, && too, operators inside quotes ignored). The existing 120-term round-trip test still passes.
S3 [should] Filter literals are copied outside the byte budget
ExprValue::Value(Box<PathValue>) was cloned once per item in convert_expr_val and again per comparison in eval_compare / compare_value. An escaped string literal is Cow::Owned, so $[*] ? (@ == "<65 535 × a>\u0061") against 1 000 nulls allocated ~198 MB for a 4 KB document while returning Ok(0), despite the 32 MiB extra-copy limit.
Fix: ExprValue::Value now borrows &'a PathValue<'a> from the path, eval_compare iterates borrowed slices and compare_value takes references, so the comparison path makes no copies. Regression test: tests/allocation.rs::filter_literals_are_not_copied_per_item (total allocation 198 MB before, < 2 MB after).
Notes for the description (not findings)
Value::to_vec() now returns Result<Vec<u8>>: public API break, needs a release note.
convert_to_comparable encoding changed (depth prefix is now a big-endian u16; -0.0 encodes as 0). Must not be compared against persisted keys from older versions.
Mixed Int64/UInt64 arithmetic now returns exact values; result variants differ in a few cases (-1 + 5u is UInt64(4), was Int64(4)).
Round 2 (changed code and the three ids only) is in progress; I'll follow up here with its verdict.
Rounds 2 and 3 with gpt-6.1-sol reviewed only the changed code and the open ids.
S1, S2, S3: resolved (round 2).
S4 [should], new in round 2, fixed: my first fix for S2 counted && in a text pre-scan, but & is not a delimiter in raw_string, so $.a&&a&&… is a valid unquoted name and 449 of them were wrongly rejected with RecursionLimitExceeded; the quoted form also broke Display round-tripping. Counting in the text can't be exact without tokenizing, so the operator count now lives in the parser: expr_and/expr_or fold through fold_logical_chain, which tracks &&/|| in a thread-local counter per parse_json_path call and fails before building a chain once the total exceeds MAX_JSON_PATH_LOGICAL_OPS (448). The paren pre-scan is back to parens only. Regressions in json_path_enforces_logical_chain_limit: $. + "a&&"×1000, quoted and unquoted, parse and round-trip; 449 terms are accepted and evaluated, 450 and 10 000 are rejected.
Round 3: S4 resolved, no new findings. APPROVE.
Final state of the fixes (not yet pushed; 7 files, +221/−59 over b74a5a1): src/constants.rs, src/core/databend/de.rs, src/core/databend/util.rs, src/jsonpath/parser.rs, src/jsonpath/selector.rs, tests/allocation.rs, tests/it/malformed.rs.
Checks: cargo clippy --all-targets --all-features clean, cargo fmt --check clean, cargo test --all-features green (34 + 2 + 137 + 83 tests, doctests).
Reviewed with sol-loop: 3 rounds with gpt-6.1-sol. Sol verdict: approved.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The 512-level recursion limit can still allow stack overflow on a default-sized debug thread stack, and the enlarged test stack does not validate safe rejection under normal conditions.
Deserialize binary extensions as bytes, not hexadecimal strings
src/core/databend/ser.rs:261
The new binary serialization cannot round-trip through the Serde deserializer: deserialize_bytes and deserialize_byte_buf in de.rs still call visit_string(self.read_string()?), which formats binary extensions as hexadecimal text. Byte-only visitors therefore fail, while visitors accepting strings receive the hexadecimal characters instead of the original bytes. Please decode binary extensions through visit_bytes/visit_byte_buf in those methods and add a Serde byte-buffer round-trip test.
The reason will be displayed to describe this comment to others. Learn more.
Checked 561960b. S1 and S3 are fixed (same approach as in my local fix), CI is green. One item is still open, so I'm holding the approval for now:
S4 [should] src/jsonpath/parser.rscheck_nesting: counting && in the text rejects valid field names.raw_string treats | as a delimiter but not &, so a&& is a legal unquoted name. On this head:
let name = "a&&".repeat(300);parse_json_path(format!("$.{name}").as_bytes());// Err(RecursionLimitExceeded), no expression in the pathlet p = parse_json_path(format!("$.\"{name}\"").as_bytes()).unwrap();parse_json_path(p.to_string().as_bytes());// Err: Display writes the name unquoted, round-trip breaks
Sol flagged the same bug in my first attempt at S2; a text scan can't tell .a&& (name) from 1.&& (number literal followed by an operator) without tokenizing. The fix that passed round 3 counts the operators where the parser folds them instead, so the parenthesis pre-scan stays parens-only:
thread_local!{/// `&&` and `||` folded so far by the current `parse_json_path` call.staticLOGICAL_OPS:Cell<usize> = const{Cell::new(0)};}pubfnparse_json_path(input:&[u8]) -> Result<JsonPath<'_>,Error>{check_paren_depth(input)?;LOGICAL_OPS.set(0);// nom reports the limit as a plain Error, and a failed optional rule leaves// its text unparsed, so both failure paths consult the count.let limit_hit = || ifLOGICAL_OPS.get() > MAX_JSON_PATH_LOGICAL_OPS{Error::RecursionLimitExceeded}else{Error::InvalidJsonPath};matchjson_path(input){Ok((rest, json_path)) => {if !rest.is_empty(){returnErr(limit_hit());}Ok(json_path)}Err(nom::Err::Error(_) | nom::Err::Failure(_)) => Err(limit_hit()),Err(nom::Err::Incomplete(_)) => unreachable!(),}}/// Folds `exprs` into a left-deep chain of `op`, failing before building it/// once the whole path has more than MAX_JSON_PATH_LOGICAL_OPS operators.fnfold_logical_chain(op:BinaryOperator,exprs:Vec<Expr<'_>>) -> Result<Expr<'_>,ErrorKind>{let ops = LOGICAL_OPS.get() + exprs.len().saturating_sub(1);LOGICAL_OPS.set(ops);if ops > MAX_JSON_PATH_LOGICAL_OPS{returnErr(ErrorKind::TooLarge);}letmut exprs = exprs.into_iter();let first = exprs.next().ok_or(ErrorKind::SeparatedList)?;Ok(exprs.fold(first, |left, right| Expr::BinaryOp{op: op.clone(),left:Box::new(left),right:Box::new(right),}))}// expr_and / expr_or: map_res(separated_list1(...), |exprs| fold_logical_chain(BinaryOperator::And, exprs))
Regression to add to json_path_enforces_logical_chain_limit: $. + "a&&"×1000, quoted and unquoted, parses and Display round-trips; operator chains over the limit still return RecursionLimitExceeded.
Decoding:
- Check every header, JEntry and payload against the data length, and
reject unknown JEntry types, container entries under scalar headers,
and non-string object keys instead of panicking or reading past the end.
- Validate strings as UTF-8, with an ASCII fast path.
- Limit nesting to 512 levels in the text parser and in every recursive
walk over JSONB (`Error::RecursionLimitExceeded`).
- Do not size allocations from claimed lengths.
- Iterators stop after the first error, and indexing reports errors in
the elements it skips.
- Format and compare decimals with any scale without panicking.
Encoding:
- `Value::to_vec` and `Value::write_to_vec` return `Result`, and fail
with `Error::ValueTooLarge` for values the length fields cannot hold;
a failed write leaves the buffer unchanged.
Performance:
- Walk undecoded entries (`RawEntry`, `Entries`) for comparison,
comparable encoding, `to_string` and serialization.
- Write JSON text directly (`JsonWriter`), matching serde_json's output.
Inline the small functions on the hot paths that stopped being inlined
under `opt-level = "s"` once JEntry decoding returned `Result`:
`read_jentry`, `read_u32`, `slice`, `JEntry::decode_jentry`,
`RawEntry::into_item`, the iterators' `next` and `read_next`, and
`push_comparable_tag`. Key lookups called `read_jentry` for every key,
which doubled `select_by_path` on twitter.json.
Check null and boolean JEntry lengths with one shift and mask instead of
three comparisons.
Write comparable-encoding depths below 255 as one byte again, and deeper
ones as 255 followed by a big-endian `u16`. The `u16` for every depth made
the output 6% to 14% larger and `convert_to_comparable` up to 37% slower,
and changed the encoding of every value. Now values nested less than 255
levels encode exactly as before, and deeper ones still order correctly.
b41sh
changed the title
fix: Harden JSONB decoding, number arithmetic and JSONPath
fix: Harden JSONB encoding and decoding against malformed input
Nov 16, 2026
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Binary round-trips fail, malformed values can compare as equal, ignored inputs bypass validation, and recursive JSONPath traversal remains vulnerable to counter overflow and stack exhaustion.
Reviewed the split-out part 1 against acf9004. Checks on this head: cargo fmt --check clean, cargo clippy --all-targets --all-features clean, cargo test --all-features green (36 + 1 + 131 + 83, doctests).
What I verified beyond the test suite:
to_string / to_pretty_string produce the same bytes as serde_json::to_string(&raw) / to_string_pretty for floats (1e21, 1e-7, 5e-324, -0.0, 1.7976931348623157e308), control-char / non-ASCII escapes, and empty nested containers.
convert_to_comparable byte order agrees with partial_cmp across the one-byte / three-byte depth boundary (depths 253–256, 300, and sibling arrays at depth 254/255).
deserialize_ignored_any skips container-valued unknown struct fields and IgnoredAny array elements, and the following elements still decode.
Stack use at MAX_NESTING_DEPTH: in release every recursive API (to_value, from_slice, from_raw_jsonb::<serde_json::Value>, serde / JsonWriter output, partial_cmp, contains, strip_nulls, convert_to_comparable, jaq, text parse, Value Drop/Display) handles 512 nested arrays and objects on a 512 KiB thread stack. In a debug build on a 2 MiB stack, to_value, from_slice, from_raw_jsonb, partial_cmp, contains, jaq and Value Drop overflow at 512 levels (to_string, serde output, strip_nulls, convert_to_comparable and the text parser are fine). Databend runs tokio threads at 20 MiB and std threads at 5 MiB, so this is not a blocker, but see the doc nit below.
Verdict: one should fix, rest are nits. Happy to approve once the bytes round trip is handled or explicitly deferred.
S1 [should] serialize_bytes and deserialize_bytes / deserialize_byte_buf disagree
src/core/databend/ser.rs now encodes serialize_bytes as a Binary extension (good, it was todo!()), but src/core/databend/de.rs:525-537 still implement deserialize_bytes / deserialize_byte_buf as visitor.visit_string(self.read_string()?), and read_str formats an extension payload with Display, i.e. hex text. So a serde_bytes-style type does not round-trip:
Suggested fix: in deserialize_bytes / deserialize_byte_buf, read the scalar JEntry; if it is EXTENSION_TAG and ExtensionValue::decode gives Binary(b), call visitor.visit_borrowed_bytes(b) (the payload borrows 'de), otherwise keep the current string path. Add a round-trip test next to test_serialize_bytes_as_binary in tests/it/encode.rs. Copilot flagged the same thing on the previous head.
Nits
src/constants.rs doc on MAX_NESTING_DEPTH: "about 1.5 KiB of stack per level ... fits tokio's default 2 MiB" only holds for optimized builds; debug to_value needs roughly 4–9 KiB per level and overflows a 2 MiB thread at 512 (the tests use a 64 MiB stack for that reason). Worth stating "in release builds; debug builds need a larger stack" so nobody reads it as a guarantee.
src/value.rsDisplay: Value::Number(Float64(NaN|±inf)) now prints null (was NaN/inf), and strings are JSON-escaped instead of {:?} (fixes \u{7f}, \0-style non-JSON output). Both are correct, but they are user-visible changes of Value::to_string() and should be listed under "Breaking changes" in the description.
src/parser.rsfrom_slice: a decode failure of any kind falls back to parse_value(buf), so a valid JSONB nested deeper than 512 surfaces as a text Syntax error rather than RecursionLimitExceeded. Not new behavior in spirit, but the error is misleading now that the limit exists; consider only falling back when the error is not RecursionLimitExceeded.
src/core/databend/json_writer.rs relies on serde_json >= 1.0.147 (zmij) for the float output to match Serialize; Cargo.toml still says serde_json = "1.0". Bumping the minimum makes the "same JSON as serde_json" claim hold for downstream resolvers too.
src/functions/array.rs: the element_counts / retain_by_membership rewrite is a clean-up rather than hardening. Behavior is unchanged as far as I can tell (array_distinct on a scalar, array_overlap early exit), just noting it for anyone bisecting later.
PR description: "the benches from #PR3" should point at the actual PR number once it exists.
`deserialize_bytes` and `deserialize_byte_buf` read binary values as
their hexadecimal text, so bytes written by `serialize_bytes` came back as
the string "010203" instead of `[1, 2, 3]`, which byte-only visitors
rejected. They now give binary values as bytes, and strings as their
UTF-8 bytes, as in serde_json.
`deserialize_ignored_any` skipped the value's bytes unread. At the top
level it accepted any input, even an empty one, and inside containers it
accepted invalid values and bypassed the nesting limit. Ignored values are
now read like any other.
`from_slice` retried any JSONB decoding error as JSON text, so JSONB
nested deeper than 512 levels failed with a misleading syntax error. It
now returns `RecursionLimitExceeded` directly.
Also state in the `MAX_NESTING_DEPTH` docs that the stack estimate holds
for release builds only; debug builds overflow a 2 MiB stack at 512
levels.
The reason will be displayed to describe this comment to others. Learn more.
Re-reviewed e6d23d7 (the branch was rewritten since my last comments).
S1 (scalar header wrapping a container JEntry) is fixed in every decoder: to_value, from_slice, serde Serialize, from_raw_jsonb, jaq and partial_cmp all return an error on 100 000 nested wrappers instead of overflowing the stack. Verified locally.
The JSONPath changes (parser depth/operator limits, selector budgets, number literals, Display round-tripping) are no longer part of this PR, so S2, S3 and S4 are out of scope here. S2 (a flat || chain of ~10 000 terms overflows the stack in Selector::eval_filter_expr) and S3 (owned string literals cloned per compared item) also exist on main, so they are worth a follow-up PR; the parser-fold counter from my previous comment handles S2 without rejecting a&& field names.
convert_to_comparable now keeps the one-byte depth prefix for depths below 255 and escapes deeper ones with a 0xFF marker, so existing sort keys stay byte-compatible. Good.
Local checks on this head: cargo clippy --all-targets --all-features clean, cargo test --all-features green (36 + 1 + 133 + 83 tests, doctests). CI build is green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 3 stacked PRs. This PR fixes undefined behavior, panics, hangs and wrong results that fuzzing found in jsonb.
Changes
JEntrys and payloads against the data. Malformed bytes returnInvalidJsonb,InvalidEOForInvalidUtf8instead of panicking or reading past the end.RecursionLimitExceeded).Value::to_vecandwrite_to_vecreturnResultand fail withValueTooLargewhen a length does not fit the JSONB length fields.to_stringwrites the same JSON as serde_json throughJsonWriter. Comparison and comparable encoding walk undecoded entries.convert_to_comparableorders values nested more than 255 levels correctly. Values nested less than 255 levels encode exactly as onmain.serialize_byteswrites a binary value, whichdeserialize_bytesanddeserialize_byte_bufread back as bytes. Both used to be broken:serialize_byteswastodo!(), and the byte deserializers returned binary values as hexadecimal text.deserialize_ignored_anyreads the ignored value and checks it, instead of skipping its bytes unread.from_slicereturnsRecursionLimitExceededfor JSONB nested too deeply, instead of retrying the bytes as JSON text and reporting a syntax error.src/functions/array.rsis reorganized aroundelement_countsandretain_by_membership; this is a clean-up and does not change behavior.Breaking changes
Value::to_vecandValue::write_to_vecreturnResult.Value'sDisplaywrites JSON:NaNand the infinities print asnullinstead ofNaN/inf, and strings are escaped like serde_json instead of with{:?}, which wrote non-JSON escapes such as\0and\u{7f}.deserialize_bytesanddeserialize_byte_bufgive a string as its UTF-8 bytes, as in serde_json, instead of as a string.Performance
Measured with criterion and the benches in the Databend release profile (
opt-level = "s",codegen-units = 1,lto = "thin"). Each side ran twice, and the faster median is compared.partial_cmpto_stringcontainsfrom_sliceconvert_to_comparableto_valuestrip_nullsselect_by_pathselect_array_by_path($.**)array_distinct/array_overlapget_by_nameOf the 105 benchmarks, 17 are more than 5% faster, 32 are more than 5% slower, and 56 are within 5%. Building, encoding and JSONPath parsing are within about 5%.
canada/containsis not measured, because one call takes about 36 s.Tests
New tests:
malformed.rs,invalid_utf8.rs,json_text.rsandallocation.rs.--no-default-features --features databend