Repository navigation
fix(json): consume v0 SharedData to keep native stream aligned - #2004
BetterAndBetterII wants to merge 3 commits into
Conversation
JSON deprecated object decoding only skipped SharedData offsets and left the Array(Tuple(String, String)) payload in the wire stream, which desynced later columns (ClickHouse#1854). Decode the full array and merge overflow paths into the scanned JSON object.
| r := proto.NewReader(bytes.NewReader([]byte(raw))) | ||
| t, err := readSharedBinType(r) | ||
| if err != nil { | ||
| return raw, "String" |
There was a problem hiding this comment.
❌ Must fix — do not return raw binary as a String value for an unsupported type
When readSharedBinType or readSharedBinValue fails, the function returns the encoded bytes (type index + payload) as a String value, so the caller gets corrupt data and no error.
- This occurs for each type that is not in the switch.
Array(JSON)(0x30, an array of objects) is frequent in JSON data.DateTime64,Decimal,Tuple,Map,UUID, andLowCardinalityhave the same result. - Bytes that stay in the reader after the value are ignored.
make([]any, 0, n)usesnfrom the payload with no limit. A bad length causes amakeslicepanic in library code.
Decode the values in decodeSharedData_v1, where an error can be returned, and wrap the error with the path name and the type index. As an alternative, add the remaining type indexes. Limit n to the number of bytes that remain.
| obj.SetValueAtPath(path, col.Row(row, false)) | ||
| } | ||
|
|
||
| c.mergeSharedDataRow(obj, row) |
There was a problem hiding this comment.
The PR promises that overflow paths are merged into the scanned object, but only rowAsJSON does the merge. A scan into *struct or *map[string]any drops the overflow paths with no error.
scanRowObjectsends these destinations toscanIntoStructandscanIntoMap.fillStruct,fillMap,valueAtPath, andpathHasNestedValuesread onlytypedPathsanddynamicPaths.- Result: the same row gives different data for
*chcol.JSONand for*map[string]any.
Make the shared-data paths available to these functions, and add a test for each destination type.
| // skip only the UInt64 offsets and leave the Tuple payload in the stream, | ||
| // which desyncs every later column (makeslice panics, "unexpected value N | ||
| // for boolean", "unexpected packet N"). | ||
| func TestJSONDeprecatedSharedDataConsumedAndMerged(t *testing.T) { |
There was a problem hiding this comment.
The tests make the SharedData bytes with the same assumptions as the decoder, so they do not prove that the format agrees with the server.
- Add
tests/issues/issue_1854_test.go. Use aJSON(max_dynamic_paths=N)column, insert more than N paths, and select the rows with a second column after the JSON column. - Make sure that the server sends serialization version 0, and use
CheckMinServerVersionto skip on servers that cannot. - Run the native and
database/sqlAPIs, on TCP and HTTP. UseMergeTree ORDER BY tuple(), notMemory. - Include an array of objects and a scan into a map, to show the two problems in the other comments.
🤖 Claude reviewThe PR decodes the full JSON v0 SharedData column ( Key concerns:
Verified by reading:
Blind spots:
Verdict: Inline comments are attached to the relevant lines. This summary updates in place on re-review. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Reviewed by Cursor Bugbot for commit 3cb33ca. Configure here.
| } | ||
|
|
||
| c.mergeSharedDataRow(obj, row) | ||
|
|
There was a problem hiding this comment.
Map and struct scans skip overflow paths
Medium Severity
Overflow paths from SharedData are merged only when materializing a *chcol.JSON via rowAsJSON. Scanning into a map or struct still walks only typedPaths and dynamicPaths, so those destinations omit the same overflow values after a successful v0 decode.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 3cb33ca. Configure here.
| return raw, "String" | ||
| } | ||
| return v, t.name | ||
| } |
There was a problem hiding this comment.
Nested JSON overflow decoded as String
Medium Severity
readSharedBinType only handles a subset of binary type indexes (scalars, Array, Nullable, Dynamic). Types that JSON inference actually stores in SharedData—especially nested JSON / Array(JSON), and also Tuple, Map, UUID, and date-times—hit the default branch. decodeSharedDataValue then swallows that error and stores the raw payload as String, so overflow objects and arrays of objects scan as binary garbage instead of structured values.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 3cb33ca. Configure here.
AI review gate — Cursor Bugbot found issues
Unresolved findings (2): What to do:
Re-run the gate with “Re-run jobs” after an approval or a finished review. A push re-runs it automatically. |


Summary
UInt64offsets and left theArray(Tuple(String, String))payload in the native stream, desyncing every later column (makeslice, unexpected boolean/packet).Fixes #1854
Test plan
go test -count=1 -timeout 60s -run 'TestJSONDeprecatedSharedData' ./lib/column/Note
Medium Risk
Changes native-protocol JSON v0 decode and value assembly; incorrect binary parsing could misread overflow paths, but scope is limited to deprecated object format and fixes a critical stream desync bug.
Overview
Fixes JSON v0 (deprecated object) native decoding when ClickHouse sends overflow dynamic paths in SharedData as
Array(Tuple(String, String)). The decoder previously advanced the stream by UInt64 offsets only and left the tuple payload behind, which desynced every following column in the block (#1854).decodeSharedData_v1now decodes the full shared-data column, stores it onJSON.sharedData, andmergeSharedDataRow(viarowAsJSON) applies per-row(path, Dynamic binary)pairs into the returnedchcol.JSON, including a small binary-type reader for common Dynamic encodings.Resetclears shared data between batches. New tests assert stream alignment after decode and correct merged overflow fields across rows.Reviewed by Cursor Bugbot for commit 3cb33ca. Bugbot is set up for automated code reviews on this repo. Configure here.