Skip to content

fix(json): consume v0 SharedData to keep native stream aligned - #2004

Open
BetterAndBetterII wants to merge 3 commits into
ClickHouse:mainfrom
BetterAndBetterII:fix/json-v0-shared-data
Open

BetterAndBetterII wants to merge 3 commits into
ClickHouse:mainfrom
BetterAndBetterII:fix/json-v0-shared-data

Conversation

@BetterAndBetterII

@BetterAndBetterII BetterAndBetterII commented Sep 4, 2026 •

Copy link
Copy Markdown

Summary

  • JSON deprecated (v0) object decoding only skipped SharedData UInt64 offsets and left the Array(Tuple(String, String)) payload in the native stream, desyncing every later column (makeslice, unexpected boolean/packet).
  • Decode the full SharedData array and merge overflow paths into the scanned JSON object.
  • Adds unit coverage that asserts stream alignment and merged overflow values.

Fixes #1854

Test plan

  • go test -count=1 -timeout 60s -run 'TestJSONDeprecatedSharedData' ./lib/column/

Note

Medium Risk
Changes native-protocol JSON v0 decode and value assembly; incorrect binary parsing could misread overflow paths, but scope is limited to deprecated object format and fixes a critical stream desync bug.

Overview
Fixes JSON v0 (deprecated object) native decoding when ClickHouse sends overflow dynamic paths in SharedData as Array(Tuple(String, String)). The decoder previously advanced the stream by UInt64 offsets only and left the tuple payload behind, which desynced every following column in the block (#1854).

decodeSharedData_v1 now decodes the full shared-data column, stores it on JSON.sharedData, and mergeSharedDataRow (via rowAsJSON) applies per-row (path, Dynamic binary) pairs into the returned chcol.JSON, including a small binary-type reader for common Dynamic encodings. Reset clears shared data between batches. New tests assert stream alignment after decode and correct merged overflow fields across rows.

Reviewed by Cursor Bugbot for commit 3cb33ca. Bugbot is set up for automated code reviews on this repo. Configure here.

JSON deprecated object decoding only skipped SharedData offsets and left
the Array(Tuple(String, String)) payload in the wire stream, which
desynced later columns (ClickHouse#1854). Decode the full array and merge overflow
paths into the scanned JSON object.
@CLAassistant

CLAassistant commented Sep 4, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

r := proto.NewReader(bytes.NewReader([]byte(raw)))
t, err := readSharedBinType(r)
if err != nil {
return raw, "String"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Must fix — do not return raw binary as a String value for an unsupported type

When readSharedBinType or readSharedBinValue fails, the function returns the encoded bytes (type index + payload) as a String value, so the caller gets corrupt data and no error.

  • This occurs for each type that is not in the switch. Array(JSON) (0x30, an array of objects) is frequent in JSON data. DateTime64, Decimal, Tuple, Map, UUID, and LowCardinality have the same result.
  • Bytes that stay in the reader after the value are ignored.
  • make([]any, 0, n) uses n from the payload with no limit. A bad length causes a makeslice panic in library code.

Decode the values in decodeSharedData_v1, where an error can be returned, and wrap the error with the path name and the type index. As an alternative, add the remaining type indexes. Limit n to the number of bytes that remain.

Comment thread lib/column/json.go
obj.SetValueAtPath(path, col.Row(row, false))
}

c.mergeSharedDataRow(obj, row)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Should fix — merge shared data for struct and map scans also

The PR promises that overflow paths are merged into the scanned object, but only rowAsJSON does the merge. A scan into *struct or *map[string]any drops the overflow paths with no error.

  • scanRowObject sends these destinations to scanIntoStruct and scanIntoMap.
  • fillStruct, fillMap, valueAtPath, and pathHasNestedValues read only typedPaths and dynamicPaths.
  • Result: the same row gives different data for *chcol.JSON and for *map[string]any.

Make the shared-data paths available to these functions, and add a test for each destination type.

// skip only the UInt64 offsets and leave the Tuple payload in the stream,
// which desyncs every later column (makeslice panics, "unexpected value N
// for boolean", "unexpected packet N").
func TestJSONDeprecatedSharedDataConsumedAndMerged(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Should fix — add a regression test against a real server

The tests make the SharedData bytes with the same assumptions as the decoder, so they do not prove that the format agrees with the server.

  • Add tests/issues/issue_1854_test.go. Use a JSON(max_dynamic_paths=N) column, insert more than N paths, and select the rows with a second column after the JSON column.
  • Make sure that the server sends serialization version 0, and use CheckMinServerVersion to skip on servers that cannot.
  • Run the native and database/sql APIs, on TCP and HTTP. Use MergeTree ORDER BY tuple(), not Memory.
  • Include an array of objects and a scan into a map, to show the two problems in the other comments.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Claude review

The PR decodes the full JSON v0 SharedData column (Array(Tuple(String, String))) and does not skip only the UInt64 offsets. This repairs the stream desync from #1854. The stream fix is correct, but the new value merge is not complete, thus the verdict is request changes.

Key concerns:

  • A shared-data value with a type that the new decoder does not know (for example Array(JSON), DateTime64, Decimal, Tuple) is returned as raw binary bytes with the type String. The caller gets incorrect data and no error.
  • Only rowAsJSON merges the shared data. A scan into a struct or a map does not see the overflow paths.
  • There is no test against a real server and no tests/issues/issue_1854_test.go.

Verified by reading:

  • The binary type indexes agree with DataTypesBinaryEncoding.cpp.
  • The Nullable flag and the Array varuint length agree with serializeBinary.
  • The change is in the column decoder, thus it applies to native TCP and HTTP, and to the native and database/sql APIs.

Blind spots:

Verdict: ⚠️ Request changes

Inline comments are attached to the relevant lines. This summary updates in place on re-review.

@kavirajk
kavirajk requested review from a team and polyglotAI-bot as code owners November 17, 2026 07:01

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 3cb33ca. Configure here.

Comment thread lib/column/json.go
}

c.mergeSharedDataRow(obj, row)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Map and struct scans skip overflow paths

Medium Severity

Overflow paths from SharedData are merged only when materializing a *chcol.JSON via rowAsJSON. Scanning into a map or struct still walks only typedPaths and dynamicPaths, so those destinations omit the same overflow values after a successful v0 decode.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3cb33ca. Configure here.

return raw, "String"
}
return v, t.name
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested JSON overflow decoded as String

Medium Severity

readSharedBinType only handles a subset of binary type indexes (scalars, Array, Nullable, Dynamic). Types that JSON inference actually stores in SharedData—especially nested JSON / Array(JSON), and also Tuple, Map, UUID, and date-times—hit the default branch. decodeSharedDataValue then swallows that error and stores the raw payload as String, so overflow objects and arrays of objects scan as binary garbage instead of structured values.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3cb33ca. Configure here.

@github-actions

Copy link
Copy Markdown
Contributor

AI review gate — Cursor Bugbot found issues

  • Cursor Bugbot found issues that are not resolved (check concluded 'neutral').

Unresolved findings (2):

What to do:

  • Fix the findings and push. The reviewer re-reviews the new commit and the gate re-runs by itself.
  • Finding wrong, or reviewer stuck? Ask a human with write access who is not the PR author to approve this exact commit, then re-run the gate. Their approval stands in for the AI review (a new push needs a new approval).

Re-run the gate with “Re-run jobs” after an approval or a finished review. A push re-runs it automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic: makeslice: len out of range when reading JSON column with deprecated v0 wire format

3 participants