Repository navigation
Description
Observed
- Create a
JSONcolumn and insert 4 rows with heterogeneous value types across the same JSON
paths (e.g.latency_msis an array in one row, a string in another). SELECTthe column back using the Native protocol withclickhouse-gov2.- The driver panics or returns a wire decode error:
panic: runtime error: makeslice: len out of range(in production with large datasets)read data: failed to decode block: unexpected value N for booleanclickhouse [process]: unexpected packet N
Expected behaviour
The SELECT should return all rows without error.
Code example
package main
import (
"context"
"fmt"
"log"
"github.com/ClickHouse/clickhouse-go/v2"
)
// rows contains heterogeneous JSON documents. The same key appears with
// different value types across rows (Int64, String, Array, Bool, Float64,
// nested object, timestamp) so that each JSON path acquires multiple
// inferred CH types — the condition that triggers the discriminator
// index shift in the v0 deprecated decoder.
var rows = []string{
`{"latency_ms":[]}`,
`{"cpu_pct":1,"region":"x","request_id":1,"response_times":[],"service":"x","tags":[]}`,
`{"counts":[],"created_at":"x","ended_at":"x","errors":[],"labels":[],"span_ids":[],"started_at":"x","trace_id":"x","user_id":"x"}`,
`{"attachments":[],"config":{"host":"host-149","nested":{"extra":null,"name":"svc-147","parent":null},"value":null},"context":{"name":"ctx-273","parent":null,"root":false},"durations":[],"logged_at":"x","metadata":{"active":true,"ids":[null,null],"ts":"2026-05-04T15:18:01Z"},"metrics":{"primary":{"count":542265,"enabled":true,"parent":null},"samples":[18485,18893],"secondary":{"extra":null,"label":"label-166","parent":null}},"org_id":"x","stack_frames":[],"updated_at":"x"}`,
}
func main() {
conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{"127.0.0.1:9002"},
Auth: clickhouse.Auth{Database: "default", Username: "default", Password: "password"},
Protocol: clickhouse.Native,
})
if err != nil {
log.Fatal(err)
}
defer conn.Close()
ctx := context.Background()
if err := conn.Ping(ctx); err != nil {
log.Fatalf("ping: %v", err)
}
exec := func(q string) {
if err := conn.Exec(ctx, q); err != nil {
log.Fatalf("%s\n %v", q, err)
}
}
exec("DROP TABLE IF EXISTS test_json_table")
exec("CREATE TABLE test_json_table (raw JSON(max_dynamic_paths=32)) ENGINE = MergeTree ORDER BY tuple()")
batch, err := conn.PrepareBatch(ctx, "INSERT INTO test_json_table (raw)")
if err != nil {
log.Fatal(err)
}
for _, row := range rows {
if err := batch.Append(row); err != nil {
log.Fatal(err)
}
}
if err := batch.Send(); err != nil {
log.Fatal(err)
}
result, err := conn.Query(ctx, "SELECT raw FROM test_json_table SETTINGS max_memory_usage=0")
if err != nil {
log.Fatal(err)
}
defer result.Close()
var n int
for result.Next() {
var raw clickhouse.JSON
if err := result.Scan(&raw); err != nil {
log.Fatal(err)
}
n++
}
if err := result.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("read %d rows OK\n", n)
}Run with:
docker run --rm -p 9002:9000 -p 8123:8123 \
-e CLICKHOUSE_PASSWORD=password \
clickhouse/clickhouse-server:26.2
go run .Error log
read data: failed to decode block from 127.0.0.1:9002 (conn_id=1, compression=none): unexpected value 97 for boolean
In production with larger datasets the panic form is observed:
panic: runtime error: makeslice: len out of range
goroutine 120050 [running]:
github.com/ClickHouse/clickhouse-go/v2/lib/column.(*JSON).decodeObjectData_v1
lib/column/json_deprecated.go:111
github.com/ClickHouse/clickhouse-go/v2/lib/column.(*JSON).Decode
lib/column/json.go:768
github.com/ClickHouse/clickhouse-go/v2/lib/column.(*Dynamic).decodeData_v1
lib/column/dynamic_deprecated.go:174
github.com/ClickHouse/clickhouse-go/v2/lib/column.(*Dynamic).Decode
github.com/ClickHouse/clickhouse-go/v2/lib/column.(*Array).Decode
lib/column/array.go:227
github.com/ClickHouse/clickhouse-go/v2/lib/proto.(*Block).Decode
lib/proto/block.go:232
github.com/ClickHouse/clickhouse-go/v2.(*connect).readData
conn_process.go:107
Root cause (hypothesis)
ClickHouse 26.2 sends JSON column data in the deprecated v0 wire format
(output_format_native_use_flattened_dynamic_and_json_serialization=0 by default). In this format
each JSON path with multiple inferred types is encoded as a Dynamic column with N variant types.
Dynamic.decodeHeader_v1 reads the N type names from the wire, then inserts "SharedVariant" and
re-sorts the list alphabetically. This shifts the driver's discriminator→column index mapping
relative to what the server wrote. For example, if a path has variants ["Array(Int64)", "String"],
the server uses discriminator 1 = String, but the driver maps discriminator 1 → SharedVariant
(which sorts between "Array..." and "String..."). Every subsequent column read is then offset,
causing the decoder to consume the wrong bytes — producing a makeslice panic, an unexpected-boolean
error, or an unexpected-packet error depending on what those bytes look like to the next decoder.
Details
Environment
clickhouse-goversion: v2.45.0- Interface: ClickHouse API (
clickhouse.Open) - Go version: 1.26.0
- Operating system: macOS / darwin arm64
- ClickHouse version: 26.2.15.4
- Is it a ClickHouse Cloud? No (also reproduces on self-hosted)
- ClickHouse Server non-default settings: none
(output_format_native_use_flattened_dynamic_and_json_serialization=0is the default) CREATE TABLEstatement:
CREATE TABLE test_json_table (raw JSON(max_dynamic_paths=32))
ENGINE = MergeTree ORDER BY tuple()
Activity
abonander commentedon May 5, 2026
Are you certain this is the correct stack trace for this panic? There's no
make()call at that line to produce this panic:clickhouse-go/lib/column/json_deprecated.go
Line 111 in 1422cc5
amalakar commentedon May 5, 2026
@abonander there is a possibility the stack trace is not accurate for the issue I am describing. I tried to piece together what I saw in production logs, vs what I can reproduce.
abonander commentedon May 5, 2026
When running in a debugger, I found the boolean error is in
decodeBlockInfo()here:clickhouse-go/lib/proto/block.go
Lines 267 to 269 in 13b74cb
Not really sure what that means, yet. It seems like a prior packet may not have been fully consumed and we're reading junk data.
ShamsRKhan commentedon Aug 13, 2026
We hit this in production-shaped testing, root-caused it in the driver source, and have deterministic minimal repros for the two routes that trigger it. Posting here rather than opening a new issue because the symptoms in this thread —
makeslice: len out of range,unexpected value N for boolean,unexpected packet N, and the "junk data" a commenter mentions — all fall out of the same two gaps.Environment: server 26.3.17, driver v2.48.0, native protocol.
Root cause
(a)
lib/column/json_deprecated.go— shared data is skipped, desyncing the stream.After decoding typed and dynamic paths,
Decodereads only the outer offsets and never consumes the shared-data payload:That comment is accurate. One
UInt64per row is the offsets array; the values it points at are left in the stream. Every column decoded after the JSON column in that block then reads from the wrong position. That is the mechanism behindunexpected packet Nand themakeslicepanic in this thread — a length field parsed out of misaligned bytes is arbitrary, so where you land depends on what follows the JSON column in the block.(b)
lib/column/sharedvariant.go— SharedVariant is a String passthrough.Every method delegates to
stringData. So any value stored as SharedVariant — which happens when a path exceedsmax_dynamic_types— is handed to Go as its raw type-tagged binary inside a string, with no error fromQuery,Scan, orrows.Err().This is the nastier of the two. No panic, no failed query, just wrong values. We observed
"\x0e\x00\x00\x00\x00\x00\x00&@"— a0x0Etype tag followed by little-endian float6411.0— being served as a value on a live dashboard.Reproduction
Plain Go, no deps beyond the driver. R2 runs before R1 because R1's stream desync can leave the pooled connection broken.
Observed
Verbatim output of the program above (server 26.3.17, driver v2.48.0):
R2, no error anywhere on the path:
The value that lost the dynamic-type slot arrives as raw bytes; its neighbor decodes correctly — same path, same column, same result set — so a single result set mixes correct values and garbage, and nothing distinguishes them at the call site.
R1:
Four of the twelve paths (
k08–k10,satellites) are silently ABSENT from the decoded row — the shared-data overflow — and the stream then desyncs atrows.Err. Which face R1 shows varies with block layout: across variations of this shape we have also seen the query fail outright withunexpected packet 4and the pooled connection stay broken for subsequent unrelated queries. Single part, single row, noFINAL, no type mixing needed for any of them.One more data point on severity: in production-shaped testing we watched a live dashboard serve the raw-bytes values for a while, and the symptom later vanished on its own after background merges reorganised the parts. Readability tracks per-part storage layout, so the same query over the same logical data can return correct values or garbage depending on when it runs. That is a large part of why this is hard to catch in testing.
Master
Both files are byte-identical on master — checked against pseudo-version
v2.48.1-0.20260812104646-3a8dafcff5e2. Neither gap is fixed there.The wire format is decodable
clickhouse-connect (Python) fully decodes both encodings — see
datatypes/binary_value.py. So this is a driver gap rather than anything missing server-side, and there is a working reference implementation to check against.Two asks
output_format_native_use_flattened_dynamic_and_json_serializationon when the server is >= 25.6. We verified end-to-end that it avoids both routes. That closes the practical problem for most users without waiting on a decoder.Happy to test patches against our reproduction, or to open a separate issue for either root cause if you would rather track them apart.
polyglotAI-bot commentedon Nov 14, 2026
Recommendation: Keep open — fix needed
The V1
JSONdecoder has two defects. Open PR #2004 fixes the first one (the skipped shared data that breaks the stream), but not the second one (SharedVariantvalues come back as raw bytes, with no error); the review on #2004 requests changes. Next step: finish #2004 and fix theSharedVariantdecoding too (in #2004 or in a separate PR). #2114 can be closed as a duplicate of this issue.main: Reproduced on main —d124d2b0, server 26.9.1.1629, the reporter's rows and three smaller cases, no extra settings.SharedVariantdefect). Open PR fix(json): consume v0 SharedData to keep native stream aligned #2004 proposes a fix: with its diff applied, cases A and C below have no error, but case B still returns raw bytes. Related: database/sql (std) driver reuses a connection whose read stream is left mid-response after a protocol decode error #1988 (database/sqlputs the broken connection back in the pool; open PR fix: retire connections after query protocol errors #2027 proposes a fix).bugAnalysis: This client sends protocol revision 54460 (
lib/proto/const.go:23), so whenoutput_format_native_use_flattened_dynamic_and_json_serializationis 0 (the default), the server sendsJSONwith the V1 object serialization. Defect 1:decodeObjectData_v1reads only the shared-data offsets and leaves the paths and values in the stream (lib/column/json_deprecated.go:117-118), so all later data is read from the wrong position (cases A and C). The reporter's rows have 33 paths formax_dynamic_paths=32, so 1 path goes to the shared data. Defect 2:SharedVariantis aStringpassthrough (lib/column/sharedvariant.go:10-13), so a value that is overmax_dynamic_typescomes back as its type-tagged binary encoding, with no error (case B:0x0eis theFloat64tag, then 12.5 as little-endian bytes). The discriminator-shift hypothesis in the report does not agree with the run: case D has one path with four types and no shared data, and it decodes correctly, so the variant order after the sort inlib/column/dynamic_deprecated.go:108-109agrees with the server. A workaround: setoutput_format_native_use_flattened_dynamic_and_json_serialization=1inOptions.Settings; then all cases decode correctly.Reproduction: complete program and output
Output on
main(d124d2b0), ClickHouse 26.9.1.1629:Output on
mainwith the diff of PR #2004 applied:Automated triage by Polyglot AI. A maintainer makes the final decision.