Repository navigation
Conversation
Add tests/clickhouse-test-runner: a Go program that imitates clickhouse-client well enough for the upstream runner (tests/clickhouse-test) to drive it. It runs every query through clickhouse-go over the native or HTTP protocol, scans each row into ColumnType.ScanType(), and writes the values as TabSeparated, so the upstream .reference files check the decoding of the driver. - allowlist.txt: 9,903 tests over both protocols, 129 over native only, and 920 known failures in 30 groups (client bugs and shim gaps), each with a reproduction. - clickhouse-version pins the server version and the upstream tests together; a bump is an explicit pull request. - upstream-sql-tests.yml: nightly, manual, and on changes to the harness; native and HTTP, 8 shards each. - README.md and AGENTS.md describe local runs, the triage rules, how to grow the allowlist, and how to bump the version. Refs ClickHouse/integrations#810 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Summary
|
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 77a0c5f: see the summary comment and the inline threads.
AI review gate — AI review passedClean review from: Cursor Bugbot. |
- Split the test selector into several arguments of at most 64 KiB. Linux limits one argument to 128 KiB, and one selector for the full allowlist is about 360 KB, so a run with one shard failed with "Argument list too long". Both run scripts share selectors.sh now. - Read FORMAT as the output-format clause only where it can be one: at the end of a query, before an optional SETTINGS clause, and for an INSERT only before any SELECT. `SELECT format FROM t` was read as an unsupported format, and `INSERT ... SELECT format ...` could take the rest of the script as inline data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Treat only the bare io.EOF sentinel (ServerEndOfStream) as a statement without a result. clickhouse-go wraps read failures, such as a disconnect or a truncated block, with %w, and errors.Is matched those too: a failed query could then look like an empty result. - hasWord skips ClickHouse `# ` and `#!` comments, as the splitter does. - Tests for aliases and columns named `format`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SELECT check of the last commit also rejected `INSERT ... SELECT ... FROM input(...) FORMAT TSV`, which has inline data. Instead, read the word after FORMAT: a SQL keyword (FROM, WHERE, AS, ...) shows an identifier named `format`; any other word is a format name. hasWord is not used anymore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of b71abce: see the summary comment and the inline threads.
- A query: a clause keyword after FORMAT (ASC, DESC, ...) shows an identifier, so `SELECT 1 AS format ORDER BY format DESC` is not a FORMAT DESC clause. - An INSERT ... SELECT: the FORMAT clause comes after the top-level FROM of the SELECT. Before it, `format` is in the select list (`SELECT format x FROM s`, `SELECT format DIV 2 FROM s`). GLOBAL, DIV, MOD and other operator words are clause keywords too. - Unit tests for these cases, and live tests that compare the queries with the server over both protocols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI review gate — auto-approval withheldThe AI review gate is green, but the approver did not approve What to do: ask someone with write access who is not the PR author to approve this pull request. History (latest 5 heads, newest first)
|
A known client bug (the pool race, [native-pool-race-new-connection] in allowlist.txt) makes random tests with SET fail now and then over the native protocol: about 40 of 10,000 tests per CI run, a different set each time. With RETRY_FAILED_ONCE=1, the run script runs the failed tests once more. A test that passes on the second run is a warning; a test that fails twice fails the job. CI sets it; local runs that grow or check the allowlist do not. Move five tests that fail on almost every CI run into their known-failure groups: three hit the pool race at the same point each time, and two hit the HTTP server-hello bug in a LogsQL session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sixteen tests failed in two or more of five CI runs, or twice in one run with the retry. They hit the pool race ([native-pool-race-new-connection]) too often for a retry to keep CI stable, so they move into that group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With about 40 race victims per native run, one retry was not enough: now and then one victim is hit twice (05218_join_use_nulls_lc_key_full_sorting_merge failed at two different points, both with join_use_nulls lost). The run script now takes RETRY_FAILED=<n> (CI: 2). A test that passes on a retry is a warning; a test that fails on every attempt fails the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of f0c36b2: see the summary comment and the inline threads.
`SELECT 1 AS format, format x` was read as the clause `FORMAT x`. A FORMAT clause follows a complete query; when the text before the word ends with a comma, an opening parenthesis, an operator or a keyword such as SELECT, the word starts an expression, so it is an identifier. The same check applies to INSERT inline data. Over all 190,722 statements of the allowlisted tests, the old and the new parser give the same result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-star startsExpression looked back through whitespace only, so a FORMAT clause after a block comment (`*/`) or after a select-star (`SELECT * FORMAT TSV`, `FROM t SELECT * FORMAT TSV`) was read as an identifier. It now takes the last significant token before the word, outside comments and quotes. A `*` after SELECT, DISTINCT, a comma, `(` or `.` is a select-star, which can end a query; another `*` multiplies. Over all 190,722 statements of the allowlisted tests, the parser gives the same result as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 9381219: see the summary comment and the inline threads.
The invocation log (CLICKHOUSE_CLIENT_CLI_LOG) recorded the raw command line, so the value of --password went into the file in clear. CI uploads that file as an artifact when a shard fails, and GitHub does not mask secrets in artifacts. redactPassword walks the arguments by the rules of parseArgs and replaces the value that parseArgs takes as the password by `***`. The other log records hold no password; the new live test checks the whole log over both protocols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of ad93c4c: see the summary comment and the inline threads.
If the runner printed a failure for test A and then exited abnormally (for example killed with 137) before test B ran, the retry of A alone could make the shard pass, and B never ran. Now an attempt counts only when every one of its tests has a result line and the runner exited with 0 or 1; otherwise the run fails at once, and the error names the tests without a result. The retry logic moves to scripts/retry.sh. retry_test.go runs it with fake runners: all pass, a flaky test, a broken test, a runner killed after one failure, an attempt with a missing result, a killed retry, and a success with a missing result. Without the completeness check, the three cases with a missing result fail. Checked with the real runner on the pinned server: a full native shard (1,251 tests, 1,251 results) and a forced retry of a failing test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 04056_ast_json_limits fails over HTTP on every attempt in CI: after its
expected TOO_BIG_AST error, clickhouse-go opens a new HTTP connection,
and its server-info query runs in the session, where the test set
max_ast_elements = 15 ("failed to query server hello ... AST is too
big"). Group [http-server-hello-in-session].
- 05046_trino_dialect_review_regressions failed on all three attempts of
one native run, at different statements, with the Trino dialect lost.
Group [native-pool-race-new-connection].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of a865fd8: see the summary comment and the inline threads.
- The splitter trimmed the end of every statement, also INSERT inline data, where a trailing tab or space is data (in TSV, a trailing tab is an empty last column). Now inline data loses only the line breaks before the empty line that ends it. - parseSetAssignments read `SET /* c */ max_threads = 3` as the setting `/* c */ max_threads`, so the command-line value of max_threads stayed on later queries and overrode the SET. Comments outside quotes are now dropped before the names and values are read. Unit tests for both; live tests, over both protocols, that fail without the fixes: a TSV row with a trailing tab, and `--max_threads=5` with `SET /* c */ max_threads = 3`. The live tests now take the server ports from CLICKHOUSE_PORT_TCP and CLICKHOUSE_PORT_HTTP, as the runner does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@polyglotAI-bot both Majors of your review of
Unit tests for both, and live tests over native and HTTP that fail without the fixes (a TSV row with a trailing tab; |
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 01e701f: see the summary comment and the inline threads.
writeRows kept the rendered rows in a local buffer and wrote them only at the end, so an exception in a later block (for example an expected serverError) dropped the rows already received. clickhouse-client prints them, and the .reference files hold them. Now the rows rendered so far are written on every return path. A row is rendered apart first, so a row that fails to render is not written in part. A live test, over both protocols, checks `SELECT throwIf(number = 3) FROM numbers(6) SETTINGS max_block_size = 1` with its serverError hint: 0, 0, 0, then the next statement, as the real client prints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@polyglotAI-bot the Major of your review of |
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 56fdaf2: see the summary comment and the inline threads.
The live test expected exactly three rows before the exception, but the server decides how many blocks it sends first (it can squash small blocks, and over HTTP it buffers the response), so the test could fail intermittently with a correct shim. The behavior to check is the shim's: it writes every row that it received before the error. writerows_test.go checks that with a fake driver.Rows, with no server and no timing. With the old writeRows, the test gets an empty output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@polyglotAI-bot the Major of your review of |
| func TestWriteRowsBeforeStreamError(t *testing.T) { | ||
| streamErr := errors.New("exception in a later block") | ||
| r := &runner{} | ||
| err := r.writeRows(&fakeRows{vals: []uint64{0, 1, 2}, err: streamErr}, outputFormat{Name: "TabSeparated", Supported: true, Names: true}) |
There was a problem hiding this comment.
The stream-error regression now bypasses both protocols and the expected-error output path. It cannot catch failures in their interaction with writeRows.
- Per
.claude/skills/review-pr/review-core.md, tests under/tests/must not mockdriver.Rows; use real ClickHouse coverage. - Inferred from the current tests:
fakeRowssupplies fixed values directly, whileTestLiveErrorscovers only immediate exceptions.
Replace the doubles with stable live coverage of completed-row preservation and a late exception over TCP and HTTP.
A reply that holds up drops this finding at the next review.
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 54b10c0: see the summary comment and the inline threads.
- parseUseDatabase could not read `USE /* c */ db`, so the shim stopped after the server had run the statement. Comments outside quotes are now dropped before the name is read, as for SET. - The rows-before-error check runs over both protocols again. The server decides how many blocks it sends before the exception, and over native clickhouse-go can drop some of them (see below), so the live test needs some rows, all of them the values of the query, then the next statement. writerows_test.go still checks the exact output. - New known-failure group [native-rows-dropped-before-error]: rows.Next() reads the block and error channels in one select, so an error can win over blocks that were already received. With the public API, 4 of 20 runs of a query that fails after 1000 rows gave 970 or 990 rows. No allowlisted test is known to fail because of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@polyglotAI-bot both Majors of your review of
Please review again. |
| newline += pos | ||
| } | ||
| if newline <= skipWhitespaceAndComments(text, pos) { | ||
| pos = newline |
There was a problem hiding this comment.
A multiline comment after a semicolon stops an otherwise valid script on both protocols.
Inferred from code: SELECT 1; /* note continued */ SELECT 2; advances pos to the newline inside the comment. The next iteration sends continued */ SELECT 2; as SQL, causing a syntax error instead of printing the second result.
Keep block comments intact when consuming the trailing suffix, with parser and live regressions over TCP and HTTP.
A reply that holds up drops this finding at the next review.
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 30b9eb8: see the summary comment and the inline threads.
Over HTTP, the server can buffer the response and send only the exception, so zero rows before the error is a correct result there. The live test now needs at least one row over native only; over both protocols, every row must be a value of the query, followed by the next statement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@polyglotAI-bot about the three Majors of your review of
Please review again. |
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of 7402883: see the summary comment and the inline threads.
Summary
Runs a curated subset of the ClickHouse core SQL tests (
tests/queries/0_stateless) through clickhouse-go, nightly and on changes to the harness. Implements ClickHouse/integrations#810.tests/clickhouse-test-runner/is a Go program that imitatesclickhouse-clientwell enough for the upstream runner (tests/clickhouse-test) to drive it without changes. It sends every query through clickhouse-go, over the native or the HTTP protocol. It scans each row intoColumnType.ScanType(), as a user does, and writes the values asTabSeparated. So a test passes only when clickhouse-go decodes and scans every value correctly; the upstream.referencefiles are the oracle.The goal is to find client bugs, not only to pass tests. The harness already found several (see below).
What is in the PR
echohints, theTags:header,INSERT ... FORMATinline data), test hints, session handling over both protocols, and a type-directedTabSeparatedrenderer that is checked against the server's own output.allowlist.txt: 9,903 tests over both protocols, 129 over native only (WITH TOTALS: the server sends no totals in the HTTPNativeoutput), and 920 known failures in 30 groups. Each group header gives the cause and a small reproduction.KNOWN_FAILURES=<id>runs one group, to check a fix.clickhouse-version: pins the server version and the upstream tests together (26.10.4.207, tests fromv26.10.4.207-stable). A new ClickHouse release does not change CI; a bump is a pull request of its own, so other pull requests do not fail because of a release.upstream-sql-tests.yml: nightly, manual (clickhouse_versioninput to try a version), and on pull requests that change the harness. Native and HTTP, 8 shards each.README.mdandAGENTS.md: local runs, how to find the cause of a failure (compare with the realclickhouse-client:scripts/run-with-real-client.sh), the rules for the allowlist, and how to bump the version.tests/clickhouse-test-runner/CLAUDE.mdimportsAGENTS.md, and the root.claude/CLAUDE.mdpoints to it.Client bugs that the harness found
All reproduce with the public API only, over both protocols unless noted. None is fixed in this PR; each has a known-failure group. I have not filed issues yet.
Wrong values, no error:
DateTime64outside 1677-09-21 .. 2262-04-11 is wrong when read and when written (int64 nanosecond overflow in ch-goDateTime64.Timeand intimeToInt64).Time64with precision 4, 5, 7 or 8 is cut to milliseconds (ch-gotruncateDuration).Nullablevalue with a null-map byte other than 1 reads as notNULL(lib/column/nullable.gochecks== 1).NULLin aLowCardinality(Nullable(String))element of a named tuple reads as"".extremes = 1, nativeRowsgives the extremes as data rows.Session state, native:
MaxOpenConns: 1, without an error, so the session state is lost:release()frees the slot before it returns the connection to the idle list. About 1 to 5 in 2,000 iterations. This can make the nightly run fail intermittently in tests withSET, until it is fixed.Execof anINSERTthat needs data from the client (INSERT INTO t VALUES) never returns; it waits until the context ends.Queryon a statement with no result block (WITH ... INSERT) returnsio.EOFand closes the connection.rows.Next()can drop blocks that it received before an exception: it reads the block and error channels in oneselect, so the error can win (in 4 of 20 runs, 970 or 990 of 1000 rows).Errors on valid data: a tuple in a tuple,
Nullable(Tuple)andInt128in a tuple cannot be scanned intoScanType(); backtick-quoted tuple element names, someJSONdata (stream out of sync),JSON(SHARED ...), someQBitforms,Mapkeys of typeArray/Map/Tuple,Tuple(),AggregateFunction,Geometry. Over HTTP, the server-info query of a new connection runs in the user session (visible inquery_log; breaks a session withSET dialect).session_timezoneis ignored.Test plan
go test ./tests/clickhouse-test-runner/), and live tests that compare the renderer with the server'sTabSeparatedoutput (CLICKHOUSE_TEST_RUNNER_LIVE=1, run in CI shard 1).26.10.4.207: full sweeps of all 12,961.sqltests over both protocols, with each failure compared against the realclickhouse-client. Then the allowlist three times on a fresh server per protocol: 0 to 9 failures per run, each one triaged (pool race, test isolation, or memory limits that also fail through the real client).Checklist
🤖 Generated with Claude Code
Note
Low Risk
Adds test-only harness and CI; it does not change production driver APIs, though new nightly jobs can fail on known client bugs listed in the allowlist.
Overview
Adds
tests/clickhouse-test-runner, aclickhouse-clientshim that drives ClickHouse’s upstreamtests/clickhouse-testthrough clickhouse-go (native and HTTP). Queries run via the driver, results are re-encoded as TabSeparated fromScanType()values, and output is compared to upstream.referencefiles.The PR wires CI in
upstream-sql-tests.yml(nightly, manual version override, path-filtered PRs): Docker ClickHouse pinned inclickhouse-version, sparse upstream checkout, an 8-shard × 2-protocol matrix, harness unit/live tests on shard 1, andRETRY_FAILED=2for a known native pool race. Supporting pieces includeallowlist.txt(passing tests plus known-failure groups), shell scripts to fetch upstream tests and run suites, andAGENTS.md/README.mdfor triage and allowlist rules. RootCLAUDE.mdnow points agents at that doc before changing the allowlist or debugging the workflow.Reviewed by Cursor Bugbot for commit 7402883. Bugbot is set up for automated code reviews on this repo. Configure here.