Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
77a0c5f
test: run ClickHouse core SQL tests through clickhouse-go
ccoltx Nov 18, 2026
5f6e1d4
test: fix two harness issues from the Bugbot review
ccoltx Nov 18, 2026
c8702fb
test: keep native read failures as errors, and skip # comments
ccoltx Nov 18, 2026
b71abce
test: detect INSERT inline data by the word after FORMAT
ccoltx Nov 18, 2026
b6994ff
test: do not read sort keys and select-list items as FORMAT clauses
ccoltx Nov 18, 2026
1bf51b9
ci: retry failed upstream SQL tests once, until the pool race is fixed
ccoltx Nov 18, 2026
5dab946
test: track tests that hit the pool race often as known failures
ccoltx Nov 18, 2026
f0c36b2
ci: retry failed upstream SQL tests up to twice
ccoltx Nov 18, 2026
129ec0b
test: do not read an implicit alias of a `format` column as FORMAT
ccoltx Nov 18, 2026
9381219
test: find the token before FORMAT outside comments, and handle selec…
ccoltx Nov 18, 2026
3bac89c
Merge branch 'main' into ccoltx/Run-core-SQL-tests
ccoltx Nov 18, 2026
2e1459a
Merge branch 'main' into ccoltx/Run-core-SQL-tests
ccoltx Nov 18, 2026
2549804
test: read FORMAT <word> as the clause only for a real format name
ccoltx Nov 18, 2026
f2f5c87
test: run a leading function call as a query under implicit_select
ccoltx Nov 18, 2026
9ecc4eb
test: run a leading VALUES or TABLE statement as a query
ccoltx Nov 18, 2026
245b739
test: a table alias that is a format name is not a FORMAT clause
ccoltx Nov 18, 2026
774dccc
test: a keyword alias before FORMAT does not hide the clause
ccoltx Nov 18, 2026
d489e2e
test: upload hidden test outputs, turn off the runner's own retries
polyglotAI-bot Nov 19, 2026
3b8f18e
test: drop include-hidden-files from the artifact upload
polyglotAI-bot Nov 19, 2026
989708e
test: write the Raw header without escapes, skip heredocs in word sca…
polyglotAI-bot Nov 19, 2026
ad93c4c
test: hide the --password value in the invocation log
polyglotAI-bot Nov 19, 2026
3fca4c4
test: a retry replaces only an attempt that completed
ccoltx Nov 20, 2026
a865fd8
test: track two more tests that hit known client bugs
ccoltx Nov 20, 2026
01e701f
test: keep trailing blanks of inline data, ignore comments in SET names
ccoltx Nov 21, 2026
56fdaf2
test: write the rows received before a stream error
ccoltx Nov 21, 2026
54b10c0
test: check rows before a stream error with a unit test, not a live test
ccoltx Nov 21, 2026
30b9eb8
test: read USE names without comments; check stream errors live again
ccoltx Nov 21, 2026
7402883
test: accept zero rows before a stream error over HTTP
ccoltx Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,4 +105,5 @@ A vulnerability that is not public yet goes into a private security report (**Se
- Register `t.Cleanup(func() { conn.Close() })` for every connection a test opens — both a native `driver.Conn` and a `database/sql` `*sql.DB` — so subtests don't leak connections across the native/HTTP runs.
- Integration tests also run on ClickHouse Cloud, where the service has more than one replica and each new connection can go to a different replica. Do not use the `Memory` engine for a table that a test writes and then reads: its data stays on the replica that got the `INSERT`, so another replica reads an empty table and the test is flaky. Use `MergeTree ORDER BY tuple()` (or a real sorting key). A `TEMPORARY TABLE` in a session is the only exception. For the same reason, read `system.*` log tables with `clusterAllReplicas('default', ...)` and flush them with `SYSTEM FLUSH LOGS ON CLUSTER 'default'` when `clickhouse_tests.CloudClickHouse` is true.
- New ClickHouse type support requires: a column implementation in `lib/column/`, a round-trip test in `tests/`, and an example in `examples/clickhouse_api/`.
- Before you change the upstream SQL test allowlist, or examine a failure of the `Upstream SQL tests` workflow, read `tests/clickhouse-test-runner/AGENTS.md`.
- When referencing ClickHouse SQL types, functions, or log messages in text, wrap them in backticks: `` `DateTime64` ``, `` `MergeTree` ``.
124 changes: 124 additions & 0 deletions .github/workflows/upstream-sql-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Upstream SQL tests

# Runs a curated subset of the ClickHouse core SQL tests
# (tests/queries/0_stateless) through clickhouse-go. See
# tests/clickhouse-test-runner/README.md.

permissions:
contents: read

on:
workflow_dispatch:
inputs:
clickhouse_version:
description: "ClickHouse version to try, for example 26.11.1.10 (default: tests/clickhouse-test-runner/clickhouse-version)"
required: false
default: ""
type: string
schedule:
- cron: "0 4 * * *"
push:
branches:
- main
paths:
- "tests/clickhouse-test-runner/**"
- ".github/workflows/upstream-sql-tests.yml"
pull_request:
paths:
- "tests/clickhouse-test-runner/**"
- ".github/workflows/upstream-sql-tests.yml"

concurrency:
group: "${{ github.workflow }}-${{ github.ref }}"
cancel-in-progress: true

jobs:
upstream-sql-tests:
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
protocol: [native, http]
# Round-robin shards of the allowlist, about 1,250 tests each. A full
# run of one protocol takes about 14 minutes at -j 8 on 18 cores, so
# a shard takes a few minutes on a 4-vCPU runner. Change SHARD_TOTAL
# below together with this list.
shard: [1, 2, 3, 4, 5, 6, 7, 8]
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "1.27"

- name: Install Python
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install Python dependencies of tests/clickhouse-test
run: pip install jinja2

# The server version and the upstream tests are pinned together in
# tests/clickhouse-test-runner/clickhouse-version: the `.reference`
# files of another release do not match the output of the server. A
# new server release changes nothing here until a pull request bumps
# the pin. The `clickhouse_version` input of a manual run tries
# another version without a commit.
- name: Select the ClickHouse version
id: version
env:
INPUT_VERSION: ${{ github.event.inputs.clickhouse_version }}
run: |
version="${INPUT_VERSION:-$(tests/clickhouse-test-runner/scripts/pinned-version.sh)}"
if ! [[ "${version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "::error::clickhouse_version must be a full version such as 26.10.4.207 (got: '${version}')"
exit 1
fi
echo "version=${version}" >> "${GITHUB_OUTPUT}"

- name: Start ClickHouse ${{ steps.version.outputs.version }}
env:
CLICKHOUSE_VERSION: ${{ steps.version.outputs.version }}
run: docker compose up --wait

# Also generates errorcodes.go from the same checkout.
- name: Get the upstream tests of ClickHouse ${{ steps.version.outputs.version }}
id: upstream
run: tests/clickhouse-test-runner/scripts/fetch-upstream.sh "${{ steps.version.outputs.version }}"

- name: Unit and live tests of the harness
if: matrix.shard == 1
env:
CLICKHOUSE_TEST_RUNNER_LIVE: "1"
run: go test -count=1 ./tests/clickhouse-test-runner/

- name: Run upstream SQL tests
env:
CLICKHOUSE_TEST_RUNNER_PROTOCOL: ${{ matrix.protocol }}
CLICKHOUSE_CLIENT_CLI_LOG: ${{ github.workspace }}/upstream-run.log
SHARD_INDEX: ${{ matrix.shard }}
SHARD_TOTAL: 8
# Run the failed tests again, at most twice: a known client bug (the
# pool race, [native-pool-race-new-connection] in allowlist.txt)
# makes about 40 random tests with SET fail per native run, and one
# of them is now and then hit twice. A test that passes on a retry is
# a warning; a test that fails all three times fails the job.
RETRY_FAILED: "2"
run: tests/clickhouse-test-runner/scripts/run-upstream-tests.sh -j 4

- name: Upload test artifacts
if: failure()
uses: actions/upload-artifact@v4
with:
name: upstream-sql-tests-${{ steps.upstream.outputs.ref }}-${{ matrix.protocol }}-shard-${{ matrix.shard }}
retention-days: 14
if-no-files-found: ignore
path: |
upstream-run.log
tests/clickhouse-test-runner/.upstream/ClickHouse/tests/queries/**/*.stdout
tests/clickhouse-test-runner/.upstream/ClickHouse/tests/queries/**/*.stderr
tests/clickhouse-test-runner/.upstream/ClickHouse/tests/queries/**/*.diff
3 changes: 3 additions & 0 deletions tests/clickhouse-test-runner/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.bin/
.bin-real-client/
.upstream/
174 changes: 174 additions & 0 deletions tests/clickhouse-test-runner/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
# Instructions for agents: upstream SQL test harness

This harness runs the ClickHouse core SQL tests through clickhouse-go. See
[`README.md`](README.md) for how it works and how to run it.

## The goal

The goal is to find bugs in clickhouse-go. To pass more tests is not the
goal. A failed test is information. Do not hide it.

## Rules

1. **Find the cause of a failure before you do anything with the test.** A
test fails for one of three causes:
- **The client** (clickhouse-go, or `ch-go`, which it uses to decode the
data). This is what the harness is for.
- **The shim** (the program in this directory). It does not do what
`clickhouse-client` does.
- **The environment.** The test cannot pass through any client on this
server (see [Environment causes](#environment-causes)).
2. **The default for a client bug is a fix in the client.** Open a pull
request that fixes clickhouse-go, with a regression test in
`tests/issues/` (see the root `CLAUDE.md`). When the bug is in `ch-go`,
report it in `ClickHouse/ch-go` and link the report from the
clickhouse-go issue.
3. **The default for a shim bug is a fix in the shim.** Make the shim copy
`clickhouse-client` more exactly. Do not change the shim to hide a client
bug, for example by a different scan type or a workaround for a wrong
value. The shim must use the client as a user does: scan each value into
`ColumnType.ScanType()`.
4. **Remove a test from the allowlist only when it cannot pass for an
environment cause** (see rules 6 and 7). Write the reason in the commit
message.
5. **A test that fails because of a client bug or a shim gap stays
visible.** Do not delete it. Put it in [`allowlist.txt`](allowlist.txt)
as a known failure, `# <test> # [<id>]`, in the group of its cause. Each
group has a header that describes the bug, gives a small reproduction and
links the issue. Make a new group when no group fits. See the README for
the format and for how to run one group (`KNOWN_FAILURES=<id>`).
6. **A test that cannot pass over one protocol** for an environment cause
(for example `WITH TOTALS` over HTTP) gets `# only=<protocol>: <reason>`.
Do not use this for a client bug: use a known failure.
7. **Do not list a test that cannot pass for an environment cause** over
both protocols.

## How to find the cause

Do these steps in this order.

1. **Run the test over both protocols:**

```bash
UPSTREAM_TEST_LIST=<(echo <test>) tests/clickhouse-test-runner/scripts/run-upstream-tests.sh
UPSTREAM_TEST_LIST=<(echo <test>) CLICKHOUSE_TEST_RUNNER_PROTOCOL=http tests/clickhouse-test-runner/scripts/run-upstream-tests.sh
```

A test that passes over one protocol and fails over the other points to
the client, unless the cause is in the list below.
2. **Run the test through the real `clickhouse-client`** in the server
container, on the same server:

```bash
UPSTREAM_TEST_LIST=<(echo <test>) tests/clickhouse-test-runner/scripts/run-with-real-client.sh
```

If it also fails here, the cause is the environment. If it passes here,
the cause is the client or the shim.
3. **Find the statement that fails.** Set `CLICKHOUSE_CLIENT_CLI_LOG` to a
file; the log shows each statement and the error. Read the `.diff`,
`.stdout` and `.stderr` files next to the test.
4. **Compare the shim with the server on that one statement.** Run it
through the shim (`go run ./tests/clickhouse-test-runner -q "<SQL>"`)
and through `curl` with `FORMAT TSV`. Then make the statement smaller
until it is as small as possible.
5. **Decide between the client and the shim with a small Go program** that
uses only the public clickhouse-go API (`Query`, `Scan`, `ScanType()`). If
the program also gives a wrong value or an error, the bug is in the
client. If the program gives the correct value, the bug is in the shim.
Try both protocols.

### Environment causes

A test cannot pass through any client on the CI server when it needs one of
these:

- A cluster, shards or replicas (`CLUSTER_DOESNT_EXIST`, `test_shard_localhost`).
- ZooKeeper or Keeper (`NO_ZOOKEEPER`, `Replicated*` engines).
- External services: S3, Azure, HDFS, Kafka, MySQL, PostgreSQL, MongoDB, URL
sources.
- Server configuration of the upstream CI: named collections, dictionaries,
storage policies, disks, users and grants, macros
(`NAMED_COLLECTION_DOESNT_EXIST`, `NO_ELEMENTS_IN_CONFIG`, `UNKNOWN_POLICY`,
`UNKNOWN_DISK`).
- Features that the server build or platform does not have
(`SUPPORT_IS_DISABLED`, "not supported on Darwin").
- Stateful datasets (`--no-stateful`) or long tests (`--no-long`).
- Behavior of the HTTP interface itself. For example, the server sends no
`WITH TOTALS` and no `extremes` in the `Native` format over HTTP, and an
HTTP request fails when the current database of the session was dropped.
- Features of `clickhouse-client` itself that are not part of a driver:
`INTO OUTFILE`, `FROM INFILE`, client-side parsing of `INSERT` data, and
client-only settings.
- The tz database. ClickHouse and Go can give different historical offsets
for the same zone (for example `Europe/Amsterdam` before 1937).
- Tests that conflict with other tests of the set: a fixed table UUID, or
server state that a test leaves behind (failpoints that a failed test did
not disable, row policies). Such a test passes alone on a fresh server and
fails in a full run. A failure message with "Injected failure" comes from a
failpoint of another test.
- Memory limits of the server, when the test also fails through the real
`clickhouse-client`.

These are **not** environment causes. Find the bug:

- A value that is different from the server's value.
- A decode error, or an error such as `unexpected packet`.
- A `ScanType()` that `Scan` does not accept.
- A failure over one protocol only, when the cause is not in the list above.

## Conditions for the allowlist

- Add a test only when it passes over **both** protocols, or over one
protocol with an `only=` reason (rule 6).
- Run new tests three times, also with a higher `-j` than CI, on a **fresh**
server (`make down && CLICKHOUSE_VERSION=latest make up`). Tests change the
state of the server (row policies, settings profiles, users). A test that
passes only after other tests ran must not go in.
- Do not run two sweeps at the same time on one server. The memory pressure
causes `MEMORY_LIMIT_EXCEEDED` failures that are not real.
- CI runs the failed tests again, at most twice (`RETRY_FAILED=2`), because
of the pool race. Read the warnings of the passed-on-retry tests in the
nightly runs: a test that is there often has a cause of its own, or hits
the race often enough to belong in its group. Do not use the retry when you
grow or check the allowlist.
- When a test in the allowlist starts to fail intermittently, find the cause
first (rule 1). Intermittent native failures of tests that use `SET` can
come from the pool race of `[native-pool-race-new-connection]`: the session
loses the `SET`. Remove it only when the cause is in the environment or in
the test itself, and write the reason.

## Bump the server version

The server version and the upstream tests are pinned together in
[`clickhouse-version`](clickhouse-version). A new ClickHouse release changes
nothing in CI until a pull request bumps the pin, so other pull requests do
not fail because of a release. Bump the pin in a pull request of its own,
and make that pull request pass before it merges:

1. Select the version: a full version with a server image tag and a
`v<version>-stable` tag upstream, normally the version of the `latest`
image.
2. Try it: start a manual run with the `clickhouse_version` input, or run
the allowlist locally on that version (`fetch-upstream.sh <version>`, and
a fresh server of that version).
3. For each allowlisted test that now fails, find the cause (rule 1). A
release changes the tests and the server output, so many tests can fail
at the same time with no change in the client. To see whether the client
is the cause, run the same test through the real client
(`run-with-real-client.sh`) on the same server.
4. Run the known failures (`KNOWN_FAILURES=all`). Enable the tests that pass
now.
5. Fix the names of allowlisted tests that the run reports as not in the
checkout (upstream renamed or removed them).
6. Optionally, try the new tests of the release (see "Grow the allowlist" in
the README).
7. Change `clickhouse-version`, run `fetch-upstream.sh`, and run
`gen-formats.sh` against a server of the new version. Commit
`errorcodes.go` and `formats.go` with the new version.

## Keep the documents in sync

When you change the behavior of the shim (flags, statement handling, output
formats, settings), update [`README.md`](README.md), this file and the
workflow in the same change.
1 change: 1 addition & 0 deletions tests/clickhouse-test-runner/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
@AGENTS.md
Loading
Loading