Skip to content

fix: don't count time blocked on a slow consumer against ReadTimeout - #2063

Merged
chernser merged 3 commits into
ClickHouse:mainfrom
ninadsutrave:fix/read-timeout-per-packet
Nov 20, 2026
Merged

chernser merged 3 commits into
ClickHouse:mainfrom
ninadsutrave:fix/read-timeout-per-packet

Conversation

@ninadsutrave

@ninadsutrave ninadsutrave commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1672.

On the native protocol the read deadline for a query's result stream is set once, before the read loop. When a rows.Next() consumer is slow, the stream channel fills and the reader blocks inside on.data while that deadline keeps running, so the next socket read fails with i/o timeout even though the server responded promptly.

The read deadline is now reset after a data block has been handed to the consumer, so time spent waiting on the consumer no longer counts. A query that keeps the server busy past ReadTimeout before sending data, such as SELECT sleep(3) with a 1s timeout, still fails as before. Once blocks are flowing, each one restarts the timeout, so a long result that keeps streaming is no longer cut off by its total duration; a context deadline still applies and takes precedence.

tests/issues/issue_1672_test.go covers both cases: a slow consumer reading a multi-block result (fails with i/o timeout before, passes after) and a slow server query (times out both before and after).

Checklist

  • Unit and integration tests covering the common scenarios were added

Note

Medium Risk
Touches native protocol read-loop timeout behavior for all streaming queries; wrong reset logic could mask slow servers or cause hung reads.

Overview
Resets the connection read deadline after each data block is delivered to the query consumer (including empty blocks), so idle time waiting on a slow rows.Next() loop no longer burns the configured ReadTimeout while the server has already sent data.

Adds issue #1672 regression tests: a slow consumer over a long multi-block SELECT must complete when total wall time exceeds ReadTimeout, while a SELECT sleep(3) with a 1s ReadTimeout still fails with os.ErrDeadlineExceeded.

Reviewed by Cursor Bugbot for commit ebe1d3a. Bugbot is set up for automated code reviews on this repo. Configure here.

@CLAassistant

CLAassistant commented Oct 13, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ninadsutrave

Copy link
Copy Markdown
Contributor Author

CI hasn't run here yet, I think it needs approval since this is my first PR. Could someone kick it off when there's time?

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Claude review

Resets the native read deadline after each data block is given to the consumer, so the time that the reader waits on a slow rows.Next() consumer does not count against ReadTimeout. The change is correct and small. I found no problems that block the merge.

What I checked:

  • handle is the only caller of on.data. Only conn_query.go sets on.data, so exec, async insert, and batch paths do not change.
  • The reset runs in the same goroutine as the deferred clearReadWriteTimeout in processImpl, so there is no new race on the deadline.
  • A context deadline continues to take precedence, because startReadWriteTimeout applies SetDeadline after SetReadDeadline.
  • The new behavior agrees with docs/config-reference.mdx, which already says that ReadTimeout is "applied per block, not entire query".
  • The HTTP path does not use this code. The database/sql surface uses the same connect.query path, so the fix applies there also.
  • The regression test is in tests/issues/issue_1672_test.go, registers t.Cleanup for the connection, and uses no table.

Remaining limit (not a blocker): only data blocks with rows reset the deadline. Progress, log, and profile-event packets do not. A query for which the server sends only such packets for longer than ReadTimeout between two data blocks continues to fail, as it did before this change.

Blind spots: I did not run the tests. I did not confirm with a live server that the slow consumer subtest fails without the fix.

Verdict: ✅ Approve

Inline comments are attached to the relevant lines. This summary updates in place on re-review.

@chernser chernser left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ninadsutrave
please add reset on empty blocks:

Long scan with selective WHERE: If ClickHouse scans data for longer than ReadTimeout while emitting only 0-row ServerData blocks, PR does not reset ReadTimeout because c.startReadWriteTimeout(ctx) sits inside if block.Rows() != 0 && on.data != nil. Moving the reset outside that if (at the end of the ServerData, ServerTotals, ServerExtremes case, when on.data != nil) would cover empty data blocks as well without touching progress or log packets.

@ninadsutrave

Copy link
Copy Markdown
Contributor Author

Done, the reset now runs for every data block, empty or not.

While testing it, though, I couldn't get the server to send empty blocks for a selective scan. With SELECT number FROM numbers(60) WHERE sleepEachRow(0.05) = 1 and max_block_size = 10 (3s total, ReadTimeout 1s), the latest server only sends Progress and ProfileEvents packets every ~100ms and no Data packet until the end, so it still times out after 1s.

If you want that case covered, the reset would need to run on Progress packets too. That changes ReadTimeout from "time between blocks" to "time the server is silent", so I've left it out for now. Happy to add it here or in a follow-up if you'd prefer.

@ninadsutrave
ninadsutrave requested review from a team and polyglotAI-bot as code owners November 20, 2026 06:11
@chernser

Copy link
Copy Markdown

@ninadsutrave

This is fine. Thank you for the update.

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Co-signing @chernser's approval of ebe1d3a: the AI review gate is green on this commit.

@chernser
chernser merged commit 8d53e99 into ClickHouse:main Nov 20, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReadTimeout is not applied correctly

4 participants