Repository navigation
hnsw: vector cache lazy commit, plus cache_bytes and concurrent-insert fixes - #40
Open
tomas-villagesql wants to merge 3 commits into
Open
tomas-villagesql wants to merge 3 commits into
tomas-villagesql wants to merge 3 commits into
Conversation
|
Please add a kind label with /kind. |
|
Please add an area label with /area. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Needs approval from an approver in each of these files:Approvers can indicate their approval by writing |
tomas-villagesql
force-pushed
the
tomas/vector-cache-prefetch
branch
from
November 9, 2026 17:28
5badd22 to
d289c4d
Compare
tomas-villagesql
force-pushed
the
tomas/vector-cache-lazy-commit
branch
2 times, most recently
from
November 15, 2026 10:39
5938434 to
c6db084
Compare
tomas-villagesql
force-pushed
the
tomas/vector-cache-prefetch
branch
from
November 15, 2026 10:39
d289c4d to
c493ff3
Compare
The slab was a std::vector resized to the whole budget, which zero-filled every page: a 512 MB max_cache_size cost 512 MB on the first query against an index, however little of it the index needed, and the zeroing ran under the cache mutex, stalling every scan that started meanwhile. Map it anonymously instead. Addresses are still fixed for the cache's life, but the OS backs a page only when a fill first writes to it, so the budget becomes a ceiling rather than a reservation. A mapping the machine cannot back leaves the cache at capacity 0, running from pages as with no cache. vector_cache_test checks with mincore that the slab past the filled slots is not resident.
cache_bytes counts the resident entries of every live cache, but nothing subtracted a cache's entries when it went away. The SDK destroys an index's store, and its caches with it, after the index is dropped, so a DROP left the dropped index's bytes counted and the next index built on the same server read as using both. VectorCache's destructor now subtracts what is still resident. Retired entries are not subtracted twice: invalidate() already took them off.
insert() dropped the cache entry for the VID it had just written on its way out, as a guard against a reused VID serving the previous occupant's vector. By then link_neighbours() has made the node reachable, so a concurrent insert can reach it, miss, and admit its vector. The invalidate then retired that entry, which held the right vector, and a later read filled it again: a tombstone and an extra miss per node reached in the window. A 3-thread multi-row load of fashion-mnist 60k left 2025 / 4185 / 7251 tombstones at M = 8 / 16 / 32; one thread left none. create_node() now invalidates the VID at the first level it creates, right after the VID is known and before anything links to the node. The guard against a reused VID is unchanged; an entry admitted once the node is reachable stays resident. hnsw_vector_cache_concurrent_insert runs three connections inserting into one index at once and expects no tombstones (200 before this change).
tomas-villagesql
force-pushed
the
tomas/vector-cache-prefetch
branch
from
November 17, 2026 20:21
c493ff3 to
e327cb4
Compare
tomas-villagesql
force-pushed
the
tomas/vector-cache-lazy-commit
branch
from
November 17, 2026 20:21
c6db084 to
79c0e30
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #38 (prefetch), which is on #37 (fill-once cache) and #39 (from_binary). Review only the three commits below; retarget as the stack merges.
Commits
0d30a07reserve the vector cache slab, commit it only as it fillsThe slab was a
std::vectorresized to the whole budget, which zero-filled every page: the default 512 MBmax_cache_sizecost 512 MB on the first query against an index, and the zeroing ran under the cache mutex. It is now an anonymousmmap: addresses stay fixed for the cache's life, but the OS backs a page only when a fill first writes to it. A mapping the machine cannot back leaves the cache at capacity 0, running from pages as with no cache.6f3632dtake a destroyed vector cache off the cache_bytes gaugeNothing subtracted a cache's entries from
cache_byteswhen it was destroyed, so afterDROPthe next index on the same server read as using both caches.~VectorCache()now subtracts what is still resident.7025083invalidate a new node's VID before it is linked, not afterinsert()invalidated its own VID on the way out, as a guard against a reused VID. By thenlink_neighbours()had made the node reachable, so a concurrent insert could admit its (correct) vector and the invalidate retired it: a tombstone and an extra miss per node reached in that window. The invalidate now runs increate_node()at the first level, before anything links to the node.Tests
vector_cache_test:test_budget_is_committed_only_as_filled(mincore: pages past the filled slots are not resident) andtest_destroyed_cache_leaves_resident_bytes(failed before6f3632d).hnsw_vector_cache_concurrent_insert: three connections inserting into one index at once; tombstones 200 before7025083, 0 after over 5 repeats.hnsw_vector_cache_insertstill passes.make test6/6 pass.GCP verification (fashion-mnist 60k x 784, efc 200, cache_mode=1)
cache_bytesafter build / DROP / rebuild / DROP: 96.6 / 96.6 / 193 / 193 MB before6f3632d, 96.6 / 0 / 96.6 / 0 MB after.7025083: M=8 1674 -> 0, M=16 3667 -> 0, M=32 6883 -> 0; misses back to ~60000 per index. Recall, distance_calls and build time unchanged.🤖 Generated with Claude Code