Skip to content

hnsw: vector cache lazy commit, plus cache_bytes and concurrent-insert fixes - #40

Open
tomas-villagesql wants to merge 3 commits into
tomas/vector-cache-prefetchfrom
tomas/vector-cache-lazy-commit
Open

tomas-villagesql wants to merge 3 commits into
tomas/vector-cache-prefetchfrom
tomas/vector-cache-lazy-commit

Conversation

@tomas-villagesql

Copy link
Copy Markdown
Member

Stacked on #38 (prefetch), which is on #37 (fill-once cache) and #39 (from_binary). Review only the three commits below; retarget as the stack merges.

Commits

0d30a07 reserve the vector cache slab, commit it only as it fills
The slab was a std::vector resized to the whole budget, which zero-filled every page: the default 512 MB max_cache_size cost 512 MB on the first query against an index, and the zeroing ran under the cache mutex. It is now an anonymous mmap: addresses stay fixed for the cache's life, but the OS backs a page only when a fill first writes to it. A mapping the machine cannot back leaves the cache at capacity 0, running from pages as with no cache.

6f3632d take a destroyed vector cache off the cache_bytes gauge
Nothing subtracted a cache's entries from cache_bytes when it was destroyed, so after DROP the next index on the same server read as using both caches. ~VectorCache() now subtracts what is still resident.

7025083 invalidate a new node's VID before it is linked, not after
insert() invalidated its own VID on the way out, as a guard against a reused VID. By then link_neighbours() had made the node reachable, so a concurrent insert could admit its (correct) vector and the invalidate retired it: a tombstone and an extra miss per node reached in that window. The invalidate now runs in create_node() at the first level, before anything links to the node.

Tests

  • vector_cache_test: test_budget_is_committed_only_as_filled (mincore: pages past the filled slots are not resident) and test_destroyed_cache_leaves_resident_bytes (failed before 6f3632d).
  • New MTR hnsw_vector_cache_concurrent_insert: three connections inserting into one index at once; tombstones 200 before 7025083, 0 after over 5 repeats. hnsw_vector_cache_insert still passes.
  • make test 6/6 pass.

GCP verification (fashion-mnist 60k x 784, efc 200, cache_mode=1)

  • cache_bytes after build / DROP / rebuild / DROP: 96.6 / 96.6 / 193 / 193 MB before 6f3632d, 96.6 / 0 / 96.6 / 0 MB after.
  • Three concurrent multi-row inserters, tombstones before -> after 7025083: M=8 1674 -> 0, M=16 3667 -> 0, M=32 6883 -> 0; misses back to ~60000 per index. Recall, distance_calls and build time unchanged.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

Please add a kind label with /kind.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

Please add an area label with /area.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-prefetch branch from 5badd22 to d289c4d Compare November 9, 2026 17:28
@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-lazy-commit branch 2 times, most recently from 5938434 to c6db084 Compare November 15, 2026 10:39
@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-prefetch branch from d289c4d to c493ff3 Compare November 15, 2026 10:39
The slab was a std::vector resized to the whole budget, which zero-filled
every page: a 512 MB max_cache_size cost 512 MB on the first query against
an index, however little of it the index needed, and the zeroing ran under
the cache mutex, stalling every scan that started meanwhile.

Map it anonymously instead. Addresses are still fixed for the cache's life,
but the OS backs a page only when a fill first writes to it, so the budget
becomes a ceiling rather than a reservation. A mapping the machine cannot
back leaves the cache at capacity 0, running from pages as with no cache.

vector_cache_test checks with mincore that the slab past the filled slots
is not resident.
cache_bytes counts the resident entries of every live cache, but nothing
subtracted a cache's entries when it went away. The SDK destroys an index's
store, and its caches with it, after the index is dropped, so a DROP left
the dropped index's bytes counted and the next index built on the same
server read as using both.

VectorCache's destructor now subtracts what is still resident. Retired
entries are not subtracted twice: invalidate() already took them off.
insert() dropped the cache entry for the VID it had just written on its way
out, as a guard against a reused VID serving the previous occupant's vector.
By then link_neighbours() has made the node reachable, so a concurrent
insert can reach it, miss, and admit its vector. The invalidate then retired
that entry, which held the right vector, and a later read filled it again:
a tombstone and an extra miss per node reached in the window. A 3-thread
multi-row load of fashion-mnist 60k left 2025 / 4185 / 7251 tombstones at
M = 8 / 16 / 32; one thread left none.

create_node() now invalidates the VID at the first level it creates, right
after the VID is known and before anything links to the node. The guard
against a reused VID is unchanged; an entry admitted once the node is
reachable stays resident.

hnsw_vector_cache_concurrent_insert runs three connections inserting into
one index at once and expects no tombstones (200 before this change).
@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-prefetch branch from c493ff3 to e327cb4 Compare November 17, 2026 20:21
@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-lazy-commit branch from c6db084 to 79c0e30 Compare November 17, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant