Skip to content

hnsw: fill-once int16 vector cache - #37

Open
tomas-villagesql wants to merge 2 commits into
mainfrom
tomas/vector-cache-fill-once
Open

tomas-villagesql wants to merge 2 commits into
mainfrom
tomas/vector-cache-fill-once

Conversation

@tomas-villagesql

@tomas-villagesql tomas-villagesql commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

A per-index cache of vectors in the form the distance kernel consumes: decoded and quantized to int16 (MariaDB MHNSW's recipe: per-vector scale, int16 components, precomputed abs2), read in place by the kernel with no copy. Squared L2 only; cosine and inner-product kernels follow in #36, stacked on this.

Design

  • Fill once. A slot is written once and never rewritten; when the cache is full it stops admitting and a miss is compared in its stored f32 form. No eviction: on GCP at 60k x 784, an evicting cache held below the working set ran at 0.66x of no cache, and above it never evicted. Entries' addresses and contents are stable for the cache's life, so readers are lock-free with no pin or epoch.
  • Mixed forms per comparison. A cached vector compares in int16, a refused one in f32. Quantizing every refused read instead measured 3.6x slower on build and 0.41x on queries once the cache was full.
  • Open-addressed index (splitmix64, linear probing, <= 50% load). The slab is reserved up front for the whole budget, so slot addresses never move; a budget change rebuilds the cache.
  • Insert path. create_node() seeds the per-insert DecodedVectorCache (hnsw: decode each node's vector once per graph call #31) with the new node, so the insert never fetches its own node back by VID, never admits it to the cache, and insert()'s VID-reuse invalidate has no just-admitted entry to retire (otherwise one dead slot per insert, which a fill-once cache never reclaims).
  • Controls: global sys vars vsql_vector.cache_mode (0 = off, 1 = vectors) and vsql_vector.max_cache_size. Off by default.
  • Status: vsql_vector.cache_hits, cache_misses, cache_admissions_refused, cache_tombstones, cache_bytes, cache_limit_changes, cache_limit_slots.

Testing

Unit tests vector_cache_test and decoded_vector_cache_test; make test 6/6 pass.

MTR hnsw_vector_cache_insert: 300 inserts with the cache on and M = 2, so nearly every insert overflows a neighbour and takes the shrink path. The cache is used and cache_tombstones stays 0; without the create_node() seed it records 295.

Known: bin_to_svector fails on current main independently of this PR (the server rejects binary values for SVECTOR).

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Please add a kind label with /kind.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Please add an area label with /area.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-fill-once branch from 360ed50 to 1bae0cc Compare November 7, 2026 15:36
@tomas-villagesql
tomas-villagesql changed the base branch from main to tomas/svector-from-binary-dispatch November 7, 2026 15:36
Base automatically changed from tomas/svector-from-binary-dispatch to main November 14, 2026 07:41
@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-fill-once branch from 1bae0cc to 76fff30 Compare November 15, 2026 10:39
An insert compares its new node against the neighbours whose lists it
overflows. If that fetched the new node back by VID, the cache would
admit it and insert()'s VID-reuse invalidate would retire it: one dead
slot per insert, never reclaimed. create_node() seeds the per-insert
vectors with the new node instead.

300 inserts with M = 2 (so nearly every insert takes the shrink path):
the cache is used and cache_tombstones stays 0. Without the seed it
records 295.
@tomas-villagesql
tomas-villagesql force-pushed the tomas/vector-cache-fill-once branch from 76fff30 to 4c5acff Compare November 17, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant