Repository navigation
feat(turbo): support product quantization end to end - #833
richyreachy wants to merge 37 commits into
Conversation
Merge refactor/ivf_on_turbo_pq into feat/turbo-pq, preserving the HNSW codebook lifecycle and the IVF posting-format and provider-read fixes. Include the existing IVF PQ Python configuration and persistence support.
| int train(IndexHolder::Pointer holder) override; | ||
|
|
||
| int prepare_build() override { | ||
| if (dist_table_.empty()) compute_dist_table(); |
There was a problem hiding this comment.
是的，只读查询走 ADC，使用查询 LUT，不需要 dist_table_。这张表用于 SDC，即建图或追加数据时计算两个 PQ code 的距离。反序列化后如果恢复写入，需要通过 prepare_build() 重建。已补充注释和验证。
There was a problem hiding this comment.
那是不是readonly Hnsw打开的时候不需要调用这个？这个的代价有多大呀
There was a problem hiding this comment.
只读 HNSW 打开时不需要调用 prepare_build()，只在可写打开时初始化 SDC 距离表。
There was a problem hiding this comment.
但现在readonly也初始化了，如果代价大的话是不是应该不调用 prepare_build()？不大就算了
| // Storage's boolean also controls write access. Reopening an existing | ||
| // writable index must not turn BufferStorage into a read-only pool. | ||
| int ret = storage_->open(file_path, storage_options.create_new || | ||
| (!storage_options.read_only && | ||
| ailego::File::IsExist(file_path))); |
There was a problem hiding this comment.
这里似乎有个潜在问题，bufferpool模式下打开可写索引似乎可能会无法走到 bypass-only路径， @iaojnh 师兄看看要不要改掉create_if_missing和writeable的语义复用 （不太清楚这边的逻辑背景和设计）
There was a problem hiding this comment.
这里复用了 create_if_missing 和 writable 的语义。此次修改是修复已有索引以可写方式 reopen 时被错误打开为只读的问题。当前 bypass-only 仅支持只读，可写池依赖页表和脏页管理，不能直接采用该回退路径。建议将创建权限与写权限独立传递，保留现有的只读 bypass-only 行为。@iaojnh
Summary
Integrate Turbo product quantization into the index lifecycle and public APIs, with initial support for HNSW and IVF.
Users can configure PQ4/PQ8 and optional OPQ through
QuantizeType.PQandQuantizerParam. Training runs duringCollection.optimize(), while queries before optimization use the raw Flat index.Changes