Bounded first page pads with shorter-prefix characters instead of exact readings for common syllables #17

Status (2026-10-19)

The user-visible defect is fixed and live; open for oracle certification. #18 landed the bounded-page fix, and it has been in production since 2026-10-16 (dabf60c0). On main and on the live demo, all nine syllables below now show the same page 0 on the bounded and complete paths, with no shorter-prefix padding (for example fan → 分 份 婚 訓 粉 瞓, siu → 小 少 消 笑 燒 銷). The table below is the historical 2026-10-10 reproduction. Neither the pre-fix build nor the current build shows the duplicate 分 it lists, so compare captures against current output rather than against the table.

Remaining: Done-when item 1 (TypeDuck-HK librime v1.1.2 page-size-6 captures for the nine inputs) and the ratchet part of item 4. hou already matches the first six rows of the checked-in v1.1.2 fixtures (captured at page size 50), so eight captures remain.

Blocked by: None. #20 landed in 039708e2 on 2026-10-20. Assert the captures on the shipped compiled assets (table.bin plus prism), not only through cantonese_parity, which builds its engine from the source dictionary. #20 added the compiled-asset harness for this: typeduck_v112_unpatched_case and leading_candidate_texts in crates/yune-rime-api/tests/yune_web.rs, and the compiled engine in cantonese_parity. The oracle rows #31 needs can be captured in the same Windows session. The former "blocks #8" link was removed because item 5 is met.

Context

For the TypeDuck Jyutping profile the engine computes only the first page of candidates on each keystroke (the bounded refresh), and materializes the complete list only when the user pages forward. The shipped schema also sets combine_candidates: true, which merges rows that share the same text, and prefix_fallback: true, which supplies characters for a shorter prefix when the page would otherwise be short.

Problem

On a fresh profile with no learned rows, the bounded first page differs from the complete list's first page for about one in five common syllables. The bounded path truncates the exact-match rows to the page size first, then collapses duplicate texts, and fills the gap with prefix-fallback rows. The complete path keeps the next exact readings. Measured on 2026-10-10 against the shipped jyut6ping3_mobile assets through the CLI (fan typed alone versus fan{Page_Down}{Page_Up}), 9 of 41 syllables differ:

The replaced characters are readings of a shorter prefix (faa, ho, maa, and so on), not of the typed syllable. Which page is correct must be pinned by the oracle (the reference TypeDuck-HK librime v1.1.2 build with the same schema), but a prefix-fallback row displacing an exact reading of the full syllable is not what either the complete path or librime produces, so the bounded page is the defect.

Mechanism, in crates/yune-core/src/translator/mod.rs: the bounded exact path calls order_lookup_candidate_refs(&mut selected, limit) (line 5471), which truncates to limit; combine_duplicate_text_candidates_with_full_input_anchor (called at line 5552) then collapses same-text rows with no backfill; the prefix-fallback insertion that follows fills the shortfall. The complete path never truncates before collapsing. The same under-fill is what stops #8's learned-phrase shortcut: with a learned row present, the bounded page for fan lands the learned row's anchor on a fallback row.

Approach

• In the bounded exact path, collapse duplicates before truncating, or fetch limit + number of collapsed rows and truncate after the collapse, so the window always holds limit distinct exact rows when the complete list has them.
• Do not touch the complete path, the prefix-fallback ordering, or merge_userdb_candidates.
• Pin the behavior two ways: oracle fixtures for the nine inputs above captured from TypeDuck-HK librime v1.1.2 with the same schema (non-circular, per docs/conventions.md section 7), and a self-consistency sweep test on the production path (compact table plus prism) asserting bounded page 0 equals forced-complete page 0 for a list of at least the 41 syllables used here.

Done when

• Oracle fixtures for the nine inputs exist and the bounded page matches them.
• The sweep self-consistency test passes on the production path and fails on current main.
• Existing Jyutping, Luna, and Cangjie parity fixtures pass unchanged.
• Per-key latency for the swept syllables stays within the existing browser latency gate; the signed native ratchet is re-run by a maintainer before close.
• Retyping a learned multi-syllable phrase stalls the engine for seconds #8's user-confirmed three-case comparison (cungfukhokzaap → 重複學習 via real commit/reload, explicitly seeded coengjamzitsyujapcaksi → 長音節輸入測試, and duplicate-heavy learned fan) shows identical pages on both paths, which unblocks Retyping a learned multi-syllable phrase stalls the engine for seconds #8.

How verified

cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo test -p yune-core --test cantonese_parity
cargo test -p yune-core --test upstream_luna_pinyin_parity
cargo test -p yune-rime-api --test yune_web
npm --prefix apps/yune-web/e2e run test:e2e:input-latency:public

Reproduce by hand: build cargo build --release -p yune-cli, then run the CLI twice against a fresh user directory with --schema jyut6ping3_mobile --shared-data-dir apps/yune-web/public/schema, once with --sequence "fan" and once with --sequence "fan{Page_Down}{Page_Up}", and compare the last event's candidates in the JSON output.

Background

Found while reviewing the attempt at #8 (comment of 2026-10-10), which hit this divergence on fan and stopped as instructed. The bounded refresh comes from the lazy-candidate work (M34) and the page-bounded TypeDuck refresh from the later performance milestones; see docs/history.md. No existing oracle fixture covers a syllable whose exact rows contain duplicate texts, which is why this never tripped a gate.