Skip to content

fix: restore the startup-timeout fix and refresh the dataset counts - #5

Merged
NanmiCoder merged 4 commits into
mainfrom
fix/startup-timeout-and-dataset-counts
Jul 26, 2026
Merged

NanmiCoder merged 4 commits into
mainfrom
fix/startup-timeout-and-dataset-counts

Conversation

@NanmiCoder

Copy link
Copy Markdown
Owner

Summary

Two follow-ups to #4.

The startup-timeout fix never landed. #4 was merged while its head ref was
still b0913c4, so the sixth commit — the fix for the macOS CI failure that PR's
own new job had just exposed — was left behind. Cherry-picked here unchanged.

The first start expands an 80 MB archive into a ~270 MB SQLite database and warms
the gallery filters: seconds on a local SSD, longer than the hardcoded 30 s on a
constrained runner, where the launcher declared the API dead while it was still
working. Readiness is decided by probing /health, so the ceiling is now 180 s and
configurable through OIP_STARTUP_TIMEOUT, in the frontend launcher and in the
Skill's gallery bridge.

a0567b9 drifted the README counts again. The republished dataset moved to
14,749 prompts / 24,578 images while both READMEs still said 14,693 / 24,479, so
main currently fails npm run verify:docs:

README.md: states 14,693 for prompts, but data/public-corpus.json has 14,749
README.md: states 24,479 for images, but data/public-corpus.json has 24,578
README.zh-CN.md: (same two)

That check landed in #4 and caught the drift on the very next release — the same
mistake previously survived an entire release cycle unnoticed.

Still open, needs a human

npm run test:retrieval remains red on main, unchanged by the new dataset. The
gated benchmark fails closed on related results with no judgement in
evals/retrieval/intents.jsonl:

related_precision  0.5  (gate: >= 0.95)
related_unjudged   5    (gate: == 0)
query tweet_id author
R01 · Cinematic female portrait in a neon rainy alley 2053202321799348299 @Document195
R30 · Black-and-white low-key noir alley portrait 2074019223266037846 @dreamydigiarts
R30 · Black-and-white low-key noir alley portrait 2058141014112481356 @Minahil42298354

Resolving it means looking at those images and recording a real verdict. Inventing
fixture entries to turn the suite green would destroy what the hand-judged
benchmark is for, so it is deliberately left alone.

Separately worth a decision

fetch_dataset.py extracts image packs but never removes files whose database rows
disappeared in a later release. On a checkout that had pulled images before
a0567b9, npm run verify:data now fails with 669 files on disk are not referenced by the DB. CI cannot hit this (fresh checkout, --db-only), and the
README says re-running the pull is safe, so the two disagree. Pruning orphans is a
policy call rather than an obvious bug fix — not touched here.

Test Plan

  • npm run data:pull:db verifies sha256 against the new manifest
  • verify:docs, test:api, test:gallery, npm --prefix web test, lint,
    build — all exit 0
  • OIP_STARTUP_TIMEOUT accepts a positive value and rejects 0, negative and
    non-numeric input in both the Python and Node paths
  • test:retrieval exits 1 — pre-existing, see above
  • verify:data exits 1 only on a checkout carrying pre-a0567b9 images, see above

The first start expands an 80 MB archive into a ~270 MB SQLite database and warms
the gallery filters. That is seconds on a local SSD, but the new macOS CI job hit
the hardcoded 30s ceiling and reported the API as never ready while it was still
making progress.

Readiness is decided by probing /health, so the ceiling only matters when startup
is genuinely slow: raise it to 180s and make it configurable through
OIP_STARTUP_TIMEOUT, in the frontend launcher and in the Skill's gallery bridge.
The gallery wait can afford to be generous now that a failed bind is detected from
the child exiting rather than by waiting out the timeout.

Also document that the retrieval benchmark fails closed on unjudged related
references, so a red suite is not mistaken for a broken checkout.
The republished dataset moved to 14,749 prompts and 24,578 images while both
READMEs still quoted the previous 14,693 and 24,479. Read the values from
data/public-corpus.json rather than restating them by hand.

verify_docs.py caught this on the first release after it landed, which is the
drift it exists to prevent.
The benchmark reported five unjudged related references. Reviewing the images
found four genuinely good and one wrong: "black-and-white low-key noir alley
portrait" was returning a bright, smiling, natural-daylight lifestyle portrait
that shares only the monochrome treatment and a brick wall.

The cause is what relaxation means. Relaxing `lighting:low-key` was read as "the
reference need not declare low-key", which also admitted references declaring the
opposite - and for a noir query the lighting *is* the requested grammar, so
dropping it changes the genre rather than loosening a detail. Reject a candidate
carrying a tag that contradicts the facet being relaxed. The map holds formal
opposites only, and is applied inside the related channel, so the exact channel is
untouched: strict precision, yield, average results, zero-result count and the
per-locale breakdown are all bit-identical. Related precision goes 0.9 -> 1.0 with
combined_useful_at_5 unchanged, so no coverage was lost.

Also let the fixture record a rejection. `related_rejected_ids` separates "nobody
looked at this" from "someone looked and turned it down", which the benchmark used
to conflate - a red run could not say whether it meant a review backlog or a
quality regression. The rejected reference stays recorded so its return would fail
the gate again.

Verdicts: R01 2053202321799348299 relevant; R30 2074019223266037846 relevant;
R30 2058141014112481356 rejected.
Extraction never deletes, so a release that drops records leaves its files in
images/. The README says re-running the pull is safe, but verify:data then failed
with "669 files on disk are not referenced by the DB" - the two disagreed, and the
message named no remedy.

A leftover file is a stale local artifact, not corruption: the gallery never
references it. Report it and continue, and keep it fatal only under
OIP_REQUIRE_IMAGES=1, which already gates the missing-file direction the same way.

Deleting stays opt-in through `fetch_dataset.py --prune`, which removes what the
current database does not reference and clears the emptied directories. Automatic
deletion is the wrong default: the files are re-downloadable, but someone may be
holding an older corpus deliberately.
@NanmiCoder
NanmiCoder merged commit dd8d3ce into main Jul 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant