Repository navigation
fix: restore the startup-timeout fix and refresh the dataset counts - #5
Merged
Merged
Conversation
The first start expands an 80 MB archive into a ~270 MB SQLite database and warms the gallery filters. That is seconds on a local SSD, but the new macOS CI job hit the hardcoded 30s ceiling and reported the API as never ready while it was still making progress. Readiness is decided by probing /health, so the ceiling only matters when startup is genuinely slow: raise it to 180s and make it configurable through OIP_STARTUP_TIMEOUT, in the frontend launcher and in the Skill's gallery bridge. The gallery wait can afford to be generous now that a failed bind is detected from the child exiting rather than by waiting out the timeout. Also document that the retrieval benchmark fails closed on unjudged related references, so a red suite is not mistaken for a broken checkout.
The republished dataset moved to 14,749 prompts and 24,578 images while both READMEs still quoted the previous 14,693 and 24,479. Read the values from data/public-corpus.json rather than restating them by hand. verify_docs.py caught this on the first release after it landed, which is the drift it exists to prevent.
The benchmark reported five unjudged related references. Reviewing the images found four genuinely good and one wrong: "black-and-white low-key noir alley portrait" was returning a bright, smiling, natural-daylight lifestyle portrait that shares only the monochrome treatment and a brick wall. The cause is what relaxation means. Relaxing `lighting:low-key` was read as "the reference need not declare low-key", which also admitted references declaring the opposite - and for a noir query the lighting *is* the requested grammar, so dropping it changes the genre rather than loosening a detail. Reject a candidate carrying a tag that contradicts the facet being relaxed. The map holds formal opposites only, and is applied inside the related channel, so the exact channel is untouched: strict precision, yield, average results, zero-result count and the per-locale breakdown are all bit-identical. Related precision goes 0.9 -> 1.0 with combined_useful_at_5 unchanged, so no coverage was lost. Also let the fixture record a rejection. `related_rejected_ids` separates "nobody looked at this" from "someone looked and turned it down", which the benchmark used to conflate - a red run could not say whether it meant a review backlog or a quality regression. The rejected reference stays recorded so its return would fail the gate again. Verdicts: R01 2053202321799348299 relevant; R30 2074019223266037846 relevant; R30 2058141014112481356 rejected.
Extraction never deletes, so a release that drops records leaves its files in images/. The README says re-running the pull is safe, but verify:data then failed with "669 files on disk are not referenced by the DB" - the two disagreed, and the message named no remedy. A leftover file is a stale local artifact, not corruption: the gallery never references it. Report it and continue, and keep it fatal only under OIP_REQUIRE_IMAGES=1, which already gates the missing-file direction the same way. Deleting stays opt-in through `fetch_dataset.py --prune`, which removes what the current database does not reference and clears the emptied directories. Automatic deletion is the wrong default: the files are re-downloadable, but someone may be holding an older corpus deliberately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two follow-ups to #4.
The startup-timeout fix never landed. #4 was merged while its head ref was
still
b0913c4, so the sixth commit — the fix for the macOS CI failure that PR'sown new job had just exposed — was left behind. Cherry-picked here unchanged.
The first start expands an 80 MB archive into a ~270 MB SQLite database and warms
the gallery filters: seconds on a local SSD, longer than the hardcoded 30 s on a
constrained runner, where the launcher declared the API dead while it was still
working. Readiness is decided by probing
/health, so the ceiling is now 180 s andconfigurable through
OIP_STARTUP_TIMEOUT, in the frontend launcher and in theSkill's gallery bridge.
a0567b9drifted the README counts again. The republished dataset moved to14,749 prompts / 24,578 images while both READMEs still said 14,693 / 24,479, so
maincurrently failsnpm run verify:docs:That check landed in #4 and caught the drift on the very next release — the same
mistake previously survived an entire release cycle unnoticed.
Still open, needs a human
npm run test:retrievalremains red onmain, unchanged by the new dataset. Thegated benchmark fails closed on
relatedresults with no judgement inevals/retrieval/intents.jsonl:205320232179934829920740192232660378462058141014112481356Resolving it means looking at those images and recording a real verdict. Inventing
fixture entries to turn the suite green would destroy what the hand-judged
benchmark is for, so it is deliberately left alone.
Separately worth a decision
fetch_dataset.pyextracts image packs but never removes files whose database rowsdisappeared in a later release. On a checkout that had pulled images before
a0567b9,npm run verify:datanow fails with669 files on disk are not referenced by the DB. CI cannot hit this (fresh checkout,--db-only), and theREADME says re-running the pull is safe, so the two disagree. Pruning orphans is a
policy call rather than an obvious bug fix — not touched here.
Test Plan
npm run data:pull:dbverifies sha256 against the new manifestverify:docs,test:api,test:gallery,npm --prefix web test,lint,build— all exit 0OIP_STARTUP_TIMEOUTaccepts a positive value and rejects0, negative andnon-numeric input in both the Python and Node paths
test:retrievalexits 1 — pre-existing, see aboveverify:dataexits 1 only on a checkout carrying pre-a0567b9images, see above