Repository navigation
Ship the dataset through GitHub Releases - #3
Merged
Merged
Conversation
Move all data assets (SQLite archive + images) out of Git LFS: the free LFS quota (1 GiB) cannot hold the full corpus, and every archive update would burn storage forever. Assets now live on GitHub Releases with per-asset release tags and sha256 pins in data/dataset-manifest.json; scripts/fetch_dataset.py (npm run data:pull) downloads and verifies them. - dataset grows to the full public corpus: 15,033 prompts / 25,215 images / 29,388 translations / 170,238 v2 prompt labels; the evaluation-gate selection is retired - start.sh / start.bat / Dockerfile / CI fetch data instead of lfs pull; image packs are optional (source-URL fallback covers the gallery) - verify_public_data.py checks manifest counts, referential integrity, absence of labeling process tables, and image signatures for whatever packs are present (OIP_REQUIRE_IMAGES=1 for full setups)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves all dataset assets (SQLite archive + monthly image packs) from Git LFS to GitHub Releases.
Why: the free LFS quota (1 GiB storage / 1 GiB bandwidth per month) cannot hold the full corpus, and every archive update would consume LFS storage permanently. Release assets are free, support 2 GB per file, and are not bandwidth-throttled.
What changes
scripts/fetch_dataset.pydownloads and sha256-verifies every asset listed indata/dataset-manifest.json(per-asset release tags, so unchanged months are never re-uploaded)start.sh/start.bat/ Dockerfile / CI fetch data instead ofgit lfs pull; a fresh clone is ~3 MBverify_public_data.pynow checks manifest counts, referential integrity, absence of process tables, and image signaturesVerification
npm testgreen on the new archive (retrieval benchmark: strict precision 0.9567, combined useful@5 0.6917, 0 hard violations)OIP_REQUIRE_IMAGES=1verify: 25,215 DB rows ↔ 25,215 files/api/catalogfull counts) → skill search (1,503 matches)