Skip to content

Ship the dataset through GitHub Releases - #3

Merged
NanmiCoder merged 1 commit into
mainfrom
data-channel
Jul 25, 2026
Merged

NanmiCoder merged 1 commit into
mainfrom
data-channel

Conversation

@NanmiCoder

Copy link
Copy Markdown
Owner

Moves all dataset assets (SQLite archive + monthly image packs) from Git LFS to GitHub Releases.

Why: the free LFS quota (1 GiB storage / 1 GiB bandwidth per month) cannot hold the full corpus, and every archive update would consume LFS storage permanently. Release assets are free, support 2 GB per file, and are not bandwidth-throttled.

What changes

  • scripts/fetch_dataset.py downloads and sha256-verifies every asset listed in data/dataset-manifest.json (per-asset release tags, so unchanged months are never re-uploaded)
  • start.sh / start.bat / Dockerfile / CI fetch data instead of git lfs pull; a fresh clone is ~3 MB
  • dataset grows to the full public corpus: 15,033 prompts / 25,215 images / 29,388 translations / 170,238 active v2 labels
  • verify_public_data.py now checks manifest counts, referential integrity, absence of process tables, and image signatures

Verification

  • npm test green on the new archive (retrieval benchmark: strict precision 0.9567, combined useful@5 0.6917, 0 hard violations)
  • offline full fetch + OIP_REQUIRE_IMAGES=1 verify: 25,215 DB rows ↔ 25,215 files
  • fresh-clone e2e: fetch → API (/api/catalog full counts) → skill search (1,503 matches)
  • real-release download verified against manifest sha256

Move all data assets (SQLite archive + images) out of Git LFS: the free
LFS quota (1 GiB) cannot hold the full corpus, and every archive update
would burn storage forever. Assets now live on GitHub Releases with
per-asset release tags and sha256 pins in data/dataset-manifest.json;
scripts/fetch_dataset.py (npm run data:pull) downloads and verifies them.

- dataset grows to the full public corpus: 15,033 prompts / 25,215
  images / 29,388 translations / 170,238 v2 prompt labels; the
  evaluation-gate selection is retired
- start.sh / start.bat / Dockerfile / CI fetch data instead of lfs pull;
  image packs are optional (source-URL fallback covers the gallery)
- verify_public_data.py checks manifest counts, referential integrity,
  absence of labeling process tables, and image signatures for whatever
  packs are present (OIP_REQUIRE_IMAGES=1 for full setups)
@NanmiCoder
NanmiCoder merged commit f31f308 into main Jul 25, 2026
1 check passed
@NanmiCoder
NanmiCoder deleted the data-channel branch July 25, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant