Skip to content

submit.md

Latest commit

 

History

History
157 lines (115 loc) · 8.33 KB

submit.md

File metadata and controls

157 lines (115 loc) · 8.33 KB
Raw

Kaggle Submissions

Owns: how to build, submit, track, and pull replays for an Orbit Wars submission.

Requirements

  • kaggle CLI ≥ 2.1.2 (the competitions episodes / replay / logs subcommands and the numeric ref column in competitions submissions were added in 2.x). Upgrade with pip install --user --upgrade kaggle. If kagglesdk lands without an __init__.py (silent install corruption), repair with pip install --user --force-reinstall --no-deps kagglesdk==<latest>.
  • Kaggle API token at ~/.kaggle/kaggle.json.

Per-submission workflow

Every submission gets a tracked unique ID. The flow is package → submit → record → (later) download replays. State lives in submissions/SUBMISSIONS.jsonl (append-only manifest) with one row per packaged tarball.

1. Package

python scripts/package_kaggle_submission.py \
  --weights models/value_net_v1_kaggle/<Cn>.npz \
  --tuning-jsonl data/optkaggle/<Cn>.jsonl \
  --label <Cn> \
  --note "<short description for the kaggle message>"

What it does:

  • Picks the best non-pruned trial from the tuning JSONL (highest mean_value); pass --optim-config <path> instead to skip selection.
  • Forces the packaged optim_config.json to budget.budget_s = 1.0 for Kaggle. The rest of the selected optimizer config is preserved.
  • Generates a submission ID {label}-{UTC-compact}-{config_hash[:8]} (e.g. C1-20260509T112632Z-f299cd59) and records the current short git commit.
  • Writes a tarball at submissions/submission_{id}.tar.gz containing the live main.py, the root-level local Python import closure discovered from it, the .npz, optim_config.json, and PROVENANCE.json with git_commit.
  • The live main.py recovers its install dir via __file__ with a sys._getframe().f_code.co_filename fallback (Kaggle's loader execs main.py without setting __file__ and from a different cwd — the frame's co_filename is the durable handle).
  • Appends a row to submissions/SUBMISSIONS.jsonl with submission_id, git_commit, tarball+weights sha256, tuning provenance, dev mean_value, and a placeholder kaggle_status="not_submitted".
  • Records agent_file_hashes plus the two share-MLP weight hashes in the manifest so exact duplicate submissions can later be grouped for the hyperparameter-search noise-floor estimate.
  • Prints the exact kaggle competitions submit … -m "id=… | git=… | …" line to use next.

2. Submit

Use the kaggle competitions submit line printed by the package step. The message must start with id=<submission_id> so the recorder can match later:

kaggle competitions submit orbit-wars \
  -f submissions/submission_<id>.tar.gz \
  -m "id=<id> | git=<short_sha> | <Cn> | <note>"

Wait until status leaves PENDING:

kaggle competitions submissions orbit-wars

The CLI shows a numeric ref column — that's the durable Kaggle submission ID used for episodes/replays.

3. Record

Pulls the ref, status, public score, and Kaggle-side timestamp into the manifest row:

python scripts/record_kaggle_submission.py --submission-id <id>

Matches the manifest row to the Kaggle submission by id=<submission_id> substring in the description column. Run again later to refresh publicScore as TrueSkill keeps mixing.

4. Download replays and logs

Once kaggle_submission_ref is recorded, pull replays for that submission into data/replays/kaggle/<submission_id>/:

# all replays as JSON
python scripts/download_kaggle_replays.py --submission-id <id>

# also fetch agent logs (one HTTP per seat)
python scripts/download_kaggle_replays.py --submission-id <id> --include-logs --seats 4 --sleep 0.2

# limit while iterating
python scripts/download_kaggle_replays.py --submission-id <id> --max-episodes 5

The script is idempotent (skips files already on disk, --force overrides) and updates the manifest row with replays_local_dir + replays_episodes. Replays land as episode-<EPISODE_ID>-replay.json; logs as <EPISODE_ID>.agent<seat>.log.

Typical session

# Local smoke test before submitting
mkdir -p /tmp/sub && tar -xzf submissions/submission_<id>.tar.gz -C /tmp/sub
python -c "
from kaggle_environments import make
env = make('orbit_wars', debug=True)
env.run(['/tmp/sub/main.py', 'random'])
print([(i, s.reward, s.status) for i, s in enumerate(env.steps[-1])])
"

# Submit + record
kaggle competitions submit orbit-wars -f submissions/submission_<id>.tar.gz -m "id=<id> | git=<short_sha> | <Cn> | …"
python scripts/record_kaggle_submission.py --submission-id <id>

# Later: pull replays for analysis
python scripts/download_kaggle_replays.py --submission-id <id>

# Leaderboard check
kaggle competitions leaderboard orbit-wars -s

Noise floor / protocol.json

Two separate layers — don't conflate them (the Kaggle leaderboard is TrueSkill-rated, μ₀≈600, so its rating noise is a different unit from the local screen):

  • Local T3 screen δ_min (the one the BO loop gates on). data/protocol.json:delta_min = 0.05 is the local pool-mean-placement tie band (one 40-game battery-mean SE, measured 0.043; methodology §1.1/§7.4). It needs no Kaggle data and is final (delta_min_source: local_screen_floor). Stage-3 screen_decision.py reads only this. A fresh checkout (gitignored data/ tree) falls back to the same 0.05.
  • Kaggle T4 promotion (deferred). data/protocol.json:kaggle_t4 records a pragmatic ~60-rating-point margin as provenance. The principled test is Δμ > k·√(σ²_cand + σ²_inc) once per-submission σ/games are captured — see kaggle_t4.todo.

scripts/noise_floor.py is retained for documenting the T4 layer only (it must not feed the local screen). Its modes inspect rating spread among prior submissions:

python scripts/noise_floor.py exact                      # group converged duplicate identities
python scripts/noise_floor.py historical-similar \
  --submission-id <id-a> --submission-id <id-b> \
  --similarity-note "<why these rows are near-identical>"
python scripts/noise_floor.py --write-protocol placeholder \
  --similarity-note "<note>"                              # retain delta_min=0.05

exact groups converged rows by manifest_hash once available, otherwise by git_commit + config_hash + agent_file_hashes + share_mlp1_sha256 + share_mlp2_sha256; with several usable groups it reports the largest spread. historical-similar requires a curated row set + note. --write-protocol writes the result to data/protocol.json.

Manifest fields

submissions/SUBMISSIONS.jsonl rows carry everything needed to reconstruct or re-fetch a submission:

field meaning
submission_id {label}-{UTC}-{config_hash[:8]}; appears in the kaggle message
label, note human tags
git_commit short source commit recorded at package time
agent_file_hashes source-file content hashes used with weight hashes for exact duplicate grouping until manifest_hash lands
tarball, tarball_sha256 the exact bytes Kaggle received
share_mlp1, share_mlp1_sha256, share_mlp2, share_mlp2_sha256 source .npz files for the live share-net heads and their hashes
tuning_jsonl / optim_config_source how the OptimizerConfig was chosen
config_hash, trial, dev_mean_value packaged config hash plus optimizer-search provenance
kaggle_message the -m string used at submit time
kaggle_submission_ref numeric ref; required by competitions episodes/replay/logs
kaggle_status, kaggle_public_score, kaggle_submitted_at filled by record_kaggle_submission.py
replays_local_dir, replays_episodes filled by download_kaggle_replays.py

Other CLI shortcuts

kaggle competitions download orbit-wars -p orbit-wars-data    # competition data
kaggle competitions submissions orbit-wars                     # status + ref column
kaggle competitions episodes <REF>                             # episodes for one submission
kaggle competitions replay <EPISODE_ID> [-p ./out]             # one replay JSON
kaggle competitions logs <EPISODE_ID> <SEAT> [-p ./out]        # one agent's stdout/stderr
kaggle competitions leaderboard orbit-wars -s                  # leaderboard

Single-file submissions still work for quick experiments (kaggle competitions submit orbit-wars -f main.py -m "…"), but bypass the tracking flow above — only use them for throwaway tests.