Skip to content

value_init.md

Latest commit

 

History

History
103 lines (69 loc) · 4.36 KB

value_init.md

File metadata and controls

103 lines (69 loc) · 4.36 KB
Raw

Value Spline Initialization

scripts/make_v3_init.py generates the schema-v10 warm-start checkpoint models/value_spline_v3_init.npz. The checkpoint is designed so every trainable monotone-spline slope starts gradient-alive and can be used directly with scripts/fit_value_spline.py --resume.

Use this doc when regenerating the v3 init checkpoint, changing resume behavior in fit_value_spline.py, or responding to outer-loop knot-drift warnings. For the runtime value-net contract, read value_net.md; for TD training mechanics, read td_training.md and td_outer_loop.md.

Quick Start

Regenerate and validate the init checkpoint:

python scripts/make_v3_init.py
python - <<'PY'
from value_net import load_weights
load_weights("models/value_spline_v3_init.npz")
print("ok")
PY

Run the planned five-iteration warm-start schedule:

bash scripts/run_v3init5.sh

Run focused tests after changing the init, resume path, clamps, or knot metrics:

pytest -q tests/test_fit_value_spline.py tests/test_td_outer_loop.py

Checkpoint Contract

models/value_spline_v3_init.npz is a ValueWeights schema-v10 file. The generator writes explicit literals for:

  • score splines: knots_ships, knots_prod, w_ships, w_prod, score biases;
  • time/map warps: warp_t_knots, warp slopes, map alphas, prod_reference;
  • hold/influence: H_hold, kernel_knots, kernel_values, kernel_blend_alpha;
  • gamma-logit head: final-ship and support-minus-threat knots, weights, and bias;
  • production horizon: knots_prod_horizon, w_prod_horizon.

scripts/make_v3_init.py refuses to write if any trainable slope has sigmoid(w) < 1e-5. Inactive higher-knot slopes use -7, not -100, so their forward slope is tiny but their gradient is still movable by Adam.

Resume Semantics

fit_value_spline.py --resume PATH must preserve the checkpoint's learned basis and head:

  • forces --freeze-knots;
  • forces --warmup-steps 0;
  • loads score knots, score weights, score biases, warp slopes, map alphas, w_prod_horizon;
  • carries the gamma-logit head through resume_weights.gamma_params() without reinitializing w_logit_final, w_logit_st, or bias_logit_final.

During training, post-step clamps keep the model away from gradient-dead regions:

  • bias_logit_final is clamped to [-4, 2];
  • w_logit_final, w_logit_st, and w_prod_horizon are floored at SLOPE_WEIGHT_FLOOR.

Rebaseline Knots

Score knots are frozen during resume training, but the feature distribution can drift as gamma and production horizon parameters move. fit_value_spline.py emits knot_coverage in weights.training.json; td_outer_loop.py reads it and stores a non-blocking knot_coverage verdict in each manifest iter record.

Outer-loop warnings fire when:

  • frac_above_top_knot > --knot-above-top-tau (default 0.30);
  • frac_below_first_interior_knot > --knot-below-first-tau (default 0.70).

Use a manual rebaseline when warnings persist:

python scripts/fit_value_spline.py \
  --resume runs/td_loop_<tag>/latest_good.npz \
  --rebaseline-knots models/value_spline_v3_rebaselined.npz \
  --output /tmp/unused.npz

The one-shot path recomputes knots_prod from the resume model's own forward pass using compute_warped_knots, writes the corrected checkpoint, prints old/new coverage, and exits before training. It intentionally does not auto-adjust knots inside the outer loop.

Five-Iteration Schedule

scripts/run_v3init5.sh drives td_outer_loop.py through five resumed invocations under runs/td_loop_v3init5/:

Iter lambda_td lambda_mc k
0 0.0 1.0 0.0
1 0.1 0.9 0.0
2 0.2 0.8 0.1
3 0.3 0.7 0.2
4 0.4 0.6 0.3

The schedule starts with MC supervision and no Branch A rows, then ramps TD and Branch A as the snapshot becomes less arbitrary.

Tests

Relevant tests in tests/test_fit_value_spline.py cover:

  • resume keeps the gamma-logit head instead of reinitializing it;
  • fit emits knot_coverage;
  • data-driven knot computation and warped-knot recomputation;
  • finite training metrics and gradient-safe initialization paths.

Relevant tests in tests/test_td_outer_loop.py cover knot-drift verdicts and missing-coverage handling.

Historical design rationale is archived in archived/init_weights_plan.md.