ADFA-6618: Prototype Help+Manual topics as JSON plus one derived Pebble template

[davidschachterADFA]

ADFA-6618

Prototype: store each Help+Manual topic once, as JSON, and render its page through one Pebble template derived from the export. Code On The Go already renders such rows (a Content row with a templateId holds JSON; the app renders the named template with it), so this first phase needs no app change.

Draft: a prototype for review of the approach. Phase 2 needs CodeOnTheGo #2162. With an older app, in-page navigation, printing and topic popups break on a phase-2 database.

Ships after #41 and #42, which add steps 1 to 5 of the run order (see Notes).

Why

Every Help+Manual topic is stored twice:

• as its page, i/X.html (359 pages), each carrying a ~43 KB skin shell that is almost identical across pages;
• as a JavaScript copy, i/jstopics/X.js, holding the topic's fields for Help+Manual's in-page navigation.

Change

• scripts/hm_pages_to_templates.py DB [--dry-run] [--workers N] [--verbose]:
  • Which pages: a topic is an i/X.html with an i/jstopics/X.js. The jstopics body locates the page's body.
  • Deriving the template: the pages are aligned line by line. Lines shared by every page become the template's static text; the places between them are holes, filled from the topic's fields.
    • Where pages need different patterns, the template chooses with {% if field is empty %}.
    • A pattern is accepted only if rendering it reproduces each page exactly.
    • On the 2026-10-13 database the result is helpmanual-topic.peb: 43,854 bytes, with 15 holes.
  • Rewriting the rows: each topic row becomes JSON (file plus the ten jstopics fields, each as the page shows it, inserted | raw), with templateId set. A few pages carry js_ or part_ keys where their text differs.
  • Check before writing: every converted page is rendered with the real Pebble engine before anything is written. A page that doesn't render back stays as HTML and is named.
  • Run shape: one transaction, backup and VACUUM only on change, a second run changes nothing, and a dry run writes nothing.
• scripts/hm_render_check/Render.java: renders with Pebble 4.1.1 built as the app builds its engine (default settings, the same maxRenderedSize, Gson LONG_OR_DOUBLE). It runs from source with java, with the jars taken from the Gradle or Maven cache.

Results on the 2026-10-13 database

• Conversion: 359 of 359 topics convert. The 7 non-topic pages under i/ are left alone.
• Rendering: all 359 pages render back byte-identical with Pebble 4.1.1. This was also checked independently, rendering every converted row and comparing it with the original page.
• Headless Chrome, network limited to 127.0.0.1:
  • 0 DOM differences (full outerHTML);
  • 0 differing pixels in 718 screenshots at phone and desktop widths;
  • 27 interaction cases identical on the original and converted pages: next, previous, contents, Back, expanding sections, image zoom, search with highlighting, permalink, print, the _hmXtopic.htm page, toolbar icons, and the phone navigation bar.
• Sizes:

Stage	File bytes	Change
Original	175,982,592
Phase 1 (this PR)	175,437,824	-544,768
Phase 2 (simulated)	174,927,872	-509,952 more

• Phase 1: the 359 page rows go from 971,640 to 422,321 stored bytes, plus the 43,854-byte template, which is stored uncompressed because the app's loader reads Templates as plain text. Decoded, the pages go from 19.3 MB to 4.2 MB.
• Render time on a desktop JVM: 0.1 to 0.3 ms per page (median). The phone is not yet measured.

Device check

On a Pixel 6 Pro (Android 17), with the installed debug build and the database in /sdcard/Download. The log confirmed each switch ("Swapped to the debug database").

Font scale 1.0, with the unmodified 2026-10-13 database and then with this step's output:

• Pages: ten topic pages, including the two with js_ and part_ keys, plus the search page and the print window (which loads jstopics).
• Interactions: the expanding-text toggle, expanding sections, image zoom, and in-page navigation to the next topic.
• Checks on each: title, headings, links, toggles, loaded and broken images, the 27 icons and their references, and script errors.
• Result: all identical, and all 15 screenshots pixel-identical below the status bar. The search page shows one script error with both databases (xmessage.js reading searchSession); it isn't new.

Font scale 2.0, five pages plus the same interactions:

• Checks: all identical.
• Screenshots: 4 of 7 pixel-identical. The other 3 differ in text anti-aliasing only:
  • layout measured on the page (header, body, scroller and heading boxes, scroll position) is identical to a thousandth of a CSS pixel;
  • a second run of the unmodified database differed from its own first run in the same way on one page.

Phase 2 (--drop-jstopics, 5e43917)

• Why an option of the same step: the jstopics template reads js_ keys that this run writes into the page JSON, so the check before dropping a row has to use exactly the JSON being stored. Without the option, a phase-1 database works with today's app.
• The template: derived from the export and stored as helpmanual-jstopic.peb (606 bytes, plain text like the page template).
• Check before dropping:
  • Only i/X.html pages are topics, as in the app's mapping.
  • A row, chunks included, is dropped only when its page ends up stored as JSON, and the real engine rebuilds that file from the page JSON. The rebuilt file must have the same field values after JavaScript unescaping, and the same surrounding text.
  • Any row left in place fails the run.
• Later runs:
  • They check every dropped topic against the stored template, reading the body form from it.
  • A page that goes back to HTML gets its row written back, built the way the app builds it, and the run exits 1. If the row can't be built, the run writes nothing.
• Review fixes: bab8567, fd9d6b2 and 3c98b71. The output for the 2026-10-13 database is byte-identical before and after them (sha256 ad12ee32...).
• App rule (CodeOnTheGo #2162): a miss for i/jstopics/X.js renders helpmanual-jstopic.peb with i/X.html's JSON as text/javascript. Both transports strip ?_= query strings before the lookup.
• Real data:
  • all 359 rows dropped;
  • the file goes from 175,982,592 to 174,927,872 bytes (-1,054,720 with phase 1);
  • every jstopics URL, served by a local stand-in for the app rule, has the stored file's field values;
  • 27 functional cases in Chrome are identical;
  • a second run changes nothing.
• Checked independently: rendering the template with each page's JSON gives the original field values for all 359.
• On the phone, with a debug build of CodeOnTheGo #2162 and a --drop-jstopics database: in-page navigation and the print window loaded their jstopics files through the app rule (status 200). Every check was identical to the unmodified database at font scale 1.0 and 2.0; details are in #2162.

Notes

• Run order:

  1. remove_skin_font_comment
  2. bundle_remote_images
  3. remove_remote_loads
  4. share_skin_sprite
  5. check_remote_loads
  6. the media steps, with the snapshot before them
  7. this step
  8. check_remote_loads again

  check_remote_loads reads Templates and JSON pages, so it works on either side.

• Media rollback: it can't undo the converted pages. Start it from this step's pre-run backup, kept with the rollback bundle, then rerun this step. This is documented in the docstring and in CLAUDE.md.

• Combined database: a database with every step from ADFA-6589: Keep documentation content on the device #41, ADFA-4195: Move Help+Manual's inline icon sprite into one shared script #42, ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38, ADFA-6596: Convert the documentation videos to VP9 WebM from their originals #40 and this PR ran end to end on the 2026-10-13 database: 112,283,648 bytes, with all checks passing.

• ZOOMSTOP/ZOOMRESTART: the markers sit in the skin, outside the body, so nothing needs stripping.

• CI: the step needs Java 17 and the Pebble jars, which no workflow provides yet.

• ADFA-4195: Move Help+Manual's inline icon sprite into one shared script #42 still helps: run first, it keeps the 16,817-byte sprite out of the uncompressed template.

Testing

• Tests: scripts/test_hm_pages_to_templates.py, 36 tests, run under pytest and as a plain script. They skip if java or the jars are missing.
• Mutations: 33 of 35 fail the test named for them. The two survivors are second guards behind the X.html rule, which make them unreachable on their own. The second review found three more survivors; tests added in 3c98b71 kill them.
• Review: the independent review's ten findings are answered in bab8567 and fd9d6b2, and the second review's three in 3c98b71 (see the comments).

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent review

An independent Claude Code review covered this PR at 5e43917. It has no critical findings; one high, two medium, three low and four nits. The derivation and the render check do what this description says on the 2026-10-13 database, and every number in it was reproduced. Three findings need fixing before merge. All are open.

ID	Sev	File:line	Finding
R43-01	high	hm_pages_to_templates.py:1069	Topic pages not named .html: jstopics dropped, pages stay HTML, exit 0
R43-02	medium	hm_pages_to_templates.py:1078	A page reverted to HTML after phase 2 loses its jstopics; exit 0
R43-03	medium	hm_pages_to_templates.py:113	The media rollback can't undo i/ pages after this step; undocumented
R43-04	low	CLAUDE.md; docstring :108	Step missing from CLAUDE.md; run order omits share_skin_sprite.py
R43-05	low	hm_pages_to_templates.py:964	A rerun after the drop assumes one body form: false "broken", exit 1
R43-06	low	test_hm_pages_to_templates.py	No tests for R43-01, 02 and 05; 7 guard mutants survive
R43-07	nit	hm_pages_to_templates.py:771	Render pins Gson 2.10.1; the app's :common uses 2.12.1
R43-08	nit	hm_pages_to_templates.py:344	Dead code: Derivation.base, the harness database argument, Render --db
R43-09	nit	helpmanual-topic.peb:444	Home link keyed on hmPrevLink is empty: fits this data only
R43-10	nit	hm_pages_to_templates.py:896	LIKE prefix is case-blind and _ is a wildcard; the slice assumes the prefix

R43-01 (also CodeOnTheGo #2162's R2162-04).

• Reproduced: with synthetic topics stored as i/tN.htm, --drop-jstopics printed "6 of 6 ... render back", wrote 0 pages as JSON, and dropped all 6 jstopics rows with exit 0.
• Cause: candidates are any page, but the drop looks the page up as {stem}.html, and a row is droppable once its JSON passed the check, whether or not it was written. The app maps only to X.html, so every jstopics request for such a topic gets a 404.
• Today: all 359 topics are .html.
• Suggested fix:
  • accept only X.html topics;
  • drop a row only when its page is stored as JSON;
  • add a .htm test.

R43-02. On a second run where a page no longer renders back, the page goes back to HTML, but its jstopics row is already gone and the run exits 0. The app needs a templated page, so that topic's in-page navigation, print and popups break. Suggested fix: write the row back, or refuse and exit 1.

R43-03. After this step, update_media_references.py --restore-originals exits 1 and leaves Help+Manual references on converted names:

• the 359 page rows are JSON and match no digest;
• 47 of the dropped jstopics rows referenced images;
• one skin image reference now lives only in Templates.

Meanwhile optimize_db_media.py --restore-originals still deletes the converted files. Suggested fix: document that a media rollback starts from this step's pre-run backup, then reruns this step.

R43-05. On an export whose pages keep Help+Manual's \n\r breaks, the second run reports six false "broken" topics and exits 1. The cause is the hard-coded BODY_FORMS[0].

Checked and found correct

• Reproduced:
  • 359 of 359 topics converted;
  • the 43,854-byte template with 15 holes;
  • page bytes 971,640 to 422,321;
  • file size 175,982,592, then 175,437,824, then 174,927,872;
  • 359 rows dropped;
  • a rerun says "Nothing to change".
• Exact rendering: the stored template plus each page's JSON reproduces all 359 pages byte for byte.
• The rebuilt jstopics files: evaluating hmLoadTopic in Node gives identical objects for all 359, independently of the script's own unescaper.
• Database diff: only the 359 page rows changed, the 359 jstopics rows are gone, and two Templates rows were added.
• Render.java against the app: the same builder, character limit, blank/null refusal and Gson policy, Pebble 4.1.1 and unbescape 1.1.6.
• Escaping: {{, {% and {# are escaped, newline trimming is compensated, and every value tag uses | raw or | escape('js'). CRLF, BOM and non-ASCII round-trip.
• Drop safety, apart from R43-01 and 02: checks before dropping, chunks with the row, one transaction, the backup before BEGIN, correct exit codes.
• Interface with #2162: the same template name, the same page JSON, text/javascript, query strings stripped, dotted names handled.
• Mutations: 41 run; every mutated guard fails its test, apart from the 7 survivors in R43-06.

Not verified:

• the Chrome and device results, which weren't rerun;
• render time on a phone;
• a release (R8) build.

🤖 Generated with Claude Code

[davidschachterADFA]

Review answered

All ten findings are fixed in bab8567 and fd9d6b2. For the 2026-10-13 database the output is byte-identical before and after the fixes (sha256 ad12ee32...), and a second run changes nothing.

ID	Sev	Status
R43-01	high	Fixed: only i/X.html pages are topics; a row is dropped only when its page is stored as JSON; .htm test
R43-02	medium	Fixed: a page back to HTML gets its row written back (built as the app builds it) and the run exits 1; if it can't be built, nothing is written
R43-03	medium	Documented: a media rollback starts from this step's pre-run backup, then reruns this step (docstring, CLAUDE.md)
R43-04	low	Fixed: CLAUDE.md bullet; the full eight-step order, share_skin_sprite.py before this step, check_remote_loads.py either side
R43-05	low	Fixed: later runs read the body form from the stored template; a form counts only when it maps the body back exactly
R43-06	low	Fixed: tests for R43-01, 02 and 05, and for six of the seven survivors
R43-07	nit	Fixed: Gson 2.12.1, as in the app's :common
R43-08	nit	Fixed: dead code removed (Derivation.base, the harness database argument, Render --db)
R43-09	nit	Fixed: comment that the Home-link condition fits this export's data
R43-10	nit	Fixed: exact prefix match, checked before slicing

Checked:

• 32 tests pass, rerun independently.
• 33 of 35 mutations fail the test named for them. The two survivors are second guards behind the X.html rule, kept as the review asked.
• The phase-2 output with the fixes matches the output without them byte for byte, checked independently on fresh copies.

New behaviour: with --drop-jstopics, any jstopics row left in place now fails the run, including a row with no X.html page.

🤖 Generated with Claude Code

[davidschachterADFA]

Independent review of the fix commits

An independent Claude Code review covered bab8567 and fd9d6b2, which answer the first independent review (R43-01 to R43-10). It has 3 nits. Every finding is fixed or documented as the response says, and the real-data output is byte-identical to the one the description names (sha256 ad12ee32...). The nits are an exit status that doesn't match the new "any row left fails" rule, three untested guards, and an unstated merge order. All are open.

ID	Sev	File:line	Finding
R43-11	nit	hm_pages_to_templates.py:1031,1292	A topic whose jstopics file lacks hmBody keeps its row with exit 0
R43-12	nit	hm_pages_to_templates.py:952,1164,1301	Three new guards survive mutation
R43-13	nit	CLAUDE.md:183; docstring :118	Docs name five scripts this branch lacks; merge order not stated

R43-11.

• Reproduced: the test fixture with i/jstopics/t3.js replaced by hmLoadTopic({ hmTitle:"Third" });, run with --drop-jstopics: exit 0, i/jstopics/t3.js still stored, and only "stays HTML: i/t3.html - its jstopics file has no hmBody field" on stderr. A page that is not valid UTF-8 takes the same path.
• Cause: those two checks put the topic in unfit before it becomes a Topic, so it is in neither kept nor orphans, and failed_drops never sees it. A topic found unfit later, by the derivation or the render check, is in kept and fails the run. Keeping the row is right, since the page stays HTML. The exit status contradicts the response's "with --drop-jstopics, any jstopics row left in place now fails the run", and the PR body's "Any row left in place fails the run".
• Suggested fix: with --drop-jstopics, count a candidate's row that is still stored and not dropped as kept, whatever the reason, or narrow the claim.

R43-12.

• Reproduced: each mutant applied alone at fd9d6b2, all 32 tests pass:
  • the written-back row's field check reduced to if parsed is None: (line 1164), so a row that renders but holds other fields is written back anyway;
  • refuse dropped from the exit status (line 1301). The one refusal test still exits 1, only because deleting the jstopics template also marks every dropped topic broken. A refusal for a missing text/javascript content type would exit 0;
  • the jstopics/ lookup back on LIKE (line 952). test_rows_the_step_does_not_own_are_left_alone pins the page side only, and runs without --drop-jstopics, where an extra I/jstopics/ orphan changes nothing.
• Cause: these are not the two "second guards behind the X.html rule" the PR names. The tests reach each line, but no assertion depends on it.
• Suggested fix: a write-back case where the stored jstopics template reads a missing js_ key; a refusal case with the template present and no text/javascript type; the case-only prefix test run with --drop-jstopics.

R43-13.

• Reproduced: at fd9d6b2, remove_skin_font_comment.py, bundle_remote_images.py, remove_remote_loads.py and check_remote_loads.py (ADFA-6589: Keep documentation content on the device #41) and share_skin_sprite.py (ADFA-4195: Move Help+Manual's inline icon sprite into one shared script #42) are not in scripts/. CLAUDE.md's new bullet and the docstring's run order name all five as steps to run first.
• Why it matters: if this PR merges before ADFA-6589: Keep documentation content on the device #41 and ADFA-4195: Move Help+Manual's inline icon sprite into one shared script #42, CLAUDE.md tells a reader to run files that don't exist. The description says phase 2 needs CodeOnTheGo #2162, but not that it ships after ADFA-6589: Keep documentation content on the device #41 and ADFA-4195: Move Help+Manual's inline icon sprite into one shared script #42.
• Suggested fix: add "Ships after ADFA-6589: Keep documentation content on the device #41 and ADFA-4195: Move Help+Manual's inline icon sprite into one shared script #42" to the description, as ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38 does for CodeOnTheGo #2158, or mark those steps as coming from them.

Earlier findings

ID	Verdict	Evidence
R43-01	Fixed	Restoring the old candidate rule (any extension, stem by the last dot) fails test_topic_page_not_named_html_keeps_its_jstopics_row. The .html check alone and the stored_as_json filter alone are the two equivalent survivors the PR names.
R43-02	Fixed, with R43-12 left	Never writing back fails both write-back tests; no refusal without a template, writing despite a refusal, exit 0 after a write-back, text/html as the type, or UPDATE in place of INSERT each fail one.
R43-03	Documented	Docstring and CLAUDE.md say a media rollback starts from this step's pre-run backup.
R43-04	Fixed	CLAUDE.md bullet and the eight-step order; see R43-13.
R43-05	Fixed	Assuming the first body form on later runs, or accepting a form that doesn't map the body back, fails test_export_whose_bodies_keep_their_crlf_runs_twice_cleanly; body_form_of always returning the unfiltered form fails three tests.
R43-06	Fixed	The new tests cover R43-01, 02 and 05 (above); R43-12 lists what is still unpinned.
R43-07	Fixed	CodeOnTheGo's libs.versions.toml gives google-gson 2.12.1, which :common uses.
R43-08	Fixed	No Derivation.base, harness database argument, --db or sqlite-jdbc remains.
R43-09	Fixed	Comment in the docstring and at the {% if %} search.
R43-10	Fixed for pages	Going back to LIKE for pages fails test_rows_the_step_does_not_own_are_left_alone; the jstopics/ side is in R43-12.

Checked and found correct

• Real data at fd9d6b2, on copies of the 2026-10-13 database:
  • phase 1: 359 of 359 topics render back and are stored as JSON, the template is 43,854 bytes with 15 holes, pages go from 971,640 to 422,321 stored bytes, and the file is 175,437,824 bytes;
  • --drop-jstopics on a fresh copy: 359 rows dropped, 174,927,872 bytes, sha256 ad12ee328a416b40..., the description's ad12ee32...; a rerun says "Nothing to change" and leaves the sha256 as it was;
  • all 359 jstopics rows have their X.html page, so the new orphan rule fails nothing here;
  • ADFA-6589: Keep documentation content on the device #41's check_remote_loads.py reports 90 loads on the phase-2 output, as the docstring says it does on the untreated database.
• The derived template: none of its 15 holes is in an attribute that loads a file (canonical link, links, <meta>, title, body and script strings), so ADFA-6589: Keep documentation content on the device #41's check, which reads JSON strings only when they hold a <, misses nothing here.
• Write-back rows: text/javascript, templateId 0, compressed as that type says, with leftover fragment slots cleared.
• Tests at fd9d6b2 (Python 3.14, Java 17, Pebble 4.1.1 and Gson 2.12.1 from the Gradle cache): 32 pass under pytest and as a plain script.

Not verified:

• The Chrome and phone results (not rerun).
• Render time on a phone.

🤖 Generated with Claude Code

[davidschachterADFA]

Response to the review of the fix commits

ID	Status	What changed
R43-11	Fixed in 3c98b71	A topic unfit before it is read (no hmBody, not UTF-8) or unreadable keeps its row as kept; --drop-jstopics exits 1
R43-12	Fixed in 3c98b71	A test for each of the three guards; each mutant now fails one
R43-13	Fixed in 3c98b71, and in the PR body	CLAUDE.md and the docstring name #41 (ADFA-6589) and #42 (ADFA-4195); the body says this ships after both

Decisions to check:

• R43-11: the first suggested fix, not narrowing the claim. The row is still kept, as the review says it should be; it is now listed ("keeps i/jstopics/t3.js - its page stays HTML (its jstopics file has no hmBody field)") and counted in "N jstopics row(s) left in place". A row whose page cannot be read is counted the same way; that run already exited 1 for the unreadable row.
• R43-12: the three suggested cases. For the write-back case, the stored jstopics template reads js_hmTitle, which no JSON holds. It renders, but with an empty title, so the row is refused and nothing is written. The other topics then show as broken too, because a later run checks them with the stored template; the test asserts the refusal and that the file is unchanged.

Checks:

• scripts/test_hm_pages_to_templates.py: 36 pass under pytest and as a plain script (Python 3.14, Java 17, Pebble and Gson jars from the Gradle cache). Both changed Python files compile under Python 3.11.16.
• 5 mutants, each applied alone; each fails the test named for it:
  • R43-11 revert -> test_topic_found_unfit_before_it_is_read_keeps_its_row_and_fails_the_drop (exit 0);
  • unreadable topics not counted -> the same test, at the "could not be read" line;
  • the write-back field check reduced to if parsed is None: -> test_page_back_to_html_whose_row_the_stored_template_builds_wrong_writes_nothing (no refusal);
  • refuse dropped from the exit status -> test_page_back_to_html_with_no_content_type_for_its_row_fails_the_run (exit 0);
  • the jstopics/ lookup back on LIKE without the prefix filter -> test_jstopics_rows_whose_prefix_matches_only_without_case_are_not_dropped_or_counted (exit 1).
• Real data, fresh copies of the 2026-10-13 database, at 3c98b71:
  • phase 1: 359 of 359 topics render back and are stored as JSON, a 43,854-byte template with 15 holes, 971,640 -> 422,321 stored bytes, file 175,437,824 bytes, exit 0;
  • --drop-jstopics: 359 of 359 rows dropped, file 174,927,872 bytes, sha256 ad12ee328a416b40..., the same as at fd9d6b2; a rerun prints "Nothing to change." and leaves the sha256 as it was.

🤖 Generated with Claude Code