Skip to content

scope.md

Latest commit

 

History

History
552 lines (475 loc) · 34.1 KB

scope.md

File metadata and controls

552 lines (475 loc) · 34.1 KB
Raw

SpectreBoard on-device Parakeet dictation — scope

Append-only. New dated entries when decisions change; never edit/delete old entries.

2026-09-02 — Initial scope

Why: Comrade's server-side Whisper dictation (see homelab memory project_whisper_dictation_ffmpeg) works and is fixed, but round-trips audio over the network to comrade and back. User found Outspoke (github minburg/outspoke, GPLv3), an open-source Android keyboard doing fully on-device streaming ASR via NVIDIA Parakeet-TDT-0.6B-v3 (FastConformer encoder + TDT decoder/joint, ONNX Runtime Mobile) and reported it as good as or better than the comrade Whisper pipeline, zero corrections needed, on speech alone (no network round trip).

G5 NPU offload explored and abandoned this session: spent significant effort trying to get Parakeet's encoder running on the Pixel's Tensor G5 NPU via the ~/homelab/iree-stack/GoogleBeta LiteRT AOT pipeline (litert_torch export from the real nvidia/parakeet-tdt-0.6b-v3 HF checkpoint, then aot_compile_g5_tflite.py). The litert_torch conversion itself succeeded cleanly (real proven Google toolchain, handles >2GB models fine, unlike generic onnx2tf which hit hard FlatBuffers/protobuf 2GB ceilings on this model — documented separately in homelab memory project_g5_qwen_cache_bug-adjacent notes). But the actual G5 NPU AOT compile step crashes with a generic INTERNAL error from Google's closed-source beta compiler on this architecture, reproducibly, at every granularity tried: full encoder (3324 ops), and 4-way chunked splits down to ~840 ops each (both the subsampling-containing first chunk and pure repeated-Conformer-layer middle chunks). Partitioning always selects 100% of ops as NPU-eligible; the crash is in the actual native compile pass, not an op-support gap. Conclusion: this beta compiler cannot currently compile this Conformer architecture at all, at any of the granularities tried — treated as a real capability ceiling of the beta SDK, not a missing flag. Decoder/joint was separately ruled out for NPU regardless of this finding — DeepSeek's architecture review of Outspoke found decodeChunk runs the TDT joint network one encoder frame at a time, carrying LSTM state across thousands of tiny sequential inference calls per utterance — not a batchable, NPU-friendly workload by nature.

Decision: on-device Parakeet via ONNX Runtime (CPU), not NPU. SpectreBoard already depends on onnxruntime-android (used by GruScorer) and already has prior on-device Whisper-via-ORT work as precedent. The int8 ONNX files Outspoke ships (encoder-model.int8.onnx, decoder_joint-model.int8.onnx, nemo128.onnx, vocab.txt, config.json) are pulled from the phone, verified byte-identical to the canonical istupakov/parakeet-tdt-0.6b-v3-onnx HF repo, and already contain ORT-native dynamic-quantization ops ( DynamicQuantizeLSTM/DynamicQuantizeLinear) that ONNX Runtime handles natively — these are exactly the ops that blocked TFLite/G5 conversion, and are a non-issue here. Zero re-conversion needed; these are the files Outspoke itself runs today.

Port plan: ParakeetEngine.kt (Outspoke, GPLv3 — SpectreBoard/HeliBoard is also GPLv3, so this is a clean license-compatible port, not a rewrite) is a complete, self-contained SpeechEngine/ChunkStreamingEngine implementation: ONNX session management, the full greedy TDT decode loop with LSTM state carry-over, detokenization, blank-ID resolution, word-beam alternatives — all in one ~1100-line file depending only on ai.onnxruntime.* and a plain AudioChunk data holder. Porting this wholesale (per the project's own "port wholesale, don't hand-roll" lesson from the artemisd daemon) rather than reimplementing the TDT streaming contract from scratch. InferenceRepository (2544 lines — VAD/chunking/confidence-gating/post-processing orchestration around the engine) is NOT ported in this first pass; a first working version wires ParakeetEngine directly into the existing toggle-to-record dictation UX (StreamDictation-equivalent trigger in KeyboardActionListenerImpl), buffering the whole utterance and running one encode + greedy decode pass on stop, same shape as ParakeetEngine.transcribe(). True incremental streaming (partial results while still speaking, matching the current comrade-Whisper UX) is a later iteration once the basic on-device path is proven correct on-device.

Model storage: models (~700MB total, encoder int8 dominates at 622MB) are pushed to the app's private storage at runtime (matching Outspoke's own download-on-first-use pattern and SpectreBoard's existing DE/CE model-storage conventions — see homelab memory project-spectreboard's DE-vs-CE trap notes), NOT bundled into the APK. Comrade already has the exact files at ~/homelab/iree-stack/GoogleBeta/outspoke-parakeet-v3/ (encoder-model.int8.onnx, decoder_joint-model.int8.onnx, nemo128.onnx, vocab.txt, config.json).

Relationship to comrade Whisper dictation: not a replacement (yet) — the existing StreamDictation/comrade-Whisper toolbar-mic path stays as-is. This is a new, separate on-device engine, likely wired to a different toolbar key or toggle initially, so the working comrade path is never put at risk while this is being built out.

Personal adaptation pipeline — scope

Separate subsystem from the Parakeet entry above, recorded in the same file (same append-only rule applies).

2026-10-02 — Initial scope

Why: single-user keyboard on owned hardware, so none of the commercial constraints (federated learning, differential privacy, frozen base models) apply. Idea came from the user probing it with Gemini; reviewed against the real code before agreeing. Goal: autocorrect that learns from the user's own typing and corrections via an offline loop on comrade (GPU/CPU there, not the phone), with trained artifacts pushed back to the device.

Agreed decisions:

  1. Step 0/1 — persist training signal first. Before this, the only signal was Log.i("CorrectionOverride", …) in InputLogic.commitChosenWord (logcat ring buffer, effectively nothing retained) and the write-only CorrectionHistory (8 entries, never read). Persist to app storage:
    • manual picks from the strip (typed X → picked Y),
    • backspace reverts of an autocorrect (strongest negative label),
    • accepted autocorrects / typed-word commits (weak positive + the committed-text stream that becomes the personal corpus),
    • with n-gram context, and the per-candidate scorer features at commit time (dict / GRU / KenLM / spatial) so the reranker can be trained offline without re-deriving them.
    • Skip password / no-learning fields (reuse mIncognitoModeEnabled). Not for privacy — they're noise.
    • Pulled to comrade over Tailscale.
  2. Step 2 — nightly KenLM rebuild on comrade from base corpus (corpus_for_gru.txt) + personal committed text, quantized like spectre_q8.blm, gated by a held-out eval, pushed to the DE path (/data/user_de/0/com.termux.spectreboard/files/, chmod 644) with a reload mechanism (scorers currently load once per IME process).
  3. Step 3 — learned reranker weights. Pick/revert pairs are ranking decisions, so they train a small model over the four scorer features to replace the fixed lexicographic order in Suggest.rerankCombined() (dict band → GRU → KenLM → spatial). Not DPO on the GRU (too few pairs/day, overfits).
  4. Step 4 — spatial label hygiene. SpatialModelWorker already rebuilds per-key Gaussians from GESTURE_DATA; fix is excluding taps from reverted (mislabelled) words, not faster updates.
  5. Step 5 — occasional GRU continued training on personal + base text mix. Lowest priority.

Hard constraints:

  • Every rebuilt artifact must beat the current one on a held-out slice of the user's own typing before it's pushed. Learning from accepted autocorrects is a self-reinforcing loop; the eval gate + revert signal are what stop it drifting.
  • Personally-trained models never go to the public repo or the public HF repo marx161-cmd/spectreboard-models — n-gram/LM models memorise exact strings. Base models there stay as-is.

2026-10-02 (later) — Save everything, always; privacy filters removed

Changed: the initial entry said to skip password / no-learning fields "because they're noise". User overrode that: save everything, always — single-user device, user audits everything, privacy is irrelevant here.

Found while debugging why the spatial tier was dead (sp: null on every candidate): GESTURE_DATA had 0 rows since the 2026-07-25 data wipe because of three stacked inherited-from-HeliBoard privacy gates:

  1. "Discard by default" (gesture_data_background_gathering_manual_save=true) — taps dropped at field exit unless the toolbar save key was pressed.
  2. App allowlist with nothing in it (include-by-default unset → false) → every app forbidden.
  3. isSavingOk only kept words from a hash-matched official HeliBoard main dictionary (SpectreBoard's custom dict never matches), plus contacts / excluded-word / password / email / incognito / non-TYPE_CLASS_TEXT (terminal) filters, and redacted all non-target alternatives.

Decision: the discard option is removed (setting + UI row; the toolbar gathering key now saves instead of discarding); apps are included by default; all field/app/incognito/dictionary/word filters are removed; alternatives are kept unredacted (they're the reranker's negatives). AdaptationLog also logs in incognito/password fields now. What remains: the master background-gathering switch (on), SPACE_AWARE_GESTURE skip, shortcut-only skip.

Consequence carried into step 2+: passwords/OTPs will now be in the raw logs. They're not filtered at collection; if they should not surface as suggestions, filter at training time on comrade — never at collection.

2026-10-02 (evening) — KenLM v2 live; emoji deliberately not modelled

  • KenLM is now rebuilt on comrade by ~/kenlm/kenlm_rebuild.py from ChatGPT + Gemini + Claude exports + replayed personal typing, normalised the way the phone queries it (lowercase, punctuation split off, sentence per line). The old model was trained on raw cased text with punctuation attached, which KenLmScorer (lowercased context + candidates) could never match. spectre_v2_q8.blm pushed 15:56 via tmp+rename hot-reload (c04120bd).
  • The normaliser keeps only letters/digits/internal apostrophes, so emoji are not in the model. User decision: that's fine — emoji suggestions aren't wanted from KenLM. Don't "fix" this; the nightly rebuild may keep dropping them.

2026-10-17 — Dictation redesign: chunked whole-utterance Whisper on comrade; Parakeet dropped

Changed: the 2026-09-02 on-device Parakeet port is out. User verdict: it fails to do anything about half the time, in this port and in Outspoke itself. User dislikes Parakeet and explicitly does not want it fixed. Google voice typing and FUTO Voice Input were rejected as worse than Parakeet. The old comrade Whisper streaming path (StreamDictation.kt → :8770) was also rejected, for two reasons the user hit in real use:

  1. Garbage text. af_whisper cut the audio every ~3 s at VAD pauses and transcribed each piece with no context. That gave doubled, dropped and invented words at every join.
  2. Lost or deleted text. Blocks either never arrived, or arrived in a state where any edit deleted them (composing-region / revise handling). The user also speaks for longer than Whisper's 30 s window.

Decided design (agreed with user 2026-10-17):

  • Phone records Opus, the format Android records natively. Five minutes of speech is under 1 MB.
  • Chunked upload while speaking. The phone cuts at a pause near ~25 s, never mid-word, and uploads each piece immediately, so comrade transcribes while the user is still talking. When the user stops, only the last piece is left. The user's point: whole-file-at-the-end wastes idle time.
  • Context carried between pieces. Each piece is transcribed with the previous text as the Whisper prompt, limited to the last sentence or two. Passing everything risks error propagation and repetition loops.
  • Text is typed once, at the end, as plain committed text. It never goes into a composing span, and no revise or delete is ever sent. Pieces collect in the background. A small "N pieces done" indicator on the keyboard shows progress.
  • Nothing gets lost. Each recording gets a unique ID and each piece a sequence number. The phone keeps every piece until comrade acknowledges it, and retries are idempotent: a retry returns the cached result and never re-transcribes or duplicates text. If comrade is unreachable, the keyboard says so and keeps the recording for a later retry.
  • Transport: plain HTTP over Tailscale to comrade (100.108.8.60).
  • Model / hardware: whisper.cpp large-v3-turbo (~/src/whisper.cpp/models/ggml-large-v3-turbo.bin, sha256 matches HF) on the iGPU (Vulkan device 1, Radeon 760M), keeping the RX 9060 XT free for the shader backdrop. A whole-file test took 5.9 s for a 45 s clip, with no joins and no GPU reset. The same whole-file run on the 9060 XT crashed twice with Vulkan DeviceLost, and the kernel reset a compute ring each time. The suspected cause is a driver timeout under backdrop load, not proven. Don't use device 0 for this without asking.
  • Server binary: ffmpeg's whisper filter has no prompt option, so it can't carry context. Use whisper.cpp's own whisper-server, built at ~/src/whisper.cpp/build-vulkan with its prompt field. That it can target the iGPU is not yet verified.

Build order: (1) this entry, (2) verify whisper-server on the iGPU with turbo, (3) comrade chunk service, (4) SpectreBoard phone side behind the normal mic button. Each step is stoppable.

Open: what happens to the Parakeet code in spectre/parakeet/ and to the old StreamDictation / :8770 path. Not decided; leave them in place until the new path works.

2026-10-17 (later) — Transport changed: ssh pipe instead of HTTP

Changed: "plain HTTP over Tailscale" from the entry above is replaced by the user's suggestion: pipe each piece over ssh. SpectreBoard shares Termux's UID, so it can run Termux's own ssh with the existing key. Phone to comrade ssh was verified working 2026-10-17, at 0.9 s for a fresh connection.

  • One ssh connection per recording, kept open with ControlMaster, so only the first piece pays the setup cost.
  • Per piece: the audio goes to stdin of a small comrade script, and the text comes back on stdout.
  • Comrade keeps one text file per recording ID. It supplies the context prompt and makes retries idempotent, because a repeated sequence number returns the saved text.
  • whisper-server (turbo, iGPU) stays resident on comrade, bound to 127.0.0.1 only. ssh is the only way in, so no new network listener.
  • Verified 2026-10-17: whisper-server on -dev 1 with turbo did a 45 s clip in 4.6 s and accepted the prompt field. Its plain-text reply wraps lines mid-word, so the script must use the JSON reply and join the segments.

Everything else in the entry above stands: Opus, pieces cut at pauses near 25 s, context limited to the last sentence or two, text typed once at the end as plain committed text, nothing lost.

2026-10-17 (night) — Parakeet removed entirely; its plumbing is reused

Decided (user): the Parakeet code goes completely. That's ParakeetEngine, InferenceRepository, the live-rewrite TextInjector / TranscriptAligner, the correction/ and model/ packages, and acoustic suggestions. It costs space for nothing and SpectreBoard is big enough already. This answers the "Open" question two entries up.

Kept and rewired to the comrade Whisper path: the mic toggle and UI states (KeyboardViewModel / KeyboardUiState), the foreground InferenceService with its persistent notification and haptics, and AudioCaptureManager plus the VAD filters. The VAD supplies the pause points for cutting pieces near 25 s.

Constraint found: the kept files import InferenceRepository, TextInjector and others directly, so the deletion can't be done on its own without breaking the build. Removal and rewiring land together in the phone-side step, with a new small component replacing InferenceRepository behind the same transcribe(audio) -> Flow<TranscriptResult> call. It emits a single final result and no partials.

Done now:

  • Safety checkpoint: annotated tag pre-parakeet-removal on 7ad7d4d4. Restore with git checkout pre-parakeet-removal -- <path>.
  • Phone model folder /data/user_de/0/com.termux.spectreboard/files/models/parakeet-v3 (640 MB) deleted. An identical-size copy stays on comrade at ~/homelab/iree-stack/GoogleBeta/outspoke-parakeet-v3/. The phone was at 97% full on /data (7.2 GB free) afterwards.
  • Until the phone-side step ships, the installed SpectreBoard's Parakeet mic path has no model and won't work. That's accepted, since it was already unreliable.

2026-10-17 (late night) — Phone side built and installed (untested by voice)

Implemented as spectre/PieceDictation.kt. The controls are taken over from StreamDictation.kt: one buzz on start, two on stop, the animated progress notification, and one isRecording flag that drives the toolbar light. The light used to read StreamDictation.isRecording while the button toggled Parakeet, so it never lit during Parakeet dictation.

  • Pieces are cut at a 300 ms pause once ~20 s long (adaptive noise floor), or at the quietest point of the last 5 s if no pause comes by 30 s. A last piece under 0.45 s, or all quiet, is dropped.
  • Each piece is encoded as Opus in Ogg via MediaCodec/MediaMuxer (WAV fallback), saved in cacheDir/dictation/<id>/, and piped through Termux's ssh (ControlMaster) to dictate-piece. Retries back off over ~90 s.
  • Text is typed once at the end with commitText(text + " ").
  • Parakeet deleted: the spectre/parakeet/ folder, AcousticSuggestions, the Suggest.kt acoustic block, the InputLogic guard, the InferenceService manifest entry, silero_vad_v4.onnx, the DataStore dependency and the Parakeet strings.

Differences from the entries above, flagged to the user:

  • The "N pieces done" progress shows in the notification text, not on the keyboard itself.
  • New behaviour I added, needs the user's OK: if the field changed between the stop tap and the text arriving, or typing fails, the text goes to the clipboard with a toast, not into the field. A recording that couldn't reach comrade is kept, and on the next dictation it's sent and its text is recovered to the clipboard.
  • StreamDictation.kt and SpectreRecognitionService (old :8770 path, the system voice-input service, not currently selected) are left in place, unchanged.

2026-10-17 (23:40) — User confirmed it works; notification retuned

  • User verified by voice: long rants dictate fine, and the text arrives after the stop tap.
  • Stop is now unmistakable (user ask): after the stop tap the notification switches to grey "Stopped — transcribing" with an indeterminate bar until the text arrives. It used to keep the red advancing bar, so it wasn't clear whether stop had registered.
  • 30 s countdown (user ask): while recording, the bar drains from full to empty over 30 s and repeats, and the text shows the seconds left ("23s · 2 pieces done"). It's for judging Whisper's 30 s window by eye. The cycle is a plain timer and does not reset when a piece is cut.
  • Still open: the user's OK on the clipboard fallback (previous entry).

2026-10-17 (23:50) — Clipboard fallback approved

User OK on the clipboard fallback (field changed, typing failed, or recovered recording -> clipboard + toast). Dictation is verified working, so the work is pushed to GitHub.

2026-10-18 — Megaphone read-aloud key (user-verified, incl. Termux)

The unused Split toolbar slot is repurposed as a megaphone (KeyCode.READ_ALOUD -10064, icon sym_keyboard_megaphone). The user's design: one tap injects Android's KEYCODE_COPY via root input keyevent, waits 250 ms, then sends the clipboard to UDTTSR's ReadAloudService. That streams it through textclean.py and Kokoro on comrade over ssh. A tap while reading stops it. KEYCODE_COPY was chosen over Ctrl+C because Ctrl+C would SIGINT in Termux. The user confirmed it works in Termux too. Split keyboard itself remains in settings.

2026-10-23 — Keyboard shader surface (B1 in android-mods/SHADER_SPOTS.md)

What: a live shader over the keyboard, drawn inside our own app (no LSPosed hook, no SystemUI/system_server risk). The whole keyboard content is post-processed with RenderEffect.createRuntimeShaderEffect(shader, "iContent") — the real keys and LETTERS are warped as rendered pixels — and per-key press data is fed in as uniforms so the damage is anchored to the key just hit.

User decisions (2026-10-23):

  • Warp the letters, don't protect them. Explicitly the opposite of qs-shader-live's text-aware dial-back: a keyboard layout is muscle memory, "you could strip 80% of every letter and it would still be usable". No text boxes, no legibility guard.
  • Per-key drawing is not enough on its own. RuntimeShader is a Shader, so a per-key paint can fill/tint a key (and per-key press rings are cheap), but it cannot displace glyph geometry. Letter-shape destruction therefore happens in the global iContent effect; keys may still get their own press visuals later.
  • Music reactivity: beat clock if there is one, otherwise the JDSP pipes. BeatSource (cached map > desktop go) from qs-shader-live is reused in behaviour; bands.bin v3 is added underneath (SpectreBoard is UID 1000 and CAN read it — unlike SystemUI), so u_onset/u_flux/u_bass drive hits when no map matches, and the bands feed the continuous floor in every case.

Architecture:

  • New package com.termux.spectreboard.shader:
    • BandsFeed — port of ShaderEditor's opengl/JdspBandsFeed.java (mmap, seq, v3 ring, 11 features, per-band group-delay alignment). Owner is audioserver, file is 0644; verified readable from the system UID 2026-10-23.
    • BeatSource — port of qs-shader-live/BeatSource.java (media session match against /data/local/tmp/qs_shader/index.json + maps/<id>.bin, else go.txt, minus beat_offset_ms.txt). MEDIA_CONTENT_CONTROL is granted to the shared UID (verified). Writes its own status file, never qs_shader's.
    • ShaderFile — port of the qs-shader-live loader: mtime hot reload (≤1 s), //#include, uniform-declared-only setters, compile error → effect removed, never a crash.
    • KeyboardShader — Choreographer loop, uniform upload, key-hit ring, attach/detach.
  • Shader files live in ~/shaders/keyboard/keyboard.agsl (Termux home, same UID), with the app files dir as fallback. Same edit-and-watch loop as ShaderEditor.
  • Attach point: KeyboardSwitcher already resolves R.id.main_keyboard_frame after inflating input_view; the effect is attached to R.id.keyboard_view_wrapper first (keys only, suggestion strip stays stock). One switch in KeyboardShader will include the frame.
  • Loop gating: animate only while bands are live (music) or ≤1.2 s after a key hit; idle renders one static frame and stops. 60 fps cap. The key preview bubbles live in android.R.id.content (DrawingPreviewPlacerView) and are intentionally NOT warped for v1.

Uniforms v1: iResolution iTime, shader iContent, float4 iBands[27] / float4 iBandsDb[27] (108 flattened), the 11 features (u_rms…u_onset) plus u_onset_smooth u_onset_accum u_bass_accum, iBeatOn iBeatPhase iBeatCount iBpm iBarPhase iBeatsPerBar, iLive, and float4 iHit[4] + float iHitAge[4] + iHitN (key rects in effect-pixel space). bands_hist is deliberately not exposed yet.

Cost / safety: full keyboard raster per animated frame (text included) is the real cost, not the shader math — measure with dumpsys gfxinfo before making it free-running. A failed compile or a missing file must leave stock drawing untouched (falls back to setRenderEffect(null)).

Deploy: ./gradlew assembleRelease only; versionCode bumped past 4002 (ROM copy); blazer-sysapp-update install com.termux.spectreboard <apk>. Installing any family app can kill sshd/npud collateral (shared UID 1000) — batch it.

Open: suggestion strip include (default keys-only), idle shimmer (default off), per-key paint visuals, bands_hist, warping the key-preview bubbles.

2026-10-23 (tuning round, live with the user)

User feedback drove three passes, all verified by screenshot on blazer:

  1. "constant motion is way too busy" — the first version had ambient tearing driven by iTime and a 1.2 s full-effect tail. Now the shader is event-driven: drive is 0 at rest, the loop stops when the burst dies, and nothing moves if nothing happens.
  2. "the shockwave is basically instant … all letters shift" — the damage was gated by a scalar hitEnv, which is 1 everywhere on the press frame. Now the displacement is gated by a per-pixel front (wave), and the front is capped to a bubble around the pressed key. The scar was full-width; it is now a short ragged slash (LOCAL_R ≈ 0.30 × keyboard height, scar segment ±0.55 R). Confirmed: pressing space glitches only the space bar area.
  3. Cyberpunk look, still at rest — added neon edge glow traced from the real glyph/key edges plus a static teal→magenta grid/rails. Gotcha found live: the keys wrapper has a transparent background (the real background lives on main_keyboard_frame), so the first glow implementation (neon * col.a) was invisible in the gaps between keys. Fixed by adding glow in premultiplied space with its own alpha (col = half4(col.rgb + glow, col.a + glowA)).
  4. Locality verified with a debug shader that painted iHit[] rects — the rects land on the pressed keys, so the uniform plumbing is correct.

User verdict: "yes thats better 🔥". Still open: the music path (live=true / beatOn=true) has not been exercised by ear yet; the effect remains keys-only (suggestion strip stock).

2026-10-23 (later) — neon linework removed; black backdrop + typing glitch only

The static neon edge glow / grid / side rails were built, then removed at the user's request: "i liked the black backdrop the glitches when typing are perfect thats enough". The shader now draws NOTHING at rest (stock black keyboard) and only the local per-key bubble + scar when a key is pressed. LOCAL_R, WAVE_SPEED, the scar and the JDSP/beat inputs are unchanged. This is the look to protect when tuning further.

2026-10-27 — public test build (public build type) for outside testers

Trigger: woj-tek asked on HeliBoard discussion #2912 for SpectreBoard test builds. User OK'd a separate public build; personal models may be public, sensitive data may not.

Decided:

  • New build type public (initWith release), package com.termux.spectreboard.test, so it installs next to the system keyboard. assembleRelease / the Pixel's own keyboard unchanged.
  • BuildConfig.PUBLIC_BUILD: no keystroke persistence (WordData.save + AdaptationLog.append return early); homelab-only keys show a toast instead of acting (dictation, read-aloud, AMD mode/control, Cybersyn, i3 numpad, exec local/remote, number-row macros).
  • Manifest overlay: no sharedUserId=android.uid.system, no RUN_COMMAND / INTERNET / RECORD_AUDIO, no SpectreRecognitionService.
  • APK excludes libLiteRtDispatch_GoogleTensor.so + libwhisper_g5_worker.so (unused; not redistributable). Own signing key (not the Termux platform key).
  • gru_vocab.txt: sensitive lines (IPs, tailnet, Wi-Fi MAC iface, user@host prompts) replaced IN PLACE with <redacted_N> — line index == token ID, so no deletions.

2026-10-27 (later) — public build ships the KenLM + GRU tiers

User: leaving the scorer models out "is half of SpectreBoard" — bundle them. (Supersedes the earlier "personal models NEVER to public" line: user is fine publishing personal models, just not sensitive data.)

  • KenLM: NOT the nightly model (its vocab held token/password-looking strings, the Wi-Fi MAC iface, tailnet). adaptation/public_kenlm.py trains spectre_public_q8.blm from the newest nightly train.txt: words need count >= 3, mixed alnum >= 6 / digit runs / hex / homelab ids dropped, numeric runs (IPs, MACs) cut, sentences split at every drop. Same lmplz/-q 8 trie.
  • GRU: the live gru_cifg_int8.onnx unchanged (vocab already redacted; 2 more identifiers, a serial-like string and a Windows product ID, redacted → <redacted_63/64>).
  • Both ride in src/public/assets (gitignored, APK only) and are copied to filesDir on first use (PublicBuild.ensureBundledModel).

2026-10-27 (later) — neural-only swipe attempted, dropped; disclaimer instead

Tried letting the CleverKeys ONNX engine swipe alone when Google's glide lib is missing. Findings: (1) the open libjni_latinime segfaults on batch input (no gesture decoder); (2) the neural engine has never worked, in the personal build too — every predict() threw (trajectory_features length 150 vs model's fixed 250; after fixing that, still failing on another input), so gesture_ab_log has neural_top: "" on all 5,594 rows; Google did all swiping. User decision: stop, revert the attempt, ship a README disclaimer (swipe needs the user-loaded gesture library, as in HeliBoard). Neural engine repair = open item, not scheduled. Also seen: PhoneticExpander.doubleMetaphone throws StringIndexOutOfBounds at load (open).

2026-10-27 (later) — neural swipe engine: student of Google, trained on the user's swipes

Root cause of the dead engine found offline (comrade, real ONNX models): encoder needs 250 points (code pads 150); decoder needs target_tokens always padded to 20 (max_word_len) and reading the logits at the current position — the beam search fed growing lengths 1,2,3… With both fixed, encoder + decoder run (single and batched beams).

User's goal (verbatim intent): NOT "improve the Google library" (impossible). Google is the teacher: the neural model learns swipe trajectory → correct word from the user's own data, i.e. GESTURE_DATA (x/y/millis path + targetWord = the word that actually ended up committed: Google's pick, or the user's correction when Google was wrong). Train on comrade, repeatedly as data grows.

  • Shadow mode on the phone: engine runs per swipe, its guess is logged vs the final word, never shown. The personal keyboard must look and type exactly as today ("works perfectly").
  • Eval on held-out swipes each round: neural top-1 vs Google top-1 against targetWord.
  • Only when the neural model is measurably more accurate does it get a say in suggestions — and that switch is the user's decision, not automatic.
  • Data: user pushes heliboard.db to comrade when home (no adb pulls over mobile).

2026-10-27 (later) — CORRECTION + steps 1/2 built (shadow mode, training pipeline)

Correction to the entry above: the decoder-padding "second bug" was wrong — BeamSearchDecoder.kt already pads target_tokens to 20 (my Python test hadn't). The on-device error after the 250 fix was never read (rolled out of logcat); shadow mode logs it if it recurs.

Step 1 (keyboard, invisible): MAX_SEQ_LENGTH 250; neural candidate injection + gesture A/B log REMOVED from Suggest (never worked); NeuralShadow runs the engine on each recorded swipe on one background-priority thread after Google's list is out (skips if busy) and stores its top-3 as neural in that swipe's GESTURE_DATA record ([] = engine failed). Models/vocab pushed to filesDir/neural/ override the APK assets. Public build: shadow off. Step 2 (comrade, adaptation/swipe/): no float checkpoint of the shipped model exists, so onnx_import.py rebuilds it from the int8 ONNX (decoder matches 100% top-1; encoder ~8% rel. diff = the shipped int8 activation rounding, input step 0.078 ≈ 13 position levels). features.py = exact port of TrajectoryFeatures.kt; beam.py = port of BeamSearchDecoder.kt; train.py fine-tunes on GESTURE_DATA (label targetWord, newest 10% held out) and reports top-1 google / base / tuned; trie vocab = en.txt + the user's own a-z swiped words; export.py writes float32 ONNX with the phone's exact I/O contract (no int8). Base weights: ~/kenlm/swipe/. Verified on synthetic data only (synth.py); real numbers need the pushed heliboard.db. Limitation: model alphabet is a-z, so umlaut words can't be learned by this model.

2026-11-09 — Dictation lost speech from ~21 s to ~31 s (fix built + installed, awaiting voice test)

User: "if I do close to 30 s or longer it loses half the text". Evidence on comrade (~/.cache/dictate/): every recording longer than ~21 s since 10-17 was piece 0 (~20 s) + piece 1 of exactly ~0.98 s, and piece 0 reached comrade ~11-12 s after its cut. Cause: the capture thread Opus-encoded each piece itself, which took ~10 s (encoder loop waited 10 ms per input buffer), while the AudioRecord buffer held only 1 s. Speech in that window was dropped; piece 1 was just the last buffer read after the stop tap. Fix (PieceDictation.kt): capture thread only queues raw PCM; the upload thread encodes then sends; mic buffer 4 s; encoder drains output without waiting until all input is queued. Installed 19:09 via blazer-sysapp-update (temporary ledger tier). Not committed until the user confirms a 40 s+ dictation.

2026-11-09 (later) — Long dictation fix CONFIRMED; notification is on/off only, ring shows the state

  • Fix above confirmed by the user's own 58 s dictation (3 full pieces: 22.6 + 20.3 + 15.5 s, nothing lost).
  • User: drop the 30 s countdown ("I'm still trying to time it so I'm under 30"); dictation is continuous now, so the notification is just on/off. Red while dictating, grey while working, green flash when the text comes back. Drawn by shaderfx's camera ring (shaderfx/scope.md 2026-11-09 dictation entry).
  • Built: no progress bar, no countdown tick, no "N pieces done" text. Every dictation notification carries extra dev.marx.dictation.state = rec | busy | done | fail. done/fail stay posted 900 ms (so the ring can flash), then clear; "no speech heard" clears straight away. Installed 19:17, awaiting the user's look. Side effect: without a progress value the crDroid island no longer promotes the dictation notification.