diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md index 31c5149c2d407..4c67256a261b7 100644 --- a/internal/interaction-harness/README.md +++ b/internal/interaction-harness/README.md @@ -1,20 +1,23 @@ # @astryxdesign/interaction-harness Playwright instruments for touch interaction and animation proofs: real touch -input, curve sampling and storyboards. **Internal and unpublished.** Specs live -next to their component as `*.interaction.browser.spec.ts` and run under +input, frame evidence, curve sampling and storyboards. **Internal and +unpublished.** Specs live next to their component as +`*.interaction.browser.spec.ts` and run under [`playwright.interaction.config.ts`](../../playwright.interaction.config.ts) on a phone-sized touch screen (402 × 714 CSS px page, 402 × 874 screen, 3 device pixels per CSS px, touch, mobile viewport rules), in Chromium and WebKit. The harness proves its own instruments in -[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts), -on fixture pages written inline. +[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts). +The busy-main-thread proof has a negative control beside its positive check +(a rAF animation that freezes under a blocked main thread); other self-tests +check an instrument against a known answer. ## Running ```sh -pnpm storybook:build # component specs read stories from the built Storybook +pnpm storybook:build # the specs read stories from the built Storybook pnpm exec playwright install chromium webkit pnpm test:interaction --project chromium # everything, Chromium pnpm test:interaction # one spec, both engines @@ -42,6 +45,8 @@ committed: a pull request links or attaches it. | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. | | `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. | +| `frames.ts` | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `frameStats`. | +| `busy.ts` | `blockMainThread(page, ms)` and `busyWindowProof(tracked, window)`: frames presented, frames that moved, longest stall, a verdict; `presented` adds the compositor's own frames. | | `curves.ts` | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`. | | `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. | | `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event): PNG contact sheet + JSON trace. | @@ -97,6 +102,72 @@ set): a short entrance no longer races the first look on a cold page. When the action may be the page's first contact with the harness, take `watchAnimations(locator)` before it and read `watch.started()` after. +**Busy main thread.** Mark the element, start the screencast, start the +animation, wait until it runs (`waitForAnimation`: an animation cannot start +while the thread is held), block the main thread for 1 s (`spec:AST-094` +FR5's verification row), stop. Always +start the gesture or animation first and block mid-flight, never before: +in headless Chromium a touch that starts while the main thread is busy does +not scroll or start a gesture even with only passive listeners (the input +waits for the main thread). Tracking a finger under a busy main thread is +therefore proven on the iOS simulator, not here. The +screencast is captured off the main thread, so a compositor animation shows a +new position every frame of the block, and a main-thread animation shows none +(nothing changed, so no frame is sent at all: `busyWindowProof` counts the +whole window as one stall). `settledWithin` (2 px) stops judging at the easing +tail: the screencast rounds to whole pixels, so a tail that still moves shows +plateaus within a pixel or two of rest. A judged part shorter than four of the +screencast's intervals fails: the animation under test was not moving during +the block. With +`during`, `blockMainThread` starts the loop from a task queued in the frame +that saw the animation, after that frame's rendering update: WebKit hands a +newly started animation to its compositor there, and a loop inside the rAF +callback would freeze even a compositor animation. Call it before the release +that starts a settle and await it after, so the page is already waiting when +the settle begins. An animation counts once it is no longer pending and has +run at least one frame: a `minProgressMs` under 16 is raised to 16, since one +created in the frame the page looks is not on the compositor yet. The `verdict` judges the +frames the screencast delivered, whatever its rate (a loaded machine's can +deliver one frame in five or six), and tolerates nothing a frozen animation +does: at least one frame in the judged part, no stall longer than three of the +screencast's intervals (two dropped frames), and at least 90% of the frames it +delivered in the judged part presenting a new position (one miss is always +allowed; a frame it did not deliver is a gap, for the stall check, unless +it delivered fewer than four frames there: then the judged part's span counts +every frame, so one late frame is not "1 of 1 moved"). Pass +`maxMedianFrameMs` (20 at 60 Hz) only in a proof that claims display rate. +`verdict.failures` names each miss with its numbers, and `summary` carries the +raw numbers for the report. Frames are judged in presentation order: a loaded +machine can deliver them out of order. + +A just-started animation can reach the screen after the block began (50–100 ms +later in WebKit on a loaded machine). When the cast saw the element at rest +before the block (wait for `cast.frameCount()` to pass 0 before starting the +animation), judging starts at its first presented change: the judged part still +ends with the block and still needs four intervals, and `onScreenFromMs` and the +summary report when it came. An element already moving is judged from the +block's start. WebKit's casts are 2 px quantized: the frames come at the +viewport's size, but the page in them is captured at half resolution, so +positions move in 2 CSS px steps (Chromium's in 1). With under two steps of +motion per frame, a frame delivered a few ms early shows no step and reads as +not moved. The proof reports `stepPx` and `pxPerFrame` and notes motion under +two steps per frame, which never fails it. Move a proof's element at least two +steps a frame: the self-tests move 560 px in 1,600 ms, about 4.8 px a frame. + +On a loaded machine Chromium's screencast can skip frames the compositor +still presents. Pass when the compositor presented each frame (epoch ms on the +page clock, from a trace of `Display::FrameDisplayed`) as `presented`. A gap between two moved +frames with nothing delivered inside it is then measured between the +compositor's own frames: the frames it presented there count as presented +(`recorderSkipped`, in the summary), and a gap it did not present through is +a stall as before. A gap that opens or closes the judged part, or holds a +delivered frame that did not move, is judged from the screencast, so a +frozen element fails whatever else the compositor draws (a rAF animation's +block presents no compositor frame at all). With the record, the 90% counts the +compositor's frames from the first to the last moved frame; the gaps before and +after are the stall check's. Without a record (WebKit has no such trace), the +screencast is judged alone. + **Storyboards.** Deterministic: frames come from seeking held animations or from stepping scripted input, never from wall-clock capture. The JSON trace carries each frame's measurements and the spec asserts on it; the sheet is @@ -104,16 +175,18 @@ evidence. A component that finishes an animation from a timer (a transitionend backstop, a cleanup render) would do so behind a held storyboard's back: `installTimerGate` before the animation starts and `freezeTimers` while photographing. The page can still cancel a held callback, a held frame callback -gets the time it actually runs, and the harness's own frame waits are not held. +gets the time it actually runs, and the harness's own waits (its frame waits, +`blockMainThread` with `during`) are not held. An animation that repeats forever does not set a storyboard's default length; with nothing else to film, pass `times`. A touch filmstrip keeps the scripted event `timeStamp`s (Chromium), but a handler that reads `performance.now()` or `Date.now()` sees one event per photograph, about 100 ms apart, so a clock-timed release flings slower when filmed than when played: a storyboard -cannot show such a gesture; record it in real time instead. Chromium can return -an empty copy of the page's surface right after it changes ("Unable to capture -screenshot"); a storyboard takes that photograph again two frames later, up to -three attempts in all, and lists each retake under `recaptures` in the trace. +cannot show such a gesture; record it in real time instead, with +`startScreencast`. Chromium can return an empty copy of +the page's surface right after it changes ("Unable to capture screenshot"); a +storyboard takes that photograph again two frames later, up to three attempts +in all, and lists each retake under `recaptures` in the trace. ## Engine limits (measured with Playwright 1.61) @@ -126,11 +199,13 @@ three attempts in all, and lists each retake under `recaptures` in the trace. | Lifting one finger while others stay down | yes | pointer events right, but that `touchend` lists no remaining `touches`: skip with `PARTIAL_LIFT_SKIP` | | Click synthesised from a touch | yes | only from `driver.tap` (WebKit's tap gesture), not from a raw touch sequence | | Curve sampling, storyboards, input log | yes | yes | +| Screencast frame evidence (busy main thread) | yes | yes (WebKit's own screencast; frames keep coming while the main thread is blocked) | What WebKit cannot show here — native scroll hand-off, flings, a finger -lifting while another stays down, and anything iOS-only — needs Mobile Safari -itself, on a device or the iOS simulator. Skip such a proof on the driver's -flag, citing the reason: `test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`. +lifting while another stays down, finger tracking under a busy main thread, and +anything iOS-only — needs Mobile Safari itself, on a device or the iOS +simulator. Skip such a proof on the driver's flag, citing the reason: +`test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`. The WebKit input command is reached through Playwright's in-process server object, so it needs a locally launched browser (the default), not `connect()`. diff --git a/internal/interaction-harness/package.json b/internal/interaction-harness/package.json index 40cccb26ca206..942eccb9ed8d5 100644 --- a/internal/interaction-harness/package.json +++ b/internal/interaction-harness/package.json @@ -3,7 +3,7 @@ "version": "0.0.0", "private": true, "type": "module", - "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, curve sampling, storyboards). Never published.", + "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, frame evidence, curve sampling, storyboards). Never published.", "main": "./src/index.ts", "exports": { ".": "./src/index.ts", diff --git a/internal/interaction-harness/src/busy.test.ts b/internal/interaction-harness/src/busy.test.ts new file mode 100644 index 0000000000000..372860125fe6a --- /dev/null +++ b/internal/interaction-harness/src/busy.test.ts @@ -0,0 +1,503 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file busy.test.ts + * @input Synthetic tracked frames (no browser) + * @output The busy-main-thread verdict's limits: dropped frames a loaded + * machine produces pass, a frozen animation fails every criterion, a window + * with nothing moving to judge fails, and out-of-order frames are judged in + * presentation order. + * @position Unit test of ./busy and ./frames' ordering; the browser proof + * with a real compositor animation and a rAF control is in + * harness.interaction.browser.spec.ts. + */ + +import {describe, expect, it} from 'vitest'; +import {busyWindowProof} from './busy'; +import {inPresentationOrder, type TrackedFrame} from './frames'; + +const FRAME = 1000 / 60; +const BLOCK = {from: 1000, to: 1700}; + +/** A marker moving 3 px per frame from t = 800 to 1900, minus `dropped` frame indices. */ +function moving(dropped: readonly number[] = []): TrackedFrame[] { + const frames: TrackedFrame[] = []; + for (let i = 0; 800 + i * FRAME <= 1900; i += 1) { + if (dropped.includes(i)) { + continue; + } + frames.push({ + t: 800 + i * FRAME, + box: {x: 0, y: 3 * i, width: 10, height: 10}, + } as TrackedFrame); + } + return frames; +} + +/** Index of the first frame at or after `t`. */ +const at = (t: number) => Math.ceil((t - 800) / FRAME); + +/** `moving()` with each frame in `repeats` presenting the previous frame's position. */ +function repeating(repeats: readonly number[]): TrackedFrame[] { + return moving().map((f, i) => + repeats.includes(i) + ? ({...f, box: {...f.box!, y: 3 * (i - 1)}} as TrackedFrame) + : f, + ); +} + +/** A marker moving 3 px per frame from t = 800 to 1900, frames `intervals` apart in turn. */ +function cadenced(intervals: readonly number[]): TrackedFrame[] { + const frames: TrackedFrame[] = []; + for (let i = 0, t = 800; t <= 1900; i += 1) { + frames.push({ + t, + box: {x: 0, y: 3 * i, width: 10, height: 10}, + } as TrackedFrame); + t += intervals[i % intervals.length]; + } + return frames; +} + +describe('busyWindowProof verdict', () => { + it('passes a compositor animation with every frame', () => { + const proof = busyWindowProof(moving(), BLOCK); + expect(proof.verdict.failures).toEqual([]); + expect(proof.longestStallFrames).toBe(1); + }); + + it('passes one and two consecutive dropped frames (3 intervals)', () => { + const k = at(1300); + expect(busyWindowProof(moving([k]), BLOCK).verdict.failures).toEqual([]); + const two = busyWindowProof(moving([k, k + 1]), BLOCK); + expect(two.longestStallFrames).toBe(3); + expect(two.verdict.failures).toEqual([]); + }); + + it('fails three consecutive dropped frames', () => { + const k = at(1300); + const proof = busyWindowProof(moving([k, k + 1, k + 2]), BLOCK); + expect(proof.longestStallFrames).toBe(4); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('fails when more than 10% of the delivered frames do not move', () => { + const k = at(1050); + // 39 frames delivered: up to 3 may show no new position (10% = 3.9). + const scattered = [0, 12, 24].map(d => k + d); + expect( + busyWindowProof(repeating(scattered), BLOCK).verdict.failures, + ).toEqual([]); + const many = [0, 6, 12, 18, 24].map(d => k + d); + expect( + busyWindowProof(repeating(many), BLOCK).verdict.failures.join('\n'), + ).toMatch(/expected frames moved/); + }); + + it('judges a frame the screencast did not deliver as a gap for the stall check, not a miss', () => { + // The same five frames, not delivered: two-interval gaps, and every + // delivered frame moved. + const k = at(1050); + const many = [0, 6, 12, 18, 24].map(d => k + d); + const proof = busyWindowProof(moving(many), BLOCK); + expect(proof.verdict.failures).toEqual([]); + expect(proof.deliveredFrames).toBe(proof.moved); + }); + + it('judges an irregular cast on the frames it delivered', () => { + // A cast cycling 11.7, 11.7 and 17 ms: its median interval would expect + // 56 frames where it delivered 49, every one moved. + const proof = busyWindowProof(cadenced([11.7, 11.7, 17]), BLOCK); + expect(proof.verdict.failures).toEqual([]); + expect(proof.deliveredFrames).toBe(49); + expect(proof.moved).toBe(49); + // Still a stall where it delivers nothing for four frames, and still a + // miss where five delivered frames show no new position. + const gap = cadenced([11.7, 11.7, 17]).filter( + f => f.t < 1300 || f.t > 1360, + ); + expect(busyWindowProof(gap, BLOCK).verdict.failures.join('\n')).toMatch( + /longest stall/, + ); + const unmoved = cadenced([11.7, 11.7, 17]).map((f, i, all) => + [20, 26, 32, 38, 44].includes(i) + ? ({...f, box: {...all[i - 1].box!}} as TrackedFrame) + : f, + ); + expect(busyWindowProof(unmoved, BLOCK).verdict.failures.join('\n')).toMatch( + /expected frames moved/, + ); + }); + + it('judges a window that delivered fewer than four frames against its span', () => { + // A rAF animation frozen by the block, in WebKit: one frame from before + // the block arrives 45 ms into it, then nothing until after it. + const late = moving().filter(f => f.t <= 1000 || f.t >= 1750); + late.push({ + t: 1045, + box: {x: 0, y: 3 * at(1001), width: 10, height: 10}, + } as TrackedFrame); + const proof = busyWindowProof(inPresentationOrder(late), BLOCK); + expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/); + }); + + it('lets a short window miss a frame, not three', () => { + // 150 ms judged: nine frames delivered, one miss allowed (10% would be 0.9). + const short = {from: 1000, to: 1190}; + const k = at(1040); + expect(busyWindowProof(repeating([k]), short).verdict.failures).toEqual([]); + expect( + busyWindowProof( + repeating([k, k + 3, k + 6]), + short, + ).verdict.failures.join('\n'), + ).toMatch(/expected frames moved/); + }); + + it('fails a frozen animation on every criterion', () => { + // Frames before and after the block only: nothing changed during it. + const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to); + const proof = busyWindowProof(frozen, BLOCK); + expect(proof.changed).toBe(0); + expect(proof.verdict.pass).toBe(false); + expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/); + }); + + it('fails a window in which the animation under test was not moving', () => { + // Settled before the block: the judgment ends where it starts. + const settled = moving().map(f => + f.t < 950 + ? f + : ({...f, box: {...f.box!, y: 3 * at(950)}} as TrackedFrame), + ); + const before = busyWindowProof(settled, BLOCK, {settledWithin: 2}); + expect(before.verdict.pass).toBe(false); + expect(before.verdict.failures.join('\n')).toMatch( + /judged = -?\d+ frames < 4: the animation under test was not moving/, + ); + // Settled 40 ms into the block: two frames, too few to judge. + const early = moving().map(f => + f.t < 1040 + ? f + : ({...f, box: {...f.box!, y: 3 * at(1040)}} as TrackedFrame), + ); + expect( + busyWindowProof(early, BLOCK, {settledWithin: 2}).verdict.failures.join( + '\n', + ), + ).toMatch(/frames < 4: the animation under test was not moving/); + }); + + it('judges the frames a slower screencast delivers by default, and display rate only when asked', () => { + // One frame in six, as a loaded machine's screencast delivers: every + // delivered frame moved, so the default verdict passes. + const sixth = moving().filter((_, i) => i % 6 === 0); + expect(busyWindowProof(sixth, BLOCK).verdict.failures).toEqual([]); + // Half rate fails only a proof that claims display rate. + const odd = moving().filter((_, i) => i % 2 === 1); + expect(busyWindowProof(odd, BLOCK).verdict.pass).toBe(true); + expect( + busyWindowProof(odd, BLOCK, {maxMedianFrameMs: 20}).verdict.failures.join( + '\n', + ), + ).toMatch(/median frame interval/); + }); +}); + +describe('busyWindowProof settledWithin', () => { + it('ends the judgment at a rounded easing tail, which would read as a stall', () => { + // An ease-out to y = 24 as the screencast reports it: whole pixels, so the + // tail repeats values while it still moves. + const ys = [ + 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 26, 26, 25, 25, 24, 24, + 24, 24, + ]; + const frames = ys.map( + (y, i) => + ({ + t: 1000 + i * FRAME, + box: {x: 0, y, width: 10, height: 10}, + }) as TrackedFrame, + ); + const window = {from: 1000, to: 1000 + ys.length * FRAME}; + const loose = busyWindowProof(frames, window, { + settledWithin: 2, + marginMs: 0, + }); + expect(loose.verdict.failures).toEqual([]); + const strict = busyWindowProof(frames, window, { + settledWithin: 1, + marginMs: 0, + }); + expect(strict.longestStallFrames).toBeGreaterThanOrEqual(3); + }); + + it('takes the end value from settledAt over a stale last frame', () => { + // The tail reaches y = 24, then the last frame shows a stale 100 (WebKit + // presents the last committed position once an accelerated animation + // ends while the main thread is still blocked). + const ys = [ + 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 25, 24, 24, 24, 100, + ]; + const frames = ys.map( + (y, i) => + ({ + t: 1000 + i * FRAME, + box: {x: 0, y, width: 10, height: 10}, + }) as TrackedFrame, + ); + const window = {from: 1000, to: 1000 + ys.length * FRAME}; + const stale = busyWindowProof(frames, window, { + settledWithin: 2, + marginMs: 0, + }); + expect(stale.verdict.pass).toBe(false); + const fromPage = busyWindowProof(frames, window, { + settledWithin: 2, + settledAt: 24, + marginMs: 0, + }); + expect(fromPage.verdict.failures).toEqual([]); + expect(fromPage.judged.to).toBe(frames[11].t); + }); +}); + +/** Seen at rest at t = 900, then on screen from `start`, moving 3 px a frame. */ +function startingAt(start: number): TrackedFrame[] { + const frames = [ + {t: 900, box: {x: 0, y: 0, width: 10, height: 10}} as TrackedFrame, + ]; + for (let i = 1; start + (i - 1) * FRAME <= 1900; i += 1) { + frames.push({ + t: start + (i - 1) * FRAME, + box: {x: 0, y: 3 * i, width: 10, height: 10}, + } as TrackedFrame); + } + return frames; +} + +describe('busyWindowProof on-screen start', () => { + it('judges a just-started animation from its first presented change when it was seen at rest before the block, and reports when', () => { + // On screen 87 ms into the block: four intervals after the judged part's start. + const proof = busyWindowProof(startingAt(1087), BLOCK); + expect(proof.verdict.failures).toEqual([]); + expect(proof.onScreenFromMs).toBeCloseTo(87, 5); + expect(proof.judged.from).toBeCloseTo(1087, 5); + expect(proof.summary).toMatch( + /on screen from \+87 ms \(nothing moved before the block\)/, + ); + }); + + it('still fails an animation that was already moving and stalls at the block start', () => { + const k = at(1020); + const proof = busyWindowProof(moving([k, k + 1, k + 2, k + 3]), BLOCK); + expect(proof.onScreenFromMs).toBeNull(); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('still fails a stall at the start when no frame before the block shows it at rest', () => { + const unseen = startingAt(1087).filter(f => f.t >= BLOCK.from); + const proof = busyWindowProof(unseen, BLOCK); + expect(proof.onScreenFromMs).toBeNull(); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('still fails a start the block itself delays until it ends', () => { + const proof = busyWindowProof(startingAt(1720), BLOCK); + expect(proof.onScreenFromMs).toBeNull(); + expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('still fails a start that leaves fewer than minJudgedFrames inside the block', () => { + const proof = busyWindowProof(startingAt(1640), BLOCK); + expect(proof.onScreenFromMs).toBeCloseTo(640, 5); + expect(proof.verdict.failures.join('\n')).toMatch( + /frames < 4: the animation under test was not moving/, + ); + }); +}); + +describe('busyWindowProof position step', () => { + it("reports the cast's position step and motion per frame, and notes motion under two steps without failing", () => { + // 2.4 px a frame on a 2 px grid, as WebKit's casts report positions. + const grid = moving().map( + (f, i) => + ({ + ...f, + box: {...f.box!, y: 2 * Math.round((2.4 * i) / 2)}, + }) as TrackedFrame, + ); + const coarse = busyWindowProof(grid, BLOCK); + expect(coarse.stepPx).toBe(2); + expect(coarse.pxPerFrame).toBeCloseTo(2.4, 1); + expect(coarse.notes.join('\n')).toMatch( + /under two of the cast's 2 px position steps/, + ); + expect(coarse.summary).toMatch(/2\.4 px\/frame in 2 px steps; note: /); + expect(coarse.verdict.failures).toEqual([]); + // 3 px a frame in whole pixels: resolvable, nothing to note. + const fine = busyWindowProof(moving(), BLOCK); + expect(fine.stepPx).toBe(1); + expect(fine.pxPerFrame).toBeCloseTo(3, 1); + expect(fine.notes).toEqual([]); + }); +}); + +/** The compositor presenting every frame from `from` to `to`, minus `gaps` ([from, to] spans). */ +function drawn( + gaps: ReadonlyArray = [], + from = 700, + to = 2000, +): number[] { + const draws: number[] = []; + for (let t = from; t <= to; t += FRAME) { + if (!gaps.some(([a, b]) => t > a && t < b)) { + draws.push(t); + } + } + return draws; +} + +describe('busyWindowProof with the compositor record', () => { + it('with a compositor record, judges a sparse cast from its first to its last moved frame', () => { + // One screencast frame in six, as on a loaded machine, with the compositor + // presenting every frame: the frames before the first and after the last + // moved one are the stall check's, not misses. + const sparse = moving().filter((_, i) => i % 6 === 0); + expect( + busyWindowProof(sparse, BLOCK, {presented: drawn()}).verdict.failures, + ).toEqual([]); + }); + + it('with a compositor record and an on-screen start, counts the frames it credits as skipped in what it judges', () => { + // On screen 87 ms into the block, then four delivered frames that show + // no new position: that fails on a full cast. + const misses = [15, 20, 25, 30]; + const missing = startingAt(1087).map((f, i, all) => + misses.includes(i) + ? ({...f, box: {...all[i - 1].box!}} as TrackedFrame) + : f, + ); + const full = busyWindowProof(missing, BLOCK, {presented: drawn()}); + expect(full.verdict.failures.join('\n')).toMatch(/expected frames moved/); + // The screencast skipping the five frames after the on-screen frame + // changes nothing: the compositor presented them, and they are judged. + const skip = (frames: TrackedFrame[]) => + frames.filter((_, i) => i < 2 || i > 6); + const skipped = busyWindowProof(skip(missing), BLOCK, { + presented: drawn(), + }); + expect(skipped.recorderSkipped).toBe(5); + expect(skipped.verdict.failures.join('\n')).toMatch( + /expected frames moved/, + ); + // And a run with no misses never reads as more than every frame moved. + const clean = busyWindowProof(skip(startingAt(1087)), BLOCK, { + presented: drawn(), + }); + expect(clean.verdict.failures).toEqual([]); + expect(clean.movedFraction).toBeLessThanOrEqual(1); + }); + + it('counts a gap the compositor drew through as skipped, and one it did not draw through as a stall', () => { + const k = at(1300); + const gap: [number, number] = [ + 800 + (k - 1) * FRAME, + 800 + (k + 4) * FRAME, + ]; + const frames = moving([k, k + 1, k + 2, k + 3]); + expect(busyWindowProof(frames, BLOCK).verdict.pass).toBe(false); + const skipped = busyWindowProof(frames, BLOCK, {presented: drawn()}); + expect(skipped.verdict.failures).toEqual([]); + expect(skipped.recorderSkipped).toBe(4); + const stalled = busyWindowProof(frames, BLOCK, {presented: drawn([gap])}); + expect(stalled.recorderSkipped).toBe(0); + expect(stalled.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('still fails an element that does not move while the compositor presents for something else', () => { + const frozen = moving().map(f => + f.t >= BLOCK.from && f.t <= BLOCK.to + ? ({...f, box: {...f.box!, y: 3 * at(BLOCK.from)}} as TrackedFrame) + : f, + ); + const proof = busyWindowProof(frozen, BLOCK, {presented: drawn()}); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/); + }); + + it('still fails a frozen animation, whether or not the compositor presents during the block', () => { + const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to); + for (const presented of [drawn([[BLOCK.from, BLOCK.to]]), drawn()]) { + const proof = busyWindowProof(frozen, BLOCK, {presented}); + expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + } + }); + + it('keeps a gap a stall when the screencast delivered a frame inside it that did not move', () => { + const k = at(1300); + const frames = moving([k, k + 1, k + 3]).map(f => + f.t > 800 + (k + 1.5) * FRAME && f.t < 800 + (k + 2.5) * FRAME + ? ({...f, box: {...f.box!, y: 3 * (k - 1)}} as TrackedFrame) + : f, + ); + const proof = busyWindowProof(frames, BLOCK, {presented: drawn()}); + expect(proof.recorderSkipped).toBe(0); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('still counts single dropped frames the compositor did not present as missed', () => { + const k = at(1050); + const drops = [0, 6, 12, 18, 24, 30].map(d => k + d); + const frames = moving(drops); + const gaps = drops.map( + i => + [800 + (i - 0.5) * FRAME, 800 + (i + 0.5) * FRAME] as [number, number], + ); + expect( + busyWindowProof(frames, BLOCK, { + presented: drawn(gaps), + }).verdict.failures.join('\n'), + ).toMatch(/expected frames moved/); + expect( + busyWindowProof(frames, BLOCK, {presented: drawn()}).verdict.failures, + ).toEqual([]); + }); + + it('judges a gap at the edge of the judged part from the screencast alone', () => { + // Nothing delivered for the block's first 90 ms: a stall at its start, + // whatever the compositor drew, since no moved frame opens the gap. + const frames = moving().filter( + f => f.t < BLOCK.from || f.t > BLOCK.from + 90, + ); + const proof = busyWindowProof(frames, BLOCK, {presented: drawn()}); + expect(proof.recorderSkipped).toBe(0); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('treats an empty record as no record, and says so', () => { + const frames = moving([at(1300), at(1300) + 1, at(1300) + 2, at(1300) + 3]); + const proof = busyWindowProof(frames, BLOCK, {presented: []}); + expect(proof.verdict.failures).toEqual( + busyWindowProof(frames, BLOCK).verdict.failures, + ); + expect(proof.summary).toMatch(/no compositor frames recorded/); + }); +}); + +describe('inPresentationOrder', () => { + it('sorts frames delivered out of order, so motion stays monotonic', () => { + const frames = moving(); + const k = at(1040); + const shuffled = [...frames]; + [shuffled[k], shuffled[k + 1]] = [shuffled[k + 1], shuffled[k]]; + expect(busyWindowProof(shuffled, BLOCK).monotonic).toBe(false); + const proof = busyWindowProof(inPresentationOrder(shuffled), BLOCK); + expect(proof.monotonic).toBe(true); + expect(proof.verdict.failures).toEqual([]); + }); +}); diff --git a/internal/interaction-harness/src/busy.ts b/internal/interaction-harness/src/busy.ts new file mode 100644 index 0000000000000..20c3d86a1184e --- /dev/null +++ b/internal/interaction-harness/src/busy.ts @@ -0,0 +1,577 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file busy.ts + * @input Uses a Playwright `Page` + * @output `blockMainThread` — a synchronous busy loop on the page's main thread + * that reports exactly when it ran — and `busyWindowProof`, which reads frame + * evidence for that window. + * @position The busy-main-thread proof: an animation that runs on the + * compositor keeps advancing at display rate while JavaScript holds the main + * thread; one driven from the main thread (rAF, style writes, layout + * properties) freezes for the whole block. Pair it with ./frames + * (`startScreencast` + `trackMarker`), which observes off the main thread. + * + * SYNC: Keep the README's busy-main-thread section equal to this contract. + */ + +import type {Locator, Page} from '@playwright/test'; +import {frameStats, type FrameStats, type TrackedFrame} from './frames'; + +export interface BusyWindow { + /** Epoch ms when the loop started and ended, on the page's own clock. */ + readonly from: number; + readonly to: number; +} + +/** + * Holds the page's main thread for `ms` (a synchronous loop: no task, frame, + * rAF, event or microtask runs meanwhile). Resolves with the window it held. + * + * Block MID-FLIGHT only: start the gesture or animation first. A touch that + * starts while the main thread is busy does not scroll or begin a gesture in + * headless Chromium, even with only passive listeners, and an animation that + * has not started cannot start (see `waitForAnimation`). Finger tracking + * under a busy main thread is a simulator proof, not a Playwright one. + * + * With `during`, the page itself waits until an animation on that element (or + * inside it) has run `minProgressMs`, then blocks from a task it queues in + * that frame. A round trip from the test would start the block later the + * busier the machine is, leaving less of a short animation to judge. The task + * runs after the frame's rendering update, which is where WebKit hands a newly + * started animation to its compositor: a loop inside the rAF callback itself + * holds that hand-off, and even a compositor animation freezes. To catch a + * settle near its start, call it before the release that starts the settle and + * await it after: the page is already waiting when the animation begins. Any + * animation inside `during` counts, so pick an element that runs nothing + * else meanwhile. + * + * An animation counts once it is no longer pending and has run at least one + * frame (`minProgressMs` below 16 ms is raised to 16). One created in the + * frame the page looks is still pending, not yet on the compositor + * (Chromium), and in WebKit its start time resolves before its layer is + * flushed: a block that started on it froze it. + */ +/** One display frame, ms: the least progress `blockMainThread` waits for. */ +const ONE_FRAME_MS = 16; + +export async function blockMainThread( + page: Page, + ms: number, + { + during, + minProgressMs = 16, + timeout = 5_000, + }: {during?: Locator; minProgressMs?: number; timeout?: number} = {}, +): Promise { + const minProgress = Math.max(minProgressMs, ONE_FRAME_MS); + if (during == null) { + return page.evaluate(duration => { + const from = performance.timeOrigin + performance.now(); + const end = performance.now() + duration; + let spins = 0; + while (performance.now() < end) { + spins += 1; + } + return {from, to: performance.timeOrigin + performance.now(), spins}; + }, ms); + } + return during.evaluate( + (element, {duration, minProgress, wait}) => + new Promise((resolve, reject) => { + // The harness's own waits run on the real timers: a frozen timer gate + // (story.ts) holds only the page's. + const gate = ( + window as unknown as { + __harnessTimerGate?: { + frame?: typeof requestAnimationFrame; + timeout?: typeof setTimeout; + }; + } + ).__harnessTimerGate; + const frame = gate?.frame ?? window.requestAnimationFrame.bind(window); + const later = gate?.timeout ?? window.setTimeout.bind(window); + const deadline = performance.now() + wait; + const tick = () => { + const running = element + .getAnimations({subtree: true}) + .some( + a => + a.playState === 'running' && + !a.pending && + Number(a.currentTime ?? 0) >= minProgress, + ); + if (running) { + // The same loop as above, from a task queued in the frame that saw + // the animation: it runs once that frame's rendering update has + // handed the animation to the compositor. + later(() => { + const from = performance.timeOrigin + performance.now(); + const end = performance.now() + duration; + while (performance.now() < end) { + // Busy: nothing else runs. + } + resolve({from, to: performance.timeOrigin + performance.now()}); + }, 0); + } else if (performance.now() > deadline) { + reject(new Error('blockMainThread: no animation started')); + } else { + frame(tick); + } + }; + frame(tick); + }), + {duration: ms, minProgress, wait: timeout}, + ); +} + +export interface BusyVerdict { + readonly pass: boolean; + /** Each criterion the window missed, with its measured value and limit. */ + readonly failures: readonly string[]; +} + +export interface BusyProof extends FrameStats { + readonly window: BusyWindow; + /** + * The part of the window judged: inside the margins, up to `settledWithin`, + * and from `onScreenFromMs` when that is set. + */ + readonly judged: {readonly from: number; readonly to: number}; + /** + * When the tracked value was seen at rest before the judged part (a frame + * there, no change in the 250 ms before it: a just-started animation not on + * screen yet), ms after the window's start of its first presented change + * inside the block, where judging starts; null otherwise. + */ + readonly onScreenFromMs: number | null; + /** + * The cast's position step, CSS px: the largest of 0.5, 1 and 2 that every + * tracked position is a multiple of (WebKit's casts are 2: see the README). + */ + readonly stepPx: number; + /** The tracked value's mean motion per screencast interval in the judged part, px. */ + readonly pxPerFrame: number; + /** Facts worth reading beside the verdict, which never fail it. */ + readonly notes: readonly string[]; + /** + * With a compositor record (`presented`): frames the screencast skipped + * between two moved frames while the compositor presented them, which are + * neither stalls nor misses; 0 without one. + */ + readonly recorderSkipped: number; + /** + * The compositor's frame interval around the window, ms, from `presented`; + * null without a usable record (the screencast alone is judged). + */ + readonly compositorFrameMs: number | null; + /** + * The screencast's frame interval, from the frames around the window as well + * as inside it (a short window has few intervals; a frozen one has none). + */ + readonly cadenceMs: number; + /** + * Whole frame intervals in the judged part at the measured cadence (the + * compositor's, when recorded): the `minJudgedFrames` floor reads it. + */ + readonly expectedFrames: number; + /** Frames the screencast delivered in the judged part, with the marker found. */ + readonly deliveredFrames: number; + /** + * Longest time inside the window with no presented change, the window's own + * edges included. A frozen animation presents no new frame at all (nothing + * on screen changed, so the compositor sends none): its stall is the whole + * window. + */ + readonly longestStallMs: number; + /** Where the longest stall starts, ms after the window's start. */ + readonly longestStallAtMs: number; + /** + * `longestStallMs` in screencast intervals, rounded: 1 is no frame missed, + * 2 is one dropped frame. Rounding absorbs timestamp jitter. + */ + readonly longestStallFrames: number; + /** + * Frames in the window that presented a new position (the first compared + * with the last frame before the window). + */ + readonly moved: number; + /** + * `moved` (and `recorderSkipped`) over the frames judged: the frames the + * screencast delivered in the judged part, or with a compositor record the + * compositor's frames from the first to the last moved frame; + * `expectedFrames` when nothing moved, or fewer than `minJudgedFrames` + * frames were delivered. + */ + readonly movedFraction: number; + /** The judgment against the limits (see `busyWindowProof`). */ + readonly verdict: BusyVerdict; + /** One line with the raw numbers, for an annotation or a failure message. */ + readonly summary: string; +} + +/** + * Frame evidence for a busy window: how many frames the screencast presented + * inside it, how many of them moved the tracked axis, and the longest stall. + * The window is trimmed by `marginMs` at both ends so the frame the block + * started on and the one it ended on are not counted. + * + * The verdict judges the frames the screencast delivered, whatever its rate + * (a loaded machine's screencast can deliver one frame in five or six), and + * tolerates nothing a main-thread animation does (it presents no frame for + * the whole block). It passes when: + * - the judged part presented at least one frame, and, only when a proof + * claims display rate and passes `maxMedianFrameMs`, their median interval + * is at most that; + * - no stall spans more than `maxStallFrames` screencast intervals; + * - at least `minMovedFraction` of the frames judged moved the tracked axis + * (a short window may still miss one): the frames the screencast delivered + * in the judged part, so an irregular cast is judged on what it delivered + * and a frame it did not deliver is a gap for the stall check, not a miss + * (unless it delivered fewer than `minJudgedFrames` there: then every frame + * of the judged part's span counts, so one late frame is not 1/1 moved); + * with a compositor record, the compositor's frames from the first to the + * last moved frame (the gaps before and after are the stall check's); + * - the judged part spans at least `minJudgedFrames` frames. Fewer means the + * animation under test was not moving during the block (it settled before + * the block, or the block caught another animation), so there is nothing to + * judge, and that fails rather than passes. + * + * A just-started animation can reach the screen after the block began (its + * hand-off to the compositor lags under load: 50–100 ms in WebKit). When the + * tracked value was seen at rest before the judged part (a frame there, and no + * change in the 250 ms before it), judging starts at its first presented + * change inside the block, which the + * proof reports as `onScreenFromMs`. The judged part still ends with the + * block and the `minJudgedFrames` floor applies to it, so a start the block + * itself delays fails; an animation that was already moving is judged from + * the block's start, and a stall there fails as before. + * + * `stepPx` and `pxPerFrame` say whether a frame's motion is resolvable at + * all: under two position steps per frame, a frame delivered early can show + * no step and read as not moved, which the proof notes (never a failure). + * + * A loaded machine's screencast can skip frames the compositor still presents + * (Chromium). Given the compositor's record (`presented`: when it presented + * each frame), a gap between two moved frames with no frame + * delivered inside it is measured between the compositor's own frames: it is + * a stall only where the compositor presented nothing, and the frames it did + * present there are neither stalls nor misses (`recorderSkipped`). A gap + * that opens or closes the judged part, or holds a delivered frame that did + * not move, is judged from the screencast as before, so a frozen element + * fails whatever else the compositor draws. + */ +export function busyWindowProof( + tracked: readonly TrackedFrame[], + window: BusyWindow, + { + axis = 'y', + marginMs = 20, + settledWithin, + settledAt, + maxMedianFrameMs = Infinity, + maxStallFrames = 3, + minMovedFraction = 0.9, + minJudgedFrames = 4, + presented, + }: { + axis?: 'x' | 'y' | 'width' | 'height'; + marginMs?: number; + /** + * Opt-in, for a proof that claims display rate: the most the median frame + * interval may be, ms (20 at 60 Hz). Off by default, since it measures the + * screencast's own rate as much as the animation. + */ + maxMedianFrameMs?: number; + /** The most screencast intervals one stall may span (3 = two dropped frames). */ + maxStallFrames?: number; + /** + * The least fraction of the expected frames that must move. One missed + * frame is always allowed, so a window of under ten frames is not judged + * stricter than a long one. + */ + minMovedFraction?: number; + /** The fewest whole frame intervals the judged part of the window may span. */ + minJudgedFrames?: number; + /** + * Stop judging once the value is within this many px of where it ends. + * The screencast reports whole CSS pixels, so an easing tail that still + * moves shows plateaus (26, 26, 26 against a final 24) that would read + * as a stall: 2 is the usual value. + */ + settledWithin?: number; + /** + * Where the value ends, for `settledWithin`: read it from the page once the + * animation has ended (`waitForRest`). Defaults to the last frame's value, + * which can be stale: once an accelerated animation ends while the main + * thread is still blocked, WebKit presents the layer's last committed + * position until the main thread commits the end state. + */ + settledAt?: number; + /** + * When the compositor presented each frame, epoch ms on the page clock + * (Chromium's `Display::FrameDisplayed`). Empty or absent, the screencast + * is judged alone. + */ + presented?: readonly number[]; + } = {}, +): BusyProof { + let end = window.to - marginMs; + if (settledWithin != null) { + const valued = tracked.flatMap(f => + f.box == null ? [] : [{t: f.t, value: f.box[axis]}], + ); + const final = + settledAt ?? (valued.length > 0 ? valued[valued.length - 1].value : null); + const arrived = valued.find( + f => + f.t >= window.from && + final != null && + Math.abs(f.value - final) <= settledWithin, + ); + if (arrived != null) { + end = Math.min(end, arrived.t); + } + } + const inner = {from: window.from + marginMs, to: end}; + const values = tracked + .filter(f => f.t >= inner.from && f.t <= inner.to) + .map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]})); + const before = [...tracked].reverse().find(f => f.t < inner.from); + let previous: number | null = before?.box == null ? null : before.box[axis]; + const inWindow: number[] = []; + for (const v of values) { + if ( + v.value != null && + previous != null && + Math.abs(v.value - previous) > 0.25 + ) { + inWindow.push(v.t); + } + if (v.value != null) { + previous = v.value; + } + } + // Seen at rest before the judged part (a frame there, and no change in the + // 250 ms before it): the first presented change is the animation's on-screen + // start, and judging starts there. With no frame before, nothing is known, + // and the window is judged from its start. + const prior = tracked.filter(f => f.box != null && f.t < inner.from); + let leadFrom = 0; + prior.forEach((f, i) => { + if (f.t < inner.from - 250) { + leadFrom = i; + } + }); + const lead = prior.flatMap(f => (f.box == null ? [] : [f.box[axis]])); + const movedBefore = lead.some( + (value, i) => i > leadFrom && Math.abs(value - lead[i - 1]) > 0.25, + ); + const onScreenFrom = + prior.length > 0 && !movedBefore && inWindow.length > 0 + ? inWindow[0] + : null; + const judged = {from: onScreenFrom ?? inner.from, to: inner.to}; + const changes = inWindow.filter(t => t > judged.from || onScreenFrom == null); + const edges = [judged.from, ...changes, judged.to]; + const around = tracked + .filter(f => f.t >= inner.from - 250 && f.t <= inner.to + 250) + .map(f => f.t); + const intervals = around + .slice(1) + .map((t, i) => t - around[i]) + .sort((a, b) => a - b); + const stats = frameStats( + tracked.map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]})), + judged, + ); + const cadence = + intervals.length >= 4 + ? intervals[Math.floor(intervals.length / 2)] + : stats.medianFrameMs || 1000 / 60; + // The compositor's own frames, when recorded: its interval, from the frames + // around the judged part. + const draws = [...(presented ?? [])] + .filter(t => t >= judged.from - 250 && t <= judged.to + 250) + .sort((a, b) => a - b); + const drawIntervals = draws + .slice(1) + .map((t, i) => t - draws[i]) + .sort((a, b) => a - b); + const compositorFrameMs = + drawIntervals.length >= 4 + ? drawIntervals[Math.floor(drawIntervals.length / 2)] + : null; + const changed = new Set(inWindow); + const delivered = tracked + .filter(f => f.box != null && f.t > judged.from && f.t < judged.to) + .map(f => f.t); + // Each gap between presented changes, in intervals of the channel that saw + // it: the screencast's, or, for a gap between two moved frames with no + // frame delivered inside it, the compositor's own longest interval there. + const gaps = edges.slice(1).map((to, i) => { + const from = edges[i]; + const ms = to - from; + if ( + compositorFrameMs == null || + !changed.has(from) || + !changed.has(to) || + delivered.some(t => t > from && t < to) + ) { + return {from, ms, frames: Math.round(ms / cadence), skipped: 0}; + } + const span = draws.filter( + t => t > from - compositorFrameMs / 2 && t < to + compositorFrameMs / 2, + ); + const longest = + span.length >= 2 + ? Math.min(ms, Math.max(...span.slice(1).map((t, j) => t - span[j]))) + : ms; + const frames = Math.round(longest / compositorFrameMs); + return { + from, + ms: longest, + frames, + skipped: Math.max(0, Math.round(ms / compositorFrameMs) - frames), + }; + }); + const longestGap = gaps.reduce((a, b) => + b.frames > a.frames || (b.frames === a.frames && b.ms > a.ms) ? b : a, + ); + const longestStallMs = longestGap.ms; + // Where the longest stall starts, from the start of the window. + const longestStallAtMs = longestGap.from - window.from; + const longestStallFrames = longestGap.frames; + const skips = gaps.filter(g => g.skipped > 0); + const recorderSkipped = skips.reduce((sum, g) => sum + g.skipped, 0); + // With a compositor record, frames count in its intervals, and the frames + // it presented where the screencast skipped are presented frames. + const expectedFrames = Math.floor( + (judged.to - judged.from) / (compositorFrameMs ?? cadence), + ); + const moved = changes.length; + // The frames judged for motion: the screencast's own frames in the judged + // part (each `changes` entry is one of them), or, with a compositor record, + // the compositor's frames from the first to the last moved frame (the gaps + // before and after them are the stall check's). After an on-screen start + // they count from the on-screen frame, which `changes` leaves out, so the + // frames credited as skipped after it are judged too. A part that moved + // nothing, or delivered fewer than `minJudgedFrames` frames, is judged + // against its span, so a frozen animation misses here too (one late frame + // from before the block is not 1/1 moved). + const deliveredFrames = tracked.filter( + f => + f.box != null && + (onScreenFrom == null ? f.t >= judged.from : f.t > judged.from) && + f.t <= judged.to, + ).length; + const judgedFrames = + compositorFrameMs != null + ? changes.length > 0 + ? Math.round( + (changes[changes.length - 1] - (onScreenFrom ?? changes[0])) / + compositorFrameMs, + ) + (onScreenFrom == null ? 1 : 0) + : expectedFrames + : deliveredFrames < minJudgedFrames + ? expectedFrames + : deliveredFrames; + const movedFraction = (moved + recorderSkipped) / Math.max(1, judgedFrames); + // The cast's position step, and the motion per interval it has to resolve. + const positions = tracked.flatMap(f => (f.box == null ? [] : [f.box[axis]])); + const stepPx = + [2, 1, 0.5].find(step => + positions.every(p => Math.abs(p / step - Math.round(p / step)) < 0.01), + ) ?? 0.5; + const judgedValues = tracked.flatMap(f => + f.box != null && f.t >= judged.from && f.t <= judged.to + ? [{t: f.t, value: f.box[axis]}] + : [], + ); + const first = judgedValues[0]; + const last = judgedValues[judgedValues.length - 1]; + const pxPerFrame = + first == null || last == null || last.t <= first.t + ? 0 + : Math.abs(last.value - first.value) / ((last.t - first.t) / cadence); + const notes: string[] = []; + if (presented != null && compositorFrameMs == null) { + notes.push( + 'no compositor frames recorded around the window: judged from the screencast alone', + ); + } + if (recorderSkipped > 0) { + notes.push( + `the screencast skipped ${recorderSkipped} frames the compositor presented (at ${skips.map(g => `+${(g.from - window.from).toFixed(0)} ms`).join(', ')})`, + ); + } + if (pxPerFrame > 0 && pxPerFrame < 2 * stepPx) { + notes.push( + `${pxPerFrame.toFixed(1)} px/frame is under two of the cast's ${stepPx} px position steps: a frame delivered early can show no step and read as not moved`, + ); + } + const failures: string[] = []; + if (stats.frames === 0) { + failures.push('no frame presented in the judged part'); + } else if (stats.medianFrameMs > maxMedianFrameMs) { + failures.push( + `median frame interval ${stats.medianFrameMs.toFixed(1)} ms > ${maxMedianFrameMs} ms (${stats.frames} frames presented)`, + ); + } + if (longestStallFrames > maxStallFrames) { + failures.push( + `longest stall ${longestStallMs.toFixed(1)} ms = ${longestStallFrames} intervals > ${maxStallFrames}`, + ); + } + if (expectedFrames < minJudgedFrames) { + failures.push( + `${(judged.to - judged.from).toFixed(0)} ms judged = ${expectedFrames} frames < ${minJudgedFrames}: the animation under test was not moving during the block`, + ); + } + const allowedMisses = Math.max( + 1, + Math.floor((1 - minMovedFraction) * judgedFrames + 1e-9), + ); + if (judgedFrames - moved - recorderSkipped > allowedMisses) { + failures.push( + `${moved} of ${judgedFrames} expected frames moved${recorderSkipped > 0 ? ` (and ${recorderSkipped} the screencast skipped)` : ''} (${(movedFraction * 100).toFixed(0)}% < ${(minMovedFraction * 100).toFixed(0)}%)`, + ); + } + const onScreenFromMs = + onScreenFrom == null ? null : onScreenFrom - window.from; + const summary = + `${(judged.to - judged.from).toFixed(0)} ms judged at ${cadence.toFixed(1)} ms/frame: ${stats.frames} frames presented, ` + + `${moved}/${judgedFrames} moved${recorderSkipped > 0 ? ` and ${recorderSkipped} skipped by the screencast` : ''} (${(movedFraction * 100).toFixed(0)}%), ` + + `median ${stats.medianFrameMs.toFixed(1)} ms, ` + + `longest stall ${longestStallMs.toFixed(1)} ms (${longestStallFrames} intervals) at +${longestStallAtMs.toFixed(0)} ms, ` + + `${stats.monotonic ? 'monotonic' : 'not monotonic'}, ` + + `${pxPerFrame.toFixed(1)} px/frame in ${stepPx} px steps` + + (compositorFrameMs == null + ? '' + : `, compositor presenting every ${compositorFrameMs.toFixed(1)} ms`) + + (onScreenFromMs == null + ? '' + : `, on screen from +${onScreenFromMs.toFixed(0)} ms (nothing moved before the block)`) + + notes.map(note => `; note: ${note}`).join(''); + return { + ...stats, + window, + judged, + onScreenFromMs, + stepPx, + pxPerFrame, + notes, + recorderSkipped, + compositorFrameMs, + cadenceMs: cadence, + longestStallMs, + longestStallAtMs, + expectedFrames, + deliveredFrames, + longestStallFrames, + moved, + movedFraction, + verdict: {pass: failures.length === 0, failures}, + summary, + }; +} diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.ts new file mode 100644 index 0000000000000..c61ea7e171c6f --- /dev/null +++ b/internal/interaction-harness/src/frames.ts @@ -0,0 +1,465 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file frames.ts + * @input Uses a Playwright `Page`, a `Locator` for the element under test, and + * the engine's screencast (Chromium CDP `Page.startScreencast`; WebKit's + * `Screencast` domain) + * @output `startScreencast` (timestamped composited frames), `markElement` + * (a marker outline the frames can find), `trackMarker` (per-frame marker + * box), `frameStats`. + * @position Frame evidence for motion proofs. The screencast is captured off + * the page's main thread, from what the compositor presents, so it keeps + * delivering frames while a script blocks the main thread: that is what makes + * the busy-main-thread proof possible. + * + * Marker tracking needs no image library: frames are decoded and scanned in a + * scratch page of the same browser (createImageBitmap + canvas), so the + * harness adds no dependency. + * + * SYNC: Keep the README's frame-evidence section equal to what this measures. + */ + +import type {CDPSession, Locator, Page} from '@playwright/test'; +import {webkitPageSession} from './touch'; + +export interface CastFrame { + /** Presentation time, epoch ms (the compositor's frame timestamp). */ + readonly t: number; + /** Base64 PNG/JPEG. */ + readonly data: string; + readonly format: 'png' | 'jpeg'; + /** CSS px of the viewport the frame shows. */ + readonly viewportWidth: number; + readonly viewportHeight: number; +} + +export interface Screencast { + /** + * Frames captured so far: wait for the first one to know the cast saw the + * page before an animation starts (`busyWindowProof` then knows its at-rest + * position). + */ + frameCount(): number; + /** Stops and returns every frame captured, in order. */ + stop(): Promise; +} + +export interface ScreencastOptions { + /** Lossless frames by default: marker colours survive PNG, not JPEG. */ + readonly format?: 'png' | 'jpeg'; + readonly quality?: number; + /** + * Keep one presented frame in this many (default 1, every frame): a slower + * evidence channel, like a loaded machine's screencast, for proving that a + * verdict does not depend on the screencast's rate. + */ + readonly everyNthFrame?: number; +} + +/** + * Starts capturing composited frames. Chromium delivers one per presented + * frame (16.7 ms at 60 Hz), timestamped by the compositor; WebKit's own + * screencast delivers one per composited update (about every 14 ms in the + * Linux build), stamped on CLOCK_MONOTONIC and rebased here onto the wall + * clock by that clock's measured offset, so both engines share the page's + * timeline. Both keep delivering while the page's main thread is blocked, and + * neither sends a frame when nothing on screen changed. + */ +export async function startScreencast( + page: Page, + options: ScreencastOptions = {}, +): Promise { + const name = page.context().browser()?.browserType().name(); + const viewport = page.viewportSize() ?? {width: 0, height: 0}; + if (name === 'chromium') { + return chromiumScreencast(page, viewport, options); + } + if (name === 'webkit') { + return webkitScreencast(page, viewport, options); + } + throw new Error( + `startScreencast: unsupported browser ${name ?? '(unknown)'}`, + ); +} + +async function chromiumScreencast( + page: Page, + viewport: {width: number; height: number}, + {format = 'png', quality = 90, everyNthFrame = 1}: ScreencastOptions, +): Promise { + const cdp: CDPSession = await page.context().newCDPSession(page); + const frames: CastFrame[] = []; + const onFrame = (frame: { + data: string; + sessionId: number; + metadata: {timestamp?: number; deviceWidth: number; deviceHeight: number}; + }) => { + frames.push({ + t: (frame.metadata.timestamp ?? Date.now() / 1000) * 1000, + data: frame.data, + format, + viewportWidth: frame.metadata.deviceWidth || viewport.width, + viewportHeight: frame.metadata.deviceHeight || viewport.height, + }); + cdp + .send('Page.screencastFrameAck', {sessionId: frame.sessionId}) + .catch(() => {}); + }; + cdp.on('Page.screencastFrame', onFrame); + // One CSS px per frame px keeps decoding cheap; everyNthFrame 1 = every presented frame. + await cdp.send('Page.startScreencast', { + format, + quality, + maxWidth: viewport.width, + maxHeight: viewport.height, + everyNthFrame, + }); + return { + frameCount: () => frames.length, + async stop() { + await cdp.send('Page.stopScreencast'); + cdp.off('Page.screencastFrame', onFrame); + await cdp.detach().catch(() => {}); + return inPresentationOrder(frames); + }, + }; +} + +async function webkitScreencast( + page: Page, + viewport: {width: number; height: number}, + {quality = 90, everyNthFrame = 1}: ScreencastOptions, +): Promise { + const session = webkitPageSession(page); + const frames: CastFrame[] = []; + let generation = 0; + let delivered = 0; + const onFrame = (payload: unknown) => { + const frame = payload as {data: string; timestamp: number}; + // WebKit's screencast has no everyNthFrame: keep one frame in N here. + delivered += 1; + if ((delivered - 1) % everyNthFrame !== 0) { + session + .send('Screencast.screencastFrameAck', {generation}) + .catch(() => {}); + return; + } + frames.push({ + // WebKit stamps frames in seconds on the monotonic clock; rebased below. + t: frame.timestamp * 1000, + data: frame.data, + format: 'jpeg', + viewportWidth: viewport.width, + viewportHeight: viewport.height, + }); + session.send('Screencast.screencastFrameAck', {generation}).catch(() => {}); + }; + session.on('Screencast.screencastFrame', onFrame); + const started = (await session.send('Screencast.startScreencast', { + width: viewport.width, + height: viewport.height, + toolbarHeight: 0, + quality, + })) as {generation: number}; + generation = started.generation; + // WebKit stamps frames in seconds on CLOCK_MONOTONIC, which this process + // shares: rebase them by the wall clock's offset from that clock, sampled + // together. (Taking the first frame as the start shifted every frame earlier + // by however long that frame took to come, 60–100 ms on a busy page.) + const wallMinusMonotonic = Date.now() - Number(process.hrtime.bigint()) / 1e6; + return { + frameCount: () => frames.length, + async stop() { + await session.send('Screencast.stopScreencast'); + session.off?.('Screencast.screencastFrame', onFrame); + if (frames.length === 0) { + return frames; + } + return inPresentationOrder(frames).map(f => ({ + ...f, + t: f.t + wallMinusMonotonic, + })); + }, + }; +} + +/** + * Frames sorted by their presentation timestamp. A loaded machine can deliver + * screencast frames out of order (a frame stamped 35 ms arriving after one + * stamped 46 ms); judged in arrival order, a steadily moving element would + * appear to step backwards. + */ +export function inPresentationOrder( + frames: readonly T[], +): T[] { + return [...frames].sort((a, b) => a.t - b.t); +} + +/** The marker colour: magenta never occurs in Astryx themes. */ +export const MARKER_RGB: readonly [number, number, number] = [255, 0, 255]; + +export interface MarkerOptions { + readonly rgb?: readonly [number, number, number]; + /** + * `chip` (default): a small square child pinned inside the element's + * top-left corner, above every sibling (z-index max), so nothing the + * component paints can cover it. The element must be its containing block: + * a static element is made `position: relative` while marked. + * `outline`: an inset outline on the element itself (no DOM insertion), for + * elements that cannot take a child; children with their own layers can + * cover parts of it. + */ + readonly mode?: 'chip' | 'outline'; + /** Chip size or outline width, CSS px. */ + readonly size?: number; + /** Chip inset from the element's top-left corner, CSS px. */ + readonly inset?: number; +} + +/** + * Marks the element so screencast frames can find it: the mark moves, scales + * and fades with the element's own layer and never changes its layout. + * Returns an undo. + */ +export async function markElement( + locator: Locator, + options: MarkerOptions = {}, +): Promise<() => Promise> { + const { + rgb = MARKER_RGB, + mode = 'chip', + size = mode === 'chip' ? 16 : 6, + inset = 8, + } = options; + const id = `m${Math.random().toString(36).slice(2, 8)}`; + await locator.evaluate( + (element, {id: markerId, color, size: px, inset: offset, mode: kind}) => { + const el = element as HTMLElement; + el.setAttribute('data-harness-marker', markerId); + if (kind === 'outline') { + const style = document.createElement('style'); + style.setAttribute('data-harness-marker-style', markerId); + style.textContent = `[data-harness-marker="${markerId}"]{outline:${px}px solid rgb(${color.join(',')}) !important;outline-offset:-${px}px !important;}`; + document.head.append(style); + return; + } + if (getComputedStyle(el).position === 'static') { + el.setAttribute('data-harness-marker-position', el.style.position); + el.style.position = 'relative'; + } + const chip = document.createElement('div'); + chip.setAttribute('data-harness-marker-chip', markerId); + chip.setAttribute('aria-hidden', 'true'); + chip.style.cssText = `position:absolute;top:${offset}px;left:${offset}px;width:${px}px;height:${px}px;background:rgb(${color.join(',')});z-index:2147483647;pointer-events:none;`; + el.append(chip); + }, + {id, color: [...rgb], size, inset, mode}, + ); + return async () => { + await locator.page().evaluate(markerId => { + const el = document.querySelector( + `[data-harness-marker="${markerId}"]`, + ); + if (el?.hasAttribute('data-harness-marker-position')) { + el.style.position = + el.getAttribute('data-harness-marker-position') ?? ''; + el.removeAttribute('data-harness-marker-position'); + } + el?.removeAttribute('data-harness-marker'); + document + .querySelector(`[data-harness-marker-chip="${markerId}"]`) + ?.remove(); + document + .querySelector(`[data-harness-marker-style="${markerId}"]`) + ?.remove(); + }, id); + }; +} + +export interface Box { + readonly x: number; + readonly y: number; + readonly width: number; + readonly height: number; +} + +export interface TrackedFrame { + /** Epoch ms. */ + readonly t: number; + /** The marker's bounding box in CSS px, or null when it is not on screen. */ + readonly box: Box | null; + /** Marker pixels found (a fading element finds fewer at the same tolerance). */ + readonly pixels: number; +} + +/** + * Finds the marker in every frame. Decoding runs in a scratch page of the same + * browser, so neither the page under test nor Node needs an image decoder. + */ +export async function trackMarker( + page: Page, + frames: readonly CastFrame[], + { + rgb = MARKER_RGB, + tolerance = 40, + }: {rgb?: readonly [number, number, number]; tolerance?: number} = {}, +): Promise { + const scratch = await page.context().newPage(); + try { + const out: TrackedFrame[] = []; + // Batches keep each evaluate payload moderate. + for (let i = 0; i < frames.length; i += 12) { + const batch = frames.slice(i, i + 12); + const boxes = await scratch.evaluate( + async ({batch: items, color, tol}) => { + const results: { + box: {x: number; y: number; width: number; height: number} | null; + pixels: number; + }[] = []; + for (const item of items) { + const blob = await ( + await fetch(`data:image/${item.format};base64,${item.data}`) + ).blob(); + const bitmap = await createImageBitmap(blob); + const canvas = new OffscreenCanvas(bitmap.width, bitmap.height); + const context = canvas.getContext('2d'); + if (context == null) { + throw new Error('no 2d context'); + } + context.drawImage(bitmap, 0, 0); + const {data, width, height} = context.getImageData( + 0, + 0, + bitmap.width, + bitmap.height, + ); + let minX = Infinity; + let minY = Infinity; + let maxX = -Infinity; + let maxY = -Infinity; + let pixels = 0; + for (let y = 0; y < height; y += 1) { + for (let x = 0; x < width; x += 1) { + const o = (y * width + x) * 4; + if ( + Math.abs(data[o] - color[0]) <= tol && + Math.abs(data[o + 1] - color[1]) <= tol && + Math.abs(data[o + 2] - color[2]) <= tol + ) { + pixels += 1; + if (x < minX) { + minX = x; + } + if (x > maxX) { + maxX = x; + } + if (y < minY) { + minY = y; + } + if (y > maxY) { + maxY = y; + } + } + } + } + const scale = width / item.viewportWidth; + results.push({ + box: + pixels === 0 + ? null + : { + x: minX / scale, + y: minY / scale, + width: (maxX - minX + 1) / scale, + height: (maxY - minY + 1) / scale, + }, + pixels, + }); + } + return results; + }, + { + batch: batch.map(f => ({ + data: f.data, + format: f.format, + viewportWidth: f.viewportWidth, + })), + color: [...rgb], + tol: tolerance, + }, + ); + boxes.forEach((result, k) => + out.push({t: batch[k].t, box: result.box, pixels: result.pixels}), + ); + } + return out; + } finally { + await scratch.close(); + } +} + +export interface FrameStats { + readonly frames: number; + /** Frames whose tracked value differs from the frame before. */ + readonly changed: number; + /** Longest stretch without a change inside the window (an open stall at its end included), ms. */ + readonly longestStallMs: number; + /** Median spacing between frames, ms. */ + readonly medianFrameMs: number; + readonly monotonic: boolean; + readonly first: number | null; + readonly last: number | null; +} + +/** Summarises per-frame values (e.g. the marker's y) inside a time window. */ +export function frameStats( + samples: readonly {readonly t: number; readonly value: number | null}[], + window: {readonly from: number; readonly to: number} = { + from: -Infinity, + to: Infinity, + }, + epsilon = 0.25, +): FrameStats { + const inside = samples.filter( + s => s.t >= window.from && s.t <= window.to && s.value != null, + ) as {t: number; value: number}[]; + let changed = 0; + let increasing = 0; + let decreasing = 0; + let stallStart = inside[0]?.t ?? 0; + let longestStallMs = 0; + for (let i = 1; i < inside.length; i += 1) { + const delta = inside[i].value - inside[i - 1].value; + if (Math.abs(delta) > epsilon) { + changed += 1; + if (delta > 0) { + increasing += 1; + } else { + decreasing += 1; + } + longestStallMs = Math.max(longestStallMs, inside[i].t - stallStart); + stallStart = inside[i].t; + } + } + // A stall still open when the window closes counts too: a frozen animation never changes at all. + if (inside.length > 0) { + longestStallMs = Math.max( + longestStallMs, + inside[inside.length - 1].t - stallStart, + ); + } + const gaps = inside + .slice(1) + .map((s, i) => s.t - inside[i].t) + .sort((a, b) => a - b); + return { + frames: inside.length, + changed, + longestStallMs, + medianFrameMs: gaps.length ? gaps[Math.floor(gaps.length / 2)] : 0, + monotonic: increasing === 0 || decreasing === 0, + first: inside[0]?.value ?? null, + last: inside[inside.length - 1]?.value ?? null, + }; +} diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts index 7e47af95b2c81..a6c73fdf88362 100644 --- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts +++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts @@ -3,24 +3,28 @@ /** * @file harness.interaction.browser.spec.ts * @input Fixture pages written inline (no Storybook), real Chromium/WebKit - * @output Proof that each harness instrument measures what it claims: - * scripted timestamps reach the page; a flick flings natively (Chromium); - * a second finger and a pinch arrive as distinct pointers; sampled curves - * match the oracles; the animation-state waits report a running transition - * and its end, and judge rest by presented values even while an animation - * holds its end, and a watch taken on first contact ignores an animation - * already running and keeps the start of one the page cancels when it ends; individual transform properties are sampled; a sample at + * @output Proof that each harness instrument measures what it claims (the + * busy-main-thread proof with a negative control beside its positive + * check): scripted timestamps reach the + * page; a flick flings natively (Chromium); a second finger and a pinch + * arrive as distinct pointers; the screencast shows a compositor animation + * advancing through a blocked main thread while a rAF animation freezes; + * sampled curves + * match the oracles, individual transform properties included; a sample at * an end reports the time it was read at; holding takes only running * document-timeline animations, each at its own offset, holds each where it * was read, adds a second hold to the held set without moving the first - * hold's times, drops one the page cancels, and releases only them; - * storyboards write a sheet and a trace that follows the curve, an endless - * animation does not set their length, and a photograph Chromium returns - * empty is taken again two frames later; a touch filmstrip keeps scripted - * event times while a handler clock sees its pacing; the timer gate holds a - * component timer until the filmstrip is done, drops callbacks the page - * cancels, stamps held frames with the time they run, and leaves the - * harness its own frames. + * hold's times, drops one the page cancels, and releases only them; the + * animation-state waits report a running transition and its end, and judge + * rest by presented values even while an animation holds its end, and a + * watch taken on first contact ignores an animation already running and + * keeps the start of one the page cancels when it ends; storyboards write a + * sheet and a trace that follows the curve, an endless animation does not + * set their length, and a photograph Chromium returns empty is taken again + * two frames later; a touch filmstrip keeps scripted event times while a + * handler clock sees its pacing; the timer gate holds a component timer + * until the filmstrip is done, drops callbacks the page cancels, stamps held + * frames with the time they run, and leaves the harness its own frames. * @position The harness's own test. A component spec trusts these * instruments; this file is why it may. */ @@ -28,6 +32,8 @@ import {expect, test, type Page} from '@playwright/test'; import { animationStoryboard, + blockMainThread, + busyWindowProof, compareCurve, cubicBezier, curveOf, @@ -38,6 +44,7 @@ import { holdAnimations, installInputLog, installTimerGate, + markElement, merge, parseTransform, PARTIAL_LIFT_SKIP, @@ -47,12 +54,14 @@ import { sampleCurve, seekAnimations, springStep, + startScreencast, Storyboard, tap, thawTimers, timeline, touchDriver, touchStoryboard, + trackMarker, waitForAnimation, waitForRest, watchAnimations, @@ -71,6 +80,24 @@ const FIXTURE = ` + +
`; + +async function busyFixture(page: Page) { + await page.route('http://busy.test/', route => + route.fulfill({contentType: 'text/html', body: BUSY_FIXTURE}), + ); + await page.goto('http://busy.test/'); +} + // A real navigation (not setContent, whose document.open() erases the window // listeners the input log installs). async function fixture(page: Page) { @@ -242,6 +269,161 @@ test.describe('touch input', () => { }); }); +test.describe('frame evidence', () => { + test('a compositor animation advances every frame through a blocked main thread; a rAF one freezes, whatever the screencast rate', async ({ + page, + }, testInfo) => { + await busyFixture(page); + const box = page.locator('#box'); + await markElement(box, {mode: 'outline'}); + + const run = async (driver: 'compositor' | 'raf', everyNthFrame = 1) => { + await box.evaluate((el, how) => { + el.getAnimations().forEach(a => a.cancel()); + el.style.transform = 'none'; + if (how === 'compositor') { + el.animate( + [{transform: 'translateY(0px)'}, {transform: 'translateY(560px)'}], + {duration: 1600, easing: 'linear'}, + ); + } else { + const start = performance.now(); + const step = (now: number) => { + const p = Math.min(1, (now - start) / 1600); + el.style.transform = `translateY(${560 * p}px)`; + if (p < 1) { + requestAnimationFrame(step); + } + }; + requestAnimationFrame(step); + } + }, driver); + const cast = await startScreencast(page, {everyNthFrame}); + await page.waitForTimeout(200); + const window = await blockMainThread(page, 700); + await page.waitForTimeout(200); + const tracked = await trackMarker(page, await cast.stop()); + return busyWindowProof(tracked, window, {axis: 'y'}); + }; + + const compositor = await run('compositor'); + testInfo.annotations.push({ + type: 'busy-main-thread compositor', + description: compositor.summary, + }); + expect(compositor.verdict.failures, compositor.summary).toEqual([]); + expect(compositor.monotonic).toBe(true); + // Resolvable on the cast's grid: two position steps or more per frame. + expect(compositor.notes, compositor.summary).toEqual([]); + expect(compositor.pxPerFrame).toBeGreaterThanOrEqual(2 * compositor.stepPx); + + const raf = await run('raf'); + testInfo.annotations.push({ + type: 'busy-main-thread rAF control', + description: raf.summary, + }); + // Frozen for the whole block: no presented frame moved (the compositor + // sends none at all when nothing on screen changes), and the verdict that + // passes the compositor animation fails it on every criterion. + expect(raf.changed).toBe(0); + expect(raf.longestStallMs).toBeGreaterThan(600); + expect(raf.verdict.pass).toBe(false); + expect(raf.verdict.failures.join('\n')).toMatch(/longest stall/); + expect(raf.verdict.failures.join('\n')).toMatch(/expected frames moved/); + + // The verdict judges the frames the screencast delivered, not its rate: + // at one frame in six (a loaded machine's ~100 ms), the compositor + // animation still passes and the rAF one still fails. + const slow = await run('compositor', 6); + expect(slow.cadenceMs, slow.summary).toBeGreaterThan(60); + expect(slow.verdict.failures, slow.summary).toEqual([]); + const slowRaf = await run('raf', 6); + expect(slowRaf.verdict.pass, slowRaf.summary).toBe(false); + expect(slowRaf.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + test('a block that starts during a just-started transition still lets the compositor advance it', async ({ + page, + }, testInfo) => { + // `during` blocks in the frame the transition has run 16 ms. In WebKit + // the transition reaches the compositor in that frame's rendering update, + // so a loop inside the rAF callback froze it (2 of 29 frames moved). + // Linear and longer than the block, so every frame of the block moves two + // position steps or more and the transition is still running when it + // ends. The cast shows the box at rest first, so a transition that reaches + // the screen after the block began is judged from there (onScreenFromMs). + await busyFixture(page); + const box = page.locator('#box'); + await markElement(box, {mode: 'outline'}); + const cast = await startScreencast(page); + await expect.poll(() => cast.frameCount()).toBeGreaterThan(0); + await box.evaluate(el => { + el.style.transition = 'transform 1600ms linear'; + el.getBoundingClientRect(); + el.style.transform = 'translateY(560px)'; + }); + const window = await blockMainThread(page, 700, { + during: box, + minProgressMs: 16, + }); + await waitForRest(box); + const tracked = await trackMarker(page, await cast.stop()); + const proof = busyWindowProof(tracked, window, {axis: 'y'}); + testInfo.annotations.push({ + type: 'busy-main-thread transition', + description: proof.summary, + }); + expect(proof.verdict.failures, proof.summary).toEqual([]); + expect(proof.monotonic).toBe(true); + }); + + test('a block never starts on an animation that is not on the compositor yet', async ({ + page, + }, testInfo) => { + // An animation created in the frame the helper looks is still pending (not + // committed to the compositor). A block that started on it then froze it + // in Chromium: no presented frame moved. + await busyFixture(page); + const box = page.locator('#box'); + await markElement(box, {mode: 'outline'}); + await box.evaluate(el => { + el.style.willChange = 'transform'; + }); + const cast = await startScreencast(page); + // A rAF loop registered before the helper's tick runs ahead of it in every + // frame; on its 12th frame it creates the animation, and the helper's tick + // looks in that same frame. + await box.evaluate(el => { + let n = 0; + const loop = () => { + n += 1; + if (n === 12) { + el.animate( + [{transform: 'translateY(0px)'}, {transform: 'translateY(560px)'}], + {duration: 1600, easing: 'linear'}, + ); + return; + } + requestAnimationFrame(loop); + }; + requestAnimationFrame(loop); + }); + const window = await blockMainThread(page, 700, { + during: box, + minProgressMs: 0, + }); + await waitForRest(box); + const tracked = await trackMarker(page, await cast.stop()); + const proof = busyWindowProof(tracked, window, {axis: 'y'}); + testInfo.annotations.push({ + type: 'busy-main-thread pending animation', + description: proof.summary, + }); + expect(proof.verdict.failures, proof.summary).toEqual([]); + expect(proof.monotonic).toBe(true); + }); +}); + test.describe('curve sampling', () => { test('a CSS transition samples onto its cubic-bezier to 0.05 px', async ({ page, @@ -1429,6 +1611,33 @@ test.describe('storyboards', {tag: '@storyboard'}, () => { } }); + test("the harness's own waits run on the real timers: a frozen timer gate does not hold blockMainThread's block", async ({ + page, + }) => { + await fixture(page); + await installTimerGate(page); + const box = page.locator('#box'); + await box.evaluate(el => { + el.animate( + [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}], + {duration: 1_500, easing: 'linear'}, + ); + }); + await freezeTimers(page); + // Settled, or still held after 5 s. + const within = (work: Promise) => + Promise.race([ + work.then(() => 'settled'), + new Promise(resolve => + setTimeout(() => resolve('held'), 5_000), + ), + ]); + expect( + await within(blockMainThread(page, 100, {during: box, timeout: 2_000})), + ).toBe('settled'); + await thawTimers(page); + }); + test('the timer gate holds a component timer until the filmstrip is done', async ({ page, }, testInfo) => { diff --git a/internal/interaction-harness/src/index.ts b/internal/interaction-harness/src/index.ts index 537bfc4d4beca..2bcb127f36d88 100644 --- a/internal/interaction-harness/src/index.ts +++ b/internal/interaction-harness/src/index.ts @@ -12,7 +12,9 @@ * SYNC: When a module is added, export it here and describe it in README.md. */ +export * from './busy'; export * from './curves'; +export * from './frames'; export * from './math'; export * from './output'; export * from './scripts'; diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.ts index 598f75d852cfe..5a1f2237c6725 100644 --- a/internal/interaction-harness/src/story.ts +++ b/internal/interaction-harness/src/story.ts @@ -119,7 +119,7 @@ const TIMER_GATE = `(() => { const realInterval = window.setInterval.bind(window); const realFrame = window.requestAnimationFrame.bind(window); const realCancelFrame = window.cancelAnimationFrame.bind(window); - const gate = {frozen: false, queue: [], frame: realFrame}; + const gate = {frozen: false, queue: [], frame: realFrame, timeout: realTimeout}; window.__harnessTimerGate = gate; const drop = key => { gate.queue = gate.queue.filter(entry => entry.key !== key); }; window.setTimeout = (fn, ms, ...args) => {