Skip to content

Full Metal optimization stack - #16136

Open
ktf wants to merge 10 commits into
AliceO2Group:devfrom
ktf:pr16136
Open

ktf wants to merge 10 commits into
AliceO2Group:devfrom
ktf:pr16136

Conversation

@ktf

@ktf ktf commented Nov 16, 2026

Copy link
Copy Markdown
Member

The tracklet constructor follows each tracklet row by row through a
chain of dependent reads: hit data, links and three spline evaluations
of the transformation per row, which take about 60% of its time. On an
M1 Max it is bound by their latency, not by bandwidth, address
translation or ALU (ALU utilization 16%, LLC limiter 22%, MMU limiter
5%, compute occupancy 10%), so it gets faster with more threads in
flight, and two limits kept them few: the 12 KB of rows that each group
copies to threadgroup memory, and registers allocated for groups of
128 threads.

On Metal the rows are now read from device memory, and the pipeline is
compiled as for 640 threads per group while it still runs groups of
128, as the minimum blocks of launch_bounds do. Either change alone
gains nothing. Together the kernel takes 14% less time on a quarter of
a 30 kHz Pb-Pb time frame: 421 -> 362 ms (--debug 1, three interleaved
rounds), with the same number of tracks. Compiling for 512 to 768
threads is within 4%, 1024 is slower; groups of 64 or 192 threads
are slower than 128.

The sim5pp cluster dump is byte-identical to the CPU, and pbpb30 in
deterministic mode gives 22,183 tracks. Other backends compile the
existing code unchanged.

ktf added 10 commits November 16, 2026 15:45
…r it

Needs review: shared TPC cluster finder code, selected per backend by
the new parameter CF_CHECK_PAD_BASELINE_SEGMENTED, which is 1 only for
Metal. CUDA, HIP and OpenCL keep the existing scan, since its speed
there could not be measured here.

GPUTPCCFCheckPadBaseline runs one block of 576 threads per row, and
without the HIP tail filter only the up to 138 threads that own a pad
do work, each walking all time bins of its pad. Without the filter the
baseline needs, per pad, only the number of time bins with charge,
the longest run of them and the largest charge, and these combine
over consecutive segments of time. So with the new parameter all four
threads per pad scan a quarter of the time bins each, straight from
the charge map, and the quarters are joined at the end. The block
size, its static_assert, the grid and the noisy-pad flags are
unchanged; with the HIP tail filter on, the existing scan runs.

On an M1 Max the kernel takes 1.6 ms per call instead of 3.7 (median
in a Metal System Trace), and the 30-collision Pb-Pb time frame 1088
ms per event instead of 1182 (3 interleaved rounds of 10 runs, load
average 4 to 6). The clusters are byte-identical to the CPU's, also
with noisy-pad thresholds tightened until 6 to 24% of the clusters
are dropped, and the CPU backend's own output is unchanged.
The 36 GPUTPCGMMergerSectorRefit launches, with their UnpackSaveNumber and
UnpackResetIds, all ran on stream 0. Distribute them over the streams, as
MergeBorders does, with stream 0 waiting for the other streams before
UnpackGlobal.

Why this is safe: the sectors only share the atomic nUnpackedTracks. Each
sector writes its own part of mTrackIDs and reads only its own sector
tracker's output, which is complete after the SynchronizeGPU() before the
merger. The other streams wait for stream 0's transfer of the merger
memory, and UnpackGlobal, which reads every sector's mTrackIDs and the
counter, runs after the join. The only thing lost is that the sector
tracks are no longer stored sector by sector, so mSectorTrackInfoIndex
[1, NSECTORS) are no longer meaningful. Only the deterministic sort and
the merger debug dumps (SectorTrackInfoFirst / Last) read them; the
production kernels use track.Sector() and SectorTrackInfoLocalTotal().
So deterministicGPUReconstruction, and with it the debugLevel 6 dumps,
keep the serial order. The index order of the sector tracks was already
not deterministic within a sector, because of the atomic.

On an M1 Max with Metal (8 streams), pbpb30 standalone benchmark:
- the refit block alone, between two synchronizations: 145.5 ms serial,
  132 ms on the streams; the refit is mostly throughput-bound there;
- per event, 10 interleaved rounds of --runs 10: 980 ms -> 969 ms (-1%);
- FST TPC-only warm TF at TFDELAY=2: within the noise (3 rounds).
The dumps of pbpb30 and sim5pp stay identical to the CPU's, sim5pp gives
19009 / 52 / 3027 in all runs, and the FST 1,355,432 clusters.

GPUs with more compute units than the M1 Max should gain more from it.
Untested with CUDA and HIP.
Move the comparators of the TPC merger's sorts, and MergeLooperParam they
sort, from GPUTPCGMMerger.cxx and GPUTPCGMO2Output.cxx into
GPUTPCGMMergerSortComp.h, unchanged. The CUDA and HIP backends see them
because they compile the kernel sources into one unit; a backend that
runs these sorts in host code needs them from a header.
… headers on Metal

GPUTPCCFDecodeZSDenseLink decodes a page with one warp, walking it
timebin by timebin and link by link. For every link header, the warp
also collected the channels with data, a thread per channel with three
warp scans, inside a serial chain of dependent loads that is the
latency limit of the kernel.

On Metal the header loop now only walks the headers, and thread 0
stores each link's level-2 bitmask and the offset of its channel
bitmask in shared memory (+64 bytes). The channels are then collected
with one thread per byte of a link's channel bitmask: one warp scan per
32 bytes instead of about three per link. Same samples in the same
order, same pointers into the page, including into the next one, same
error paths.

On an M1 Max the decode takes 23 to 39% less time per launch in normal
multi-stream runs (pbpb30 4.9 -> 3.7 ms, 50 kHz TF0 6.7 -> 4.2 ms),
about 1 to 2% of the full time frame. The cluster dumps are
byte-identical: sim5pp against the CPU, 30 kHz (1 of 4) and 50 kHz TF0
before against after. pbpb30 in deterministic mode gives 22,183 tracks
as before, and a 36 against 36 pass comparison at 50 kHz shows no
difference in the track count.

It is Metal-only for now, since it was measured only there: CUDA, HIP,
OpenCL and the CPU compile the existing code unchanged. The chain it
shortens is there on every backend, so it could be enabled for them
once timed.
The sorts of the merged tracks by quality and by q/pT, of the looper
candidates and of the O2 output tracks, which CUDA and HIP run with
thrust, ran on a single GPU thread otherwise: about 730 ms per event on
a 30-collision Pb-Pb time frame, at up to 140 times the CPU's time. The
GPU memory is shared with the host, so sort there once the stream is
drained, with the std::sort of the CPU backend, which also orders the
elements that compare equal as on the CPU. The number of elements is
read from the device copy of the counters, as the host copy of the O2
output count is still being transferred when the sort is launched.
The border-range sorts of the track merging between sectors, 360 per
time frame, took 312 ms per event on a 30-collision Pb-Pb time frame on
a single GPU thread. On the host they take about 6 ms, plus 52 ms of
waiting for the streams.
The candidate search of the border track merging, 180 launches per time
frame of a few thousand tracks each, took 46 ms per event (kernels
serialized) on a 30-collision Pb-Pb time frame on the GPU, where each
thread first walks the sorted ranges from the start. On the host, right
after the range sorts which already drained the stream, it takes 4.6 ms,
including pointing the processors at the host view of the shared memory
and back for each launch. One host thread, so that the merging across
the central electrode stays as on the CPU. Benchmark, 6 interleaved
rounds of 10 events: -6 +- 2 ms per event.

The kernel runs on a private copy of mProcessorsShadow: SetDevicePointers
of every allocated resource points the processors at the host memory,
they are copied, and the shadow is restored from a copy before the
kernel runs, so the host threads never touch it. The copy is the
processors' constant memory, with the parameters and calibration objects
of the host. With --debug 1 and above, a change of the shadow while the
kernel runs is fatal.
The extrapolation of the sector tracks into the neighbouring sectors is
one thread per track, of which only those near a sector border work, on
a long serial Kalman chain. It took 133 ms per event (kernels
serialized) on a 30-collision Pb-Pb time frame on the GPU, against 7.8 ms
on the host, run as the CPU backend does on the host threads. The
stream is drained first, which includes the waits for the neighbouring
sectors that the chain puts in it. Benchmark, 6 interleaved rounds of
10 events: -51 +- 2 ms per event with both border merging and
extrapolation on the host, -15 +- 2 ms against a single host thread.
The merger fits one track per thread, in the order of
GPUTPCGMMergerSortTracks, which sorts by the number of clusters so that
the threads of a SIMD group run for as long as each other. On Metal the
fit is bound by the address translation of its scattered reads (on an
M1 Max during the fit: MMU limiter 48%, TLB miss rate 68%, ALU
utilization 5%): the 32 threads of a group read clusters, transformation
and calibration data of 32 unrelated sectors.

The Metal backend, which runs this sort on the host, now sorts by the
number of clusters in buckets of 16, and within a bucket by side and
sector, so that the threads of a group mostly read the same pages. On a
quarter of a 30 kHz Pb-Pb time frame the fit takes 13% less time (679 ->
593 ms, --debug 1, interleaved runs). Buckets of 1 (610 ms), 8, 32 and
64 clusters are slower; ordering by z within a sector does not help; no
sort at all takes 950 ms.

The order does not change which track is fitted how: the sim5pp cluster
dump is byte-identical to the CPU, and pbpb30 in deterministic mode
gives 22,183 tracks.
The tracklet constructor follows each tracklet row by row through a
chain of dependent reads: hit data, links and three spline evaluations
of the transformation per row, which take about 60% of its time. On an
M1 Max it is bound by their latency, not by bandwidth, address
translation or ALU (ALU utilization 16%, LLC limiter 22%, MMU limiter
5%, compute occupancy 10%), so it gets faster with more threads in
flight, and two limits kept them few: the 12 KB of rows that each group
copies to threadgroup memory, and registers allocated for groups of
128 threads.

On Metal the rows are now read from device memory, and the pipeline is
compiled as for 640 threads per group while it still runs groups of
128, as the minimum blocks of __launch_bounds__ do. Either change alone
gains nothing. Together the kernel takes 14% less time on a quarter of
a 30 kHz Pb-Pb time frame: 421 -> 362 ms (--debug 1, three interleaved
rounds), with the same number of tracks. Compiling for 512 to 768
threads is within 4%, 1024 is slower; groups of 64 or 192 threads
are slower than 128.

The sim5pp cluster dump is byte-identical to the CPU, and pbpb30 in
deterministic mode gives 22,183 tracks. Other backends compile the
existing code unchanged.
@ktf
ktf requested a review from davidrohr as a code owner November 16, 2026 14:47
@ktf

ktf commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

This is actually the full set of optimizations I have on top of the current Metal implementation. I keep it here as a placeholder and I will try to understand what can already be merged and what not.

@ktf ktf changed the title GPU Metal: let more groups of the tracklet constructor fit on a core Full Metal optimization stack Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant