Repository navigation
Conversation
…r it Needs review: shared TPC cluster finder code, selected per backend by the new parameter CF_CHECK_PAD_BASELINE_SEGMENTED, which is 1 only for Metal. CUDA, HIP and OpenCL keep the existing scan, since its speed there could not be measured here. GPUTPCCFCheckPadBaseline runs one block of 576 threads per row, and without the HIP tail filter only the up to 138 threads that own a pad do work, each walking all time bins of its pad. Without the filter the baseline needs, per pad, only the number of time bins with charge, the longest run of them and the largest charge, and these combine over consecutive segments of time. So with the new parameter all four threads per pad scan a quarter of the time bins each, straight from the charge map, and the quarters are joined at the end. The block size, its static_assert, the grid and the noisy-pad flags are unchanged; with the HIP tail filter on, the existing scan runs. On an M1 Max the kernel takes 1.6 ms per call instead of 3.7 (median in a Metal System Trace), and the 30-collision Pb-Pb time frame 1088 ms per event instead of 1182 (3 interleaved rounds of 10 runs, load average 4 to 6). The clusters are byte-identical to the CPU's, also with noisy-pad thresholds tightened until 6 to 24% of the clusters are dropped, and the CPU backend's own output is unchanged.
The 36 GPUTPCGMMergerSectorRefit launches, with their UnpackSaveNumber and UnpackResetIds, all ran on stream 0. Distribute them over the streams, as MergeBorders does, with stream 0 waiting for the other streams before UnpackGlobal. Why this is safe: the sectors only share the atomic nUnpackedTracks. Each sector writes its own part of mTrackIDs and reads only its own sector tracker's output, which is complete after the SynchronizeGPU() before the merger. The other streams wait for stream 0's transfer of the merger memory, and UnpackGlobal, which reads every sector's mTrackIDs and the counter, runs after the join. The only thing lost is that the sector tracks are no longer stored sector by sector, so mSectorTrackInfoIndex [1, NSECTORS) are no longer meaningful. Only the deterministic sort and the merger debug dumps (SectorTrackInfoFirst / Last) read them; the production kernels use track.Sector() and SectorTrackInfoLocalTotal(). So deterministicGPUReconstruction, and with it the debugLevel 6 dumps, keep the serial order. The index order of the sector tracks was already not deterministic within a sector, because of the atomic. On an M1 Max with Metal (8 streams), pbpb30 standalone benchmark: - the refit block alone, between two synchronizations: 145.5 ms serial, 132 ms on the streams; the refit is mostly throughput-bound there; - per event, 10 interleaved rounds of --runs 10: 980 ms -> 969 ms (-1%); - FST TPC-only warm TF at TFDELAY=2: within the noise (3 rounds). The dumps of pbpb30 and sim5pp stay identical to the CPU's, sim5pp gives 19009 / 52 / 3027 in all runs, and the FST 1,355,432 clusters. GPUs with more compute units than the M1 Max should gain more from it. Untested with CUDA and HIP.
Move the comparators of the TPC merger's sorts, and MergeLooperParam they sort, from GPUTPCGMMerger.cxx and GPUTPCGMO2Output.cxx into GPUTPCGMMergerSortComp.h, unchanged. The CUDA and HIP backends see them because they compile the kernel sources into one unit; a backend that runs these sorts in host code needs them from a header.
… headers on Metal GPUTPCCFDecodeZSDenseLink decodes a page with one warp, walking it timebin by timebin and link by link. For every link header, the warp also collected the channels with data, a thread per channel with three warp scans, inside a serial chain of dependent loads that is the latency limit of the kernel. On Metal the header loop now only walks the headers, and thread 0 stores each link's level-2 bitmask and the offset of its channel bitmask in shared memory (+64 bytes). The channels are then collected with one thread per byte of a link's channel bitmask: one warp scan per 32 bytes instead of about three per link. Same samples in the same order, same pointers into the page, including into the next one, same error paths. On an M1 Max the decode takes 23 to 39% less time per launch in normal multi-stream runs (pbpb30 4.9 -> 3.7 ms, 50 kHz TF0 6.7 -> 4.2 ms), about 1 to 2% of the full time frame. The cluster dumps are byte-identical: sim5pp against the CPU, 30 kHz (1 of 4) and 50 kHz TF0 before against after. pbpb30 in deterministic mode gives 22,183 tracks as before, and a 36 against 36 pass comparison at 50 kHz shows no difference in the track count. It is Metal-only for now, since it was measured only there: CUDA, HIP, OpenCL and the CPU compile the existing code unchanged. The chain it shortens is there on every backend, so it could be enabled for them once timed.
The sorts of the merged tracks by quality and by q/pT, of the looper candidates and of the O2 output tracks, which CUDA and HIP run with thrust, ran on a single GPU thread otherwise: about 730 ms per event on a 30-collision Pb-Pb time frame, at up to 140 times the CPU's time. The GPU memory is shared with the host, so sort there once the stream is drained, with the std::sort of the CPU backend, which also orders the elements that compare equal as on the CPU. The number of elements is read from the device copy of the counters, as the host copy of the O2 output count is still being transferred when the sort is launched.
The border-range sorts of the track merging between sectors, 360 per time frame, took 312 ms per event on a 30-collision Pb-Pb time frame on a single GPU thread. On the host they take about 6 ms, plus 52 ms of waiting for the streams.
The candidate search of the border track merging, 180 launches per time frame of a few thousand tracks each, took 46 ms per event (kernels serialized) on a 30-collision Pb-Pb time frame on the GPU, where each thread first walks the sorted ranges from the start. On the host, right after the range sorts which already drained the stream, it takes 4.6 ms, including pointing the processors at the host view of the shared memory and back for each launch. One host thread, so that the merging across the central electrode stays as on the CPU. Benchmark, 6 interleaved rounds of 10 events: -6 +- 2 ms per event. The kernel runs on a private copy of mProcessorsShadow: SetDevicePointers of every allocated resource points the processors at the host memory, they are copied, and the shadow is restored from a copy before the kernel runs, so the host threads never touch it. The copy is the processors' constant memory, with the parameters and calibration objects of the host. With --debug 1 and above, a change of the shadow while the kernel runs is fatal.
The extrapolation of the sector tracks into the neighbouring sectors is one thread per track, of which only those near a sector border work, on a long serial Kalman chain. It took 133 ms per event (kernels serialized) on a 30-collision Pb-Pb time frame on the GPU, against 7.8 ms on the host, run as the CPU backend does on the host threads. The stream is drained first, which includes the waits for the neighbouring sectors that the chain puts in it. Benchmark, 6 interleaved rounds of 10 events: -51 +- 2 ms per event with both border merging and extrapolation on the host, -15 +- 2 ms against a single host thread.
The merger fits one track per thread, in the order of GPUTPCGMMergerSortTracks, which sorts by the number of clusters so that the threads of a SIMD group run for as long as each other. On Metal the fit is bound by the address translation of its scattered reads (on an M1 Max during the fit: MMU limiter 48%, TLB miss rate 68%, ALU utilization 5%): the 32 threads of a group read clusters, transformation and calibration data of 32 unrelated sectors. The Metal backend, which runs this sort on the host, now sorts by the number of clusters in buckets of 16, and within a bucket by side and sector, so that the threads of a group mostly read the same pages. On a quarter of a 30 kHz Pb-Pb time frame the fit takes 13% less time (679 -> 593 ms, --debug 1, interleaved runs). Buckets of 1 (610 ms), 8, 32 and 64 clusters are slower; ordering by z within a sector does not help; no sort at all takes 950 ms. The order does not change which track is fitted how: the sim5pp cluster dump is byte-identical to the CPU, and pbpb30 in deterministic mode gives 22,183 tracks.
The tracklet constructor follows each tracklet row by row through a chain of dependent reads: hit data, links and three spline evaluations of the transformation per row, which take about 60% of its time. On an M1 Max it is bound by their latency, not by bandwidth, address translation or ALU (ALU utilization 16%, LLC limiter 22%, MMU limiter 5%, compute occupancy 10%), so it gets faster with more threads in flight, and two limits kept them few: the 12 KB of rows that each group copies to threadgroup memory, and registers allocated for groups of 128 threads. On Metal the rows are now read from device memory, and the pipeline is compiled as for 640 threads per group while it still runs groups of 128, as the minimum blocks of __launch_bounds__ do. Either change alone gains nothing. Together the kernel takes 14% less time on a quarter of a 30 kHz Pb-Pb time frame: 421 -> 362 ms (--debug 1, three interleaved rounds), with the same number of tracks. Compiling for 512 to 768 threads is within 4%, 1024 is slower; groups of 64 or 192 threads are slower than 128. The sim5pp cluster dump is byte-identical to the CPU, and pbpb30 in deterministic mode gives 22,183 tracks. Other backends compile the existing code unchanged.
Member
Author
|
This is actually the full set of optimizations I have on top of the current Metal implementation. I keep it here as a placeholder and I will try to understand what can already be merged and what not. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tracklet constructor follows each tracklet row by row through a
chain of dependent reads: hit data, links and three spline evaluations
of the transformation per row, which take about 60% of its time. On an
M1 Max it is bound by their latency, not by bandwidth, address
translation or ALU (ALU utilization 16%, LLC limiter 22%, MMU limiter
5%, compute occupancy 10%), so it gets faster with more threads in
flight, and two limits kept them few: the 12 KB of rows that each group
copies to threadgroup memory, and registers allocated for groups of
128 threads.
On Metal the rows are now read from device memory, and the pipeline is
compiled as for 640 threads per group while it still runs groups of
128, as the minimum blocks of launch_bounds do. Either change alone
gains nothing. Together the kernel takes 14% less time on a quarter of
a 30 kHz Pb-Pb time frame: 421 -> 362 ms (--debug 1, three interleaved
rounds), with the same number of tracks. Compiling for 512 to 768
threads is within 4%, 1024 is slower; groups of 64 or 192 threads
are slower than 128.
The sim5pp cluster dump is byte-identical to the CPU, and pbpb30 in
deterministic mode gives 22,183 tracks. Other backends compile the
existing code unchanged.