IPI latency could be .. better #1234

comparing the latency of an IPI on Propolis (on Gimlet, Cosmo, or really anywhere else) to an IPI on hardware is .. somewhat unfortunate. on my 7950x, for example, perf bench futex wake -t 1 says:

root@ubuntu:~# perf bench futex wake -t 1
# Running 'futex/wake' benchmark:
Run summary [PID 1948]: blocking on 1 threads (at [private] futex 0x5bc168f3c738), waking up 1 at a time.

[Run 1]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 2]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 3]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 4]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 5]: Wokeup 1 of 1 threads in 0.0270 ms
[Run 6]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 7]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 8]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 9]: Wokeup 1 of 1 threads in 0.0280 ms
[Run 10]: Wokeup 1 of 1 threads in 0.0280 ms
Wokeup 1 of 1 threads in 0.0279 ms (+-0.36%)

versus on hardware elsewhere, a 3950x:

iximeow:~> perf  bench futex wake -t 1
# Running 'futex/wake' benchmark:
Run summary [PID 2188093]: blocking on 1 threads (at [private] futex 0x55f9a693c32c), waking up 1 at a time.

[Run 1]: Wokeup 1 of 1 threads in 0.0020 ms
[Run 2]: Wokeup 1 of 1 threads in 0.0020 ms
[Run 3]: Wokeup 1 of 1 threads in 0.0020 ms
[Run 4]: Wokeup 1 of 1 threads in 0.0020 ms
[Run 5]: Wokeup 1 of 1 threads in 0.0030 ms
[Run 6]: Wokeup 1 of 1 threads in 0.0020 ms
[Run 7]: Wokeup 1 of 1 threads in 0.0010 ms
[Run 8]: Wokeup 1 of 1 threads in 0.0020 ms
[Run 9]: Wokeup 1 of 1 threads in 0.0010 ms
[Run 10]: Wokeup 1 of 1 threads in 0.0010 ms
Wokeup 1 of 1 threads in 0.0018 ms (+-11.11%)

the difference is.. dramatic. given that you're looking at an IPI for FUTEX_WAKE, and some number of IPIs in other common operations (we don't advertise invlpgb on Turin so munmap or a page permissions reduction implies an IPI to any core that could have the altered translation!) this can add up in surprising ways. for example, a lock that may be uncontended when the program is run natively is now released ~30x slower, plenty of time to become a point of contention in a VM.

first, we can see where the time in an IPI goes today. from the script I've included at #1233, here are some xAPIC IPI sequences that I'd measured on debug bits:

xapic t=2549993400124254 (vcpu=16) read: 4 bytes at fee00300
xapic t=2549993400130654 (vcpu=16) write: ICR_HI <- 12000000 (4 bytes)
xapic t=2549993400138935 (vcpu=16) write: ICR_LO <- fb (4 bytes)
xapic t=2549993400138935 (vcpu=16) ICR write: vector=251, destination=18 (physical)
    delivery mode 0x0 (Fixed) shorthand=0, status=de-assert, level=Idle, trigger=Edge
xapic t=2549993400175279 (vcpu=18) write: EOI Register <- 4 (0 bytes)
xapic t=2549993400235175 (vcpu=16) read: 4 bytes at fee00300
xapic t=2549993400239936 (vcpu=16) write: ICR_HI <- 12000000 (4 bytes)
xapic t=2549993400245477 (vcpu=16) write: ICR_LO <- fb (4 bytes)
xapic t=2549993400245477 (vcpu=16) ICR write: vector=251, destination=18 (physical)
    delivery mode 0x0 (Fixed) shorthand=0, status=de-assert, level=Idle, trigger=Edge
xapic t=2549993400262158 (vcpu=18) write: EOI Register <- 4 (0 bytes)

first, there's a read of ICR_LO to check that the APIC is not currently sending a different IPI. 6.3 microseconds later the guest writes in a destination to ICR_HI, then another 8.3 microseconds later it's at ICR_LO and we actually get started poking the target. the destination vcpu=18 is running and acknowledges the interrupt another 36 microseconds later, or so. that's around 50 microseconds all-told, so there's a substantial difference between the 20 microseconds above and the 50 here - that is largely probe effect from apic.d, but not totally (repeating now on release bits, for example, shows ~45 microseconds overall). but proportionally I think that's all still about right.

from here, there are several related items to follow up on that contribute in varying ways..

• an illumos topic, filed as #18590: @wfchandler observed elsewhere that if the guest PMU is configured at all, the PMU state save/restore adds several microseconds to exits (I don't think we know if this was Gimlet or Cosmo, but that shouldn't matter). in any case, the above is four exits in total to get an IPI sent and acknowledged. I don't think we expected this to be a remarkable cost at all when we'd integrated it.
• giving guests an x2APIC would mean one exit to program ICR, rather than xAPIC's exit for ICR_HI and a later exit for ICR_LO, which from the transcript above is about 15% of the total IPI time.
• .. conversely, AVIC support would (in the ideal case) avoid these being exits entirely. it should get to near-hardware, but when I'd taken a look at some VMs on The Clouds perf bench futex wake did not look nearly as good as I'd expected. better than above, but rather worse than native.

and with AVIC, IPIs would only be exitless if the destination vCPU is also on-CPU when it's poked. this is the case for some IPI cases (I've run servers bound by the throughput of munmap and TLB shootdowns!) but for cases like FUTEX_WAKE where a destination vCPU is probably idle on a mutex, the source still exits and the OS has to cross-call to schedule the destination. this is actually my suspicion for why the above didn't look as good as it seems.

• we should consider not trapping hlt, and probably allowing monitor/mwait, in vCPUs. this is a bit of a mixed bag, because the operator view will mostly just see "100% busy" VMs across the board - this might be something we do based on the size of VM?

for the most part, the above should actually be issues in stlouis, but at least on x2APIC (and possibly AVIC) I imagine we might end up with VM capabilities that Propolis will have to learn to ioctl.