Where is the additional latency coming from?

I'm playing Half-Life: Alyx via Virtual Desktop set to 90fps, and at first it was good, though I had to switch to H.264+ @200mbps to keep the latency in the green, but then suddenly the framerate started tanking again. Adding up all the individual latencies (game, networking, encoding, decoding) I get about 20ms, but the overall latency is around 35. Where could the additional 15ms of latency be coming from?

On a separate but related note, every so often, sometimes every few minutes, sometimes rarer, I get **really big** framerate tanking for maybe 30 seconds, with both the Game and Networking latencies going through the roof. One time I quickly launced Task Manager and saw some Windows User Experience or something similar task taking up some CPU (though less than 10%). Is there any way to tell Windows to stop launching random tasks when I'm in a VR session?