Alright I spent several hours again trying to figure this out.
First off, I pinged the server while I was playing on it.
This revealed a median ping of 85 ms, with some 800 ms outliers (every 20 seconds).
There was no packet loss.
I then decided to install Wireshark and analyze the network traffic further.
Wireshark also reports no packet drop:
but every so often the server would send me a UDP packet that had a bad checksum (see the image below).
Upon further inspection of this one error, it appears to happen a lot.
The first few communications over (my) port 62163 succeed, but then fails every following frame the server sends to me (see the image below).
All in all I don’t suspect packet loss or packet drop is to blame though.
On a side note; due to the ping spikes I suspected MMCSS might be the culprit, but no luck there either.