Issue has been found! There was a bad connection between my ISP and one of the towers they use. Hoping it will get fixed and the packet loss will subside.