There are several possibilities for why this could be happening. Before we start talking about them, I just want to reaffirm that the final hop's latency is the latency to get from you to that site, and then back again. The final hop is the *most* important number - as all of the intermediate data is just helpful information that may be useful for troubleshooting.

1) Intermediate hops may be down-prioritizing ICMP echo requests or replies. This is pretty normal - but not to the scale that you're seeing. Normally, this will demonstrate itself in one or two (or several) routers that jump up in latency occasionally, but not constantly - and usually no more than 2X the latency of the final hop (this is a general observation - not a rule - sometimes it may drop packets completely, or add a lot of latency).

2) The intermediate hops are using a different return path. This is more likely - that the packets in the middle are coming back via a different route than the packets at the end. Unfortunately, there's not any good way of figuring out what route this is - unless these routers are under your control to the degree that you can run a traceroute from those routers back to yourself. We recently had a bit of a discussion about this that may help explain how "asynchronous routing" works.

There are some other more remote possibilities, but option 2 above is the most likely culprit on what results you are seeing. This becomes even more convincing if the routers that are reporting bad latency are all inside a specific network provider's boundries - and the hops that are reporting normal latencies are outside these boundries. If that's the case, then the return routing rules for that provider probably are going through a non-optimal link.