That's not pretty!
You've got a couple of instances of possible problems here, but some more clear than others.
First, it looks like you have an internal router of some kind at hop 1 (ARIN thinks you own that IP). No packet loss - that's good.
Hop 2 is probably your border router to Eschelon Telecommunications - probably inside their complex someplace. The bit of packet loss here is slightly disturbing. I can't really see when that happened, but hop 4 is getting sparatic packet loss across the whole time in the picture, so hop 2 is probably seeing something similar. In fact, hop 12 is seeing similar patterns as well - some lost packets all over.
The inconsistent latency jumps (example: 3:35pm, 11:15am, 2:05pm) look to be bandwidth limitations at hop 4 or the hop 3->4 link (I'm guessing since I can't see the time graph for hop 3). That's pretty normal - you occasionally use your bandwidth, and when it's saturated, latency increases a bit. You'd *feel* some of this, but it shouldn't cause your email to stall.
Now, as you probably expect, the major issue starts at hop 8. Both hops 7 and hop 8 are owned by Alter.net. It's odd that it would be so absolutely obvious - this is probably a pretty core backbone for them, and problems like this would certainly be noticed.
This *COULD* be a return route problem. Because the hop 7 -> hop 8 is a ALTER.NET backbone (and problems of this magnitude and duration are a lot more likely at borders than they are inside the core of an established network backbone provider - I would expect them to see this and not allow it to happen), I would suspect that this is actually a problem with the return route. If, once your packets are in Atlanta, Alter.NET decides that there's another network that can route your packets back more effectively than they can, then we might see something like this.
The only way to know this for sure is actually to trace from the remote server back to you and see how the packets flow. Ideally, you'd be able to run PingPlotter from that network, but any traceroute would be able to tell you about the network that is being used. Especially interesting would be to do this during a period where you're seeing problems (ie: trace from the remote side back to you).
If I were you, I would start out by calling Eschelon (I assume you're buying your bandwidth from Eschelon) and ask them about this huge packet loss problem at hop 8. They can then contact Alter.net (or any other peers that may be routing data - in case the return route is different) to find out what's going on (or they may already know what's going on).
When you're seeing big periods of packet loss in PingPlotter, you might try opening a browser to that same target. There appears to be a web server running there with a relatively light-weight web page. This would be a good test to make sure the PingPlotter results are correllated with real-world problems (like inability to connect with POP and/or timing out web pages). It's always important to verify PingPlotter results against real-world impacts. If you see 80% packet loss but everything's working snappy and fast, then the network may be routing ICMP packets different than TCP packets - we need to make sure we're chasing the right problem.
The periods of red show a a few packets through, which means you might get the page to load, but it would be horribly slow. If that's the results you see, then you know the red in PingPlotter does actually correlate to network problems and you're pretty certain where the problem is: the Seattle -> Atlanta link inside Alter.net. The latency jump here is normal (that's quite a few miles), but the packet loss is not.
So, my summary:
* It looks like the major problem is the hop 7 -> hop 8 link.
* Verify this by loading the web page at the target IP during times with low packet loss and high packet loss - make sure there is a noticeable difference.
* If you can, trace from the remote side back to you and see what networks are participating in returning data to you.
* If you can confirm the problem as shown in PingPlotter is really what's happening, report this to your ISP, along with your collected PingPlotter data / graphs, and ask them what's going on.
That's the way I'd go after this.
Good luck! Let us know what you find out. Also, if you find out any additional information (like the return route) you want us to comment on, post back here.
- Pete
Previous Topic
Index