I've been trying to troubleshoot problems with some of our VPN's the last few days and have been using ping plotter to try and spot the problem. Users are reporting slow speeds and failures to connect. Our remote sites are using watchguard soho 6 boxes. When pinging the cable modem everthing is fine... when I ping the firebox I get timeouts every 20 minutes and anywhere from 10% to 40% packetloss. We have 3 VPN's using the SOHO 6 and all 3 are doing the same thing.

Anyone have any ideas on what the problem could be??

The most likely culprit sounds like something between the cable modem and the Firebox. Maybe it's an issue with the Firebox dealing with ICMP, or maybe it's a real problem.

It sounds like you're already pinging through the tunnel and around the tunnel (as outlined here: http://www.nessoft.com/kb/28).

I'd start out by trying to correlate the packet loss you see in PingPlotter with your VPN problems. If you're getting failure to connects at just the same time that PingPlotter is showing packet loss, but the VPN performance is OK when PingPlotter is showing no packet loss, then this would be a strong correlation, and you know it's not just an oddity with the way the Firebox is handling PingPlotter data.

VPNs are notoriously poor at passing PingPlotter data (and ICMP / traceroute data), so you really need to make sure the problem you're chasing is a real one, not just a red herring because of limitations in the VPN stack.

If you're having some problems correlating PingPlotter data with your user's problem, try to Ping the final destination only (in PingPlotter, View -> Ignore First Hops -> Ping Final Hop Only) because this can be more reliable with some VPNs.

Once you've correlated the PingPlotter data with the VPN problems, I'd get in touch with Watchguard and see if they have some idea of what's causing this problem. Maybe it's a configuration parameter where the bandwidth is being artificially constrained through the VPN, even though the bandwidth through the cable modem is not saturated, or maybe there is some packet priority setting that's causing packet loss.

Good luck!

- Pete

Thanks for the Reply! It turns out that only one of these remote sites has a VPN tunnel. I thought the other two did but they don't... so I don't think the VPN is the problem. Here are some screenshots of pingplotter in action. The first set is the one with VPN. I ran pingplotter against the inside and outside IPs. The second set is one of the other troubled sites. I ran PP against the outside address of modem and firewall. It seems the firewall is where the trouble is. Interesting how it times out almost every 20 minutes. I've been through my firewall configs and can't find anything that's set to 20 minutes.

here is an image from the inside IP of the firewall. The second image is from the outside address.





This second set is from the site without the VPN. First image is the outside of the firewall. Second is from the modem. This one timesout every 20 mins

Are these targets really all only 1 hop from the destination, or do you have PingPlotter set up to ping the final hop only? If there are any intermediate routers involved, it would help to be able to see the data for all of those hops.

- Pete

There are other hops involved but I wasn't seeing any problems there so I set it to ping final hop only.

Would it be helpful to setup ping plotter at one of the sites and run it on an outside address??

I'm used to having intermediate hop information, so if it's not there, I'm left with less information than normal. <img src="/forums/images/graemlins/smile.gif" alt="" /> Pinging the final hop only is useful when a software or hardware routing system isn't able to send the full route information correctly, but if it *is* working correctly, then it's better to do that.

If you're seeing packet loss only show up at the final destination, then tracing from that final destination back to the original site can be quite useful.

And yes - going outside the tunnel is a good thing to do - going to an outside site. If you can get the packet loss to show up near the beginning of the route, it's usually helpful. Go to the site where the packet loss is showing up, trace out, and see if you can duplicate the packet loss. If so, then start messing with devices (try going around the firewall, for example) until the problem stops. If the problem shows up inside your ISP (rather than your own equipment), then you can contact your ISP for help. If you can't route around the hardware firewall, then you should probably get in touch with the firewall vendor for help.

- Pete

Hi Pete,

I collected some more data from running pingplotter at one of the sites. The modem is dropping out more then the firewall... this is the opposite of what I saw when pinging from the outside...I pinged the modem, firewall, and google. Here's the results- I attached the sample sets as well

modem




google



firewall

Hello.

Statistically speaking, your ISP is the problem. 90% of the time, problems like this are the ISP. The other 10% of the time, though, it's something else - so you can't say *for sure* it's an ISP issue.

Also, based on your description, pinging from the outside shows something inside your network as the problem, and pinging from the inside to the outside shows something *outside* your network as the problem. This indicates it's the border between your network and outside - your ISP again.

On your google trace there, it looks like you focused just on 10 lost samples - and during that period, none of the downstream hops responded either. That indicates a problem with the connection between your network and outside (your ISP again).

Looks like an ISP problem, although you'd be better served by showing a whole route - looking at a few hundred samples, and then turning on the time graph for hop 1, hop 2 and for the final destination to show that the packet loss from the final destination is being introduced at hop 2 (the ISP).

- Pete

ISP came out on Friday and said they found a problem at the drop coming into the building. I'm still puzzled at why we would have similar problems at 3 other locations...

Thanks for all your help!