My company has a HQ office in California and a branch office in North Carolina.

Problem: My NC user's Citrix sessions drop every day at around 8:05. I run PingPlotter on a PC on my LAN. It shows packet loss on the ethernet (inside) interface of my managed AT&T router for typically 30 to 60 seconds whenever the user connections drop. This has been happening for months. I hope the moderator or someone can offer possible reasons why.

More background: Each location has AT&T dedicated T1. Cal and NC have a site-to-site VPN between two SonicWALL firewalls. We run MS Office apps and an order entry app on three Citrix MetaFrame servers at HQ. NC accesses the apps via the VPN using thin client terminals (Wyse Winterms). Every day, NC gets diconnected from their Citrix sessions. This is a time-specific event - usually happening near 8:05 and usually again roughly 15 minutes later. However, disconnects also happen at other times but not on such a predictable basis. Sessions also dropped when we had Sprint T1 in CAL and a BellSouth fractional T1 in NC. So I think the problem is my CAL firewall or something else in my CAL LAN. I capture firewall events to a Syslog. I don't see any events that look like the VPN renegotiating which would be the easiest explanation. I do see a fairly consistent pattern of "Broadcast packet dropped" prior to and after the dropped connections, and also typcially "ICMP packet dropped" around that time. Also occassionaly "ARP timeout" around that time. I also run PingPlotter on a PC outside my firewall and do not see the packet loss.

Any input would be very much appreciated.

PS - not sure how to attach a .png file but will try.

Thanks for the complete description of the situation.

If the red in PingPlotter always correlates back to a dropped session, then you can eliminate a *lot* of your route / hardware as a problem.

Hop 1 is showing packet loss (ie: loss of connectivity) that is also being translated to all downstream hops. This means hop 1 is not replying itself and it's also not passing any packets further down.

This means you should be focusing only on the hardware between the collecting computer and the device that normally responds at hop 1 (inclusive).

I'm not entirely sure the network topology here, but a T1 shouldn't have 95ms latencies, so I suspect that hop 2 is actually the AT&T T1 inside ATT near your CAL office, or maybe it's your sonicwall. That would mean that hop 1 is someplace inside your network - you can probably determine the device based on the IP address.

Once you figure out what the device is, then determine every cable and hardware item between that device and the computer running PingPlotter. It sounds like you're probably *not* running PingPlotter on the Citrix server. Ideally, what you're trying to do here is show two computers with similar symptoms so you can eliminate more hardware without having to swap it. If you're running PingPlotter from your Citrix server, then you should also run PingPlotter from another computer and trace to the same destination. If you have the same results, then you know that the problem lies in the common equipment and/or cabling that both of these computers use.

That common equipment will probably be (I'm not sure, just guessing):

* A router or switch
* A network cable
* The device that reports as 12.145.143.161 (Hop 1)

There might be another "leg" in there - another router/switch and another network cable. If so, try running PingPlotter from a device on a different router / switch and see if you can eliminate other equipment as a problem. Ideally, you'd be able to factor this down and elminiate hardware until you have only 3 items left to physically test - a router/switch, a cable, and the device that reports back as 12.145.143.161.

One you're there, you have to start looking for hardware failure points. The cable may be easy to swap out - you certainly want to do that, if it's easy, but the timing of your failure kind of makes me think it's something a bit "smarter" than a cable.

The only thing left then is to start looking at swapping out devices. The hop 1 device (12.145.143.161) is *certainly* a candidate problem device. Make sure the power plug is well situation. If you have an alternate power supply you can swap out, do so. Try adding a UPS between the wall and the device - just in case your neighbor powering up their "We're OPEN" neon at 8:05 is drawing too much power and causing your device to reboot. We had one customer where their coffee machine was drawing too much current and was causing an unstable router.

This probably gives you enough to go on. It is *NOT* a network component further away from you than the device that reports back at hop 1 - 12.145.143.161. It's something between your computer and that device (or that device itself).

We've seen similar problems on numerous occasions - where there's a bad device and/or power supply, and replacing it fixed the problem.

Please post back here with any additional questions, or if you find the culprit!

- Pete

Thanks for your input. I will do the following:
- check the power situation, and make sure the network equipment is all on a suitable UPS.
- run PP on all three Citrix servers.
- tweak MTU and fragmented packet settings on the firewall, and for each VPN tunnel.
- take a closer look at VPN renegotiating.

Sorry, I forgot to diagram the network...

PC (on LAN) running PP -->
Cisco 1000BT switch -->
SonicWALL Pro 200 (.162) -->
Nortel 100BT switch-->
PC (outside firewall) running PP-->
[.161] AT&T 17xx managed router (at my sitein my building) (.126) -->
[.125] AT&T remote access router (at SBC's site) -->
to North Carolina

BTW, the problem existed before I put the Nortel switch outside my firewall, so I can eliminate that as a culprit.

Will post back if/when I find the problem.