I'v been having trouble for the last few months with my isp. It was working fine, then I started to notice it slowing down and what seemed to be cutting out, although the modem never shows a disconnect. The isp has been out multiple times, saying they cant see any thing wrong. Last night I started searching around, found pingplotter and read up on it, then decided to give it a try. When my internet isn't under any load everything seems to go ok, but if I try to watch a video on youtube, netflix, or run a speed test, it seems to freeze for a few seconds and then take back off.
Working Ok Picture
Problem Picture

Running pingplotter while watching a video on netflix, I noticed as soon as my buffer would freeze, ( it went from 1 - 75% then stoped at 75% and didn't move for a few seconds) this was the same time packet loss started showing at 100%. After looking closer it seems it's not so much packet loss, but that hop 3 desides to stop communicating with my modem, Im guessing hop 3 would be the DSLAM? I'v tried three modems, two of which are brand new, they all do the same thing. The isp says theres nothing wrong with the line. So, I'm thinking they have a piece of equipment that is acting up randomly causing me to lose connection for a few seconds before it starts responding agian? Any thoughts would be great, Thanks, T.J.

download pingploter files here

Looks like a lost connection between hop 2 and hop 3. I'm not an expert on modem "retrains", but that looks a lot like what's happening here. You're completely losing connectivity between 2 and 3 for about 10 seconds at a time - and then it repeats itself a few times.

This should be "catchable" by your provider - maybe send them a copy of the attached images, and see if they can look at their logs during the periods here (1:53.30am).

You lost connectivity in Netflix. You capture a picture of the problem directly that shows the loss at hop 3, but not hop 2 - and also at the final destination. This is "textbook" - either a bad modem (obviously not, or 3 wouldn't be bad), bad wiring (or wiring related problems), or bad equipment/configuration at hop 3. You don't have a lot of tools to find the problem in more detail, and your provider should be able to help. The problem lies somewhere between the DSL modem and the device inside your provider's complex at hop 3. Lots of territory there, but your provider should also have some decent tools to narrow it down further.

Good luck. Let us know what you find here.

- Pete

Thanks for the quick reply, I'll let you know if I have any luck.

Just a quick update,
So far, we have been switched to a new pair of wires all the way to the office, tried 3 modems, connected to the phone company box out side the house, been switched to another port at the office, had the DSLAM card changed, and was put on a new cirict. We are still having the problems, after days of calling in we finaly got put through to a chronic error support team and spent probaly 8 hours on the phone with them, they said they can see a few problems but cant pin point it from there side. Tomorrow they are going to send a tech out that has more knowledge of the systems, and to sit down and look at the data iv recorded, and maybe try to run pingplotter down at the office, to see if we get the same results.

I have a question, If I go into town and try to trace route back to my computer, can I get a computer inside my network to respond to the trace. From what i understand, when i run and trace using icmp the last hop i can trace is the gateway. I also read that icmp isnt set to a port, so i cant port foward the icmp request to a computer in the network? Could i use tcp trace and forward a tcp port to my computer and get it to respond? Thanks for any help, T.J.

Tracing the other direction is often a useful exercise. Based on the data I saw of yours, though, you're probably not going to learn a lot new about your problem from tracing outside-in. The further from you the congestion appears on the route, the more useful tracing the other way (the "return" route) is. In your case, the problem occurs very close to you. Without trying it, though, you won't know for sure if it exposes some useful information or not, though.

Some routers can forward ICMP requests coming in at their NAT boundary, but usually it's easier to use TCP and forward that (which should be possible).

Even easier - you can sometimes target the internet-facing IP address and get a response from your router / modem / device. You might know this address, but if you don't you can go to someplace like whatismyip.com to find your internet-facing IP. If the latency / congestion appears at that point, then forwarding packets inside your network is not going to add any latency or much insight to the problem, so I'd always suggest trying this first. You might not get a response, though, and then you may need to forward to a device inside your network that will respond.

Just to see if I can get it to trace back, im using my cell phone to see how far I can trace back to my self. It allways stops at se3-g9-1.wotnoh.sbcglobal.net , i can never see my internet ip address in the trace. To get a computer inside the network to respond, would I just foward the ports that the upd or tcp traces are using. Or do i need to configure the compter to respond back some how? Thanks, T.J.

It depends on your computer, really, and the firewall software running on it. The easiest way is to use a TCP server that's already running on your network and forward to that (although that exposes that service to the outside world, so be careful with that too).

Some computers will also respond with a connection denied (without any configuration or service running at all), which we also recognize as a valid response and will show the time for. If you're using PingPlotter Pro, you can even enable its web service, port-forward to that from your firewall, and then target that port on the remote end. This would give you access to the web interface for PingPlotter Pro from outside your network, though.

For best results, you can play around with some ports / computers inside your own network while you're in that network and see what you can target.

ICMP is really easiest because almost everything responds to it - but it's not very forward able. Honestly, I'd not spend a lot of time tracing from the outside in right now.

A bit of advice when your tech shows up. Focus on the root problem (network video freezing up, bad voice over ip, or whatever) - not necessarily the PingPlotter data. The PingPlotter data helps you narrow down the problem space, but many techs are not going to give your data a huge amount of credibility if you start with that. Start with the fact that you're seeing Netflix problems, and tell them when - and *then* move to the PingPlotter data that quantifies the problems you're having. If your tech starts to try to discredit your collected data (PingPlotter), you can then always push back to the Netflix (or whatever) problems that occurred at the same time. If you only focus on the PingPlotter data, without another problem surfacing at the same time, they can relatively easily push back and discount the validity of PingPlotter data. We talk about this "technique" a bit here: http://www.nessoft.com/kb/46.

As long as you start with the root problem (bad network video) you can always fall back to that position if your PingPlotter data is questioned - and once you've re-established that PingPlotter data is just quantifying the network data and showing you that in more detail, you can move back down to the network problems using PingPlotter.

This is why I don't think it's worth *too* much to trace from the outside to the inside - that's not a use case that your technician is likely to spend a lot of time solving. It's not a bad piece of data to have, but you're probably not going to get a lot of value from it in your specific scenario (other scenarios are quite different - and I wouldn't make the same statement generally to everyone).

- Pete

Update for the day,
They sent another tech out today, with the same test equipment they have been using, which for the most part, just checks the line (distance, shorts, signal quality, ext..). They wanted to try another modem again, which after the other 4 not working I figured it was a waste of time, but there the techs I let them give it a try, it didn't help. They also changed some more wiring somewhere down the line, still no change.

I'd say I probably have more knowledge on how the system works than your standard every day internet user. I have been on a computer since I was 5 years old, I have became my family, and friends go to guy for computer help and networking problems, i have probably built 15 computers, set up multiple networks and internet connections. My dad spent 20 years in the Air Force working electronics, radar, jamming ext.., so I learned from a young age how to trouble shoot problems, fix electronic devices, and understand sometime electronic devices just stop working for no logical reason, there are 1000 of reasons why something electronic could malfunction. I even programmed a microcontroller, and interfaced it over Bluetooth to a cell phone. My point being, I'm not going to give up like most people, when I'm told we don't see anything, everything is working fine.

I first noticed this problem because my internet, seemed to take forever to respond at times. I started running speed test, and trying to figure out why, paying close attention to what happens at the times I see a problem. I found using speedtest.net that it didn't seem to be so much a speed issue, it acted like the internet just froze, which you can see on the meter, when it stops moving or the test takes 5x longer than normal to run. This is when I found pingplotter and decided to see if it showed the same thing. Pingplotter shows a problems the same time the speed test site freezes, this tells me something is defiantly wrong, as I can see my trace stop for 10 seconds at a time.

Most of the techs don't want to look at this data, because it's not one of there tools, they don't understand it, and they already think everything is ok, because you have DSL, and the line looks good. It wasn't till after 3 techs being here, and saying everything is ok, did the local manager finally come out, and agree to look at things the way I see it, and give me a chance to explain why I can see a problem. After he finally agreed with me, he called someone high up the chain of command that knows the other side of things (not just wires to my house), he told them to run over to the central office, and look for the same problem there, finally we are getting somewhere!!! The problem was the same connected directly to the central office, so the problem has to be in there networking equipment somewhere.

Here are some things I have learned through my days of dealing with this, and things they have said and how I had to defend why I still believe theres a problem. Hopefully somewhere down the line this helps someone else.

In most of the bigger ISP's there are many levels of support, and techs, trying to find someone that understand the whole system is like finding a needle in a hay stack.

Most of the time the techs at your house only have access to the wire/line side of the network.

Getting someone that can trouble shoot beyond that point takes someone from another department.

The techs keep saying everything is ok, theres nothing more they can do. Keep calling in and reporting the problem, the more times you call, and the more times they come out, the higher up in tech support you get.

I was told it's hard for the techs to get someone to trouble shoot the network side of things, until they have ruled out every other option, if you give up when they say that's all they can do, you might not ever get that far. On the downside be ready to spend the next few days getting to know the tech support guys by there first names and sounds of there voice.

speedtest.net is not something we can accept as a reason why your internet is not working, the freezing and delays could be on there server. True it could, but I have ran 100's of test there, I know how long it takes when it's working the right way. I may believe sometimes the sight is slow, but not test after test. speedtest is testing to one server, I am running pingplotter to google, they both stop at the same point in time. The chances of two random servers on the internet to stop responding at the same times every time, is very highly unlikely. I have my own web site, nothing big just for personal use, it is split between multiple servers, so when something on one stops the others keep the sight up so I have no down time, google is huge millions of people use it every day, the chances of them only having one server for there web site, not likely. So in theory even if one google server is having a problem, it would just route me to another server, and my trace would still go through.

It's the internet we can't control it, that's just the way it works. For me to have problems on more than one web site, not just one, means all of those sites have to be having a problem at the same time. Also as I said above, most big sites are going to have more than one server to split the load, the chances of all of these servers on the internet, all having problems at the same time, is not likely.

This is the best it's going to work. It worked for 3 years just fine, for it to stop now means something changed. I didn't move my house down the road and make the wires longer. I didn't change anything in my house. Which brings up another good point, plug your modem into the box out side your house, connect directly to it, use a network cable and not wireless, maybe try another network cable if your already using one, if you get the same problems here, you can rule out all your house wiring, until it works out side, it wont work inside, this also rules out a bad router or wireless problem ext.., the only thing now are your modem, their wires, and their network, try a new modem, and get the same results, it's some where before your house.

It could be a viris or addware. It could yes, but I have more than one computer, they all do it, the laptop thats running ping plotter was just reformated, and has had anti-viris and firewall on it the whole time, all of my computers have had anti-viris and firewall on them from the time I built them, or got them. Have the tech use his computer and get the same results, if they want you to rule this out.

I may think of more later, hopefully this helps some other people out there.

Do you have a neighbor that uses the same service? If they have similar issues, you can also help eliminate your house and wiring through that method. If they don't have similar issues, you've got someone who can maybe wifi you some decent internet.

Your best bet is to keep trying to escalate. At some point, you get to someone who understands the architecture of the service they're providing. The lower level techs all have great intentions, and many of them are very good - but they just don't have enough knowledge of the possible points of failure to really find the problem. Continuing to push consistently (sometimes for weeks - or even months) will get the problem escalated to someone who can fix it.