Topic Options
#3655 - 04/27/20 02:34 PM Performance, multi-threading, reliability issues
angelclaw Offline


Registered: 03/03/19
Posts: 4
Currently tracing ~ 300 targets with auto-save enabled and 1 second interval, and found some differences between engine options, using an i7-2700 4core/8threads cpu with SSD and Windows 10:

- ICMP.dll option is limited badly, I max at most 13-15% cpu usage and instead of getting close to 3600 samples per hour I only get ~ 1500-1600. It's slow like hell. Adjusting packet send delay doesn't work, stuck in 'Auto'.
- raw ICMP option is fast, I get 50% cpu usage on average, 3500+ samples per hour on all targets, also stuck with packet send delay in 'auto'. But it has a design flaw. Routes changes don't update in realtime and most of times the full trace is a mess which doesn't match reality (mixes old routes with new ones), it gets fixed by restarting PingPlotter or resetting targets.
- UDP mode (mtr style engine) is nowhere near MTR accuracy, 70% of all internet target have 100% loss on destination (only the destination), while the intermediary hops behave normally. Is there any way to get MTR behaviour implemented with Pingplotter so we can have a reliable and fast way to use Pingplotter? Does this mode need a special build of winpcap/npcap to behave normally?

Also, a maintenance option which restarts pingplotter is needed, after 1 day of running the process goes from ~500MB RAM usage to 2GB+, it's a serious bummer, sometimes my machine with 8GB of ram goes into serious swapfile usage when pingplotter eats 6GB+ of RAM.

May I ask, is there any software engineer that runs the latest build using hundred of targets and test for reliability and the spiking memory leaks?
All of the 3 traceroute engine options has it's own design flaw which needs to be addresed, some machines have 16cores on a desktop CPU and can't make use of proper resource utilization and reliability on Windows PCs.


Edited by angelclaw (04/27/20 02:58 PM)

Top
#3656 - 04/27/20 06:07 PM Re: Performance, multi-threading, reliability issues [Re: angelclaw]
Poe Offline
Pingman Tools Support


Registered: 02/11/19
Posts: 77
Hi angelclaw,

That for writing in and passing on this great information!

We want to make sure we provide the best answer to this that we can, so we are going to take another day to research and come up with a more in-depth answer for you.

Thanks for your patience!

-Poe

Top
#3657 - 04/28/20 07:35 PM Re: Performance, multi-threading, reliability issues [Re: Poe]
Poe Offline
Pingman Tools Support


Registered: 02/11/19
Posts: 77
Hi angelclaw,

Thanks for your patience with this, I'll follow up with tomorrow morning with an in-depth response.

Thanks,

-Poe

Top
#3659 - 04/29/20 07:23 PM Re: Performance, multi-threading, reliability issues [Re: Poe]
Poe Offline
Pingman Tools Support


Registered: 02/11/19
Posts: 77
Hi angelclaw,

I want to start out by apologizing that this has taken us so long to respond to but this was a pretty good list of issues for our QA team to investigate.

First I want to make sure that we have the issues that you reported straight. Here is a list of what we took away from your post:

1. Memory leak
"After 1 day of running the process goes from ~500MB RAM usage to 2GB+"
2. A bug prevents the user from changing packet delay from 'Auto'.
This is a real bug that we need to address but in your case, it seems you need to keep this on Auto to achieve the best results
3. ICMP.dll has bad performance with 300 targets at a 1-second interval
"ICMP.dll option is limited badly, I max at most 13-15% CPU usage and instead of getting close to 3600 samples per hour I only get ~ 1500-1600. It's slow like hell."
4. ICMP using raw sockets has route issues
"Routes changes don't update in realtime and most of the times the full trace is a mess which doesn't match reality (mixes old routes with new ones), it gets fixed by restarting PingPlotter or resetting targets"
5. UDP packets are not received as well as they should be. Using MTR provides more accurate results
"UDP mode (mtr style engine) is nowhere near MTR accuracy, 70% of all internet target have 100% loss on destination (only the destination), while the intermediary hops behave normally."

We have tickets logged for each of these issues but I wanted to respond to each of the issues above.

1. We appreciate the details provided regarding the memory leak, they helped us reproduce the issue in-house and we are currently working on tracking down the source.
2. Again, thank you for reporting the packet delay not being able to switch off of ‘Auto’! This is definitely a bug and we have a ticket logged to address it in an upcoming release.
3. As far as the issues with ICMP.dll and ICMP using raw sockets, we have tickets logged for these issues now and will be looking into them shortly. Once more, thank you for providing so many details, it makes getting to the bottom of these problems much simpler.
4. The issues with UDP you mentioned have been a hot topic in the (currently virtual) office the past couple of days. We’ve been running MTR alongside PingPlotter and have noticed some interesting behavior. We were hoping to have some more solid answers for you about it before responding, but for now here are a couple of things you might try with UDP:
A. Change the UDP port settings on the Engine settings page. We found better results by expanding the port range to 30000-40000. However, we encourage you to try and play around with those to see if certain port settings work better on your machine. We recommend trying anything between 30000 and 65535.
B. Another thing we noticed is that running MTR alongside PingPlotter seems to cause both programs to have varying levels of reliability. In other words, running MTR and PingPlotter simultaneously seems to cause both to report worse results at varying times than they would if run alone.

Thanks for hanging with me through this long-winded ticket, but we want to make sure we have addressed everything to the best of our ability! There are a few things you can currently try that might help out:

1. If you are not already on the latest version of PingPlotter (5.15.7), we recommend that you update as there have been numerous enhancements around resource usage recently. The latest version can be found here: https://pingplotter.com/download
2. If you are not running as a service, we recommend you install PingPlotter with that option enabled as users generally have better performance as a service. In that same vein -- as a temporary measure -- you can set up a Window Task Schedule to automatically restart the service for you whenever you like (presumably overnight) to fix the resource usage growth. We have a knowledge base article on how to accomplish this here http://www.pingman.com/kb/60
3. Last thing I promise — It would be most helpful if you could send in a support ticket (Help -> Email PingPlotter Support). This will help us dive into these issues further to be able to address them more thoroughly.

Let us know if you have any questions and we will be following up with you directly from your support request.

Thanks,

-Poe

Top
#3662 - 04/30/20 06:35 AM Re: Performance, multi-threading, reliability issues [Re: Poe]
angelclaw Offline


Registered: 03/03/19
Posts: 4
Hi Poe, thanks for taking your time acknowledging my complaints smile

Originally Posted By: Poe

1. We appreciate the details provided regarding the memory leak, they helped us reproduce the issue in-house and we are currently working on tracking down the source.


Glad to hear that. I see a potential downgrade from the 4.x builds where it was possible to only use memory and overwrite old samples once the samples threshold is reached. The full disk dump while tracing a lot of targets seems to take a toll on the memory usage.

Originally Posted By: Poe

2. Again, thank you for reporting the packet delay not being able to switch off of ‘Auto’! This is definitely a bug and we have a ticket logged to address it in an upcoming release.
3. As far as the issues with ICMP.dll and ICMP using raw sockets, we have tickets logged for these issues now and will be looking into them shortly. Once more, thank you for providing so many details, it makes getting to the bottom of these problems much simpler.


Great to hear that.

Originally Posted By: Poe

4. The issues with UDP you mentioned have been a hot topic in the (currently virtual) office the past couple of days. We’ve been running MTR alongside PingPlotter and have noticed some interesting behavior. We were hoping to have some more solid answers for you about it before responding, but for now here are a couple of things you might try with UDP:
A. Change the UDP port settings on the Engine settings page. We found better results by expanding the port range to 30000-40000. However, we encourage you to try and play around with those to see if certain port settings work better on your machine. We recommend trying anything between 30000 and 65535.
B. Another thing we noticed is that running MTR alongside PingPlotter seems to cause both programs to have varying levels of reliability. In other words, running MTR and PingPlotter simultaneously seems to cause both to report worse results at varying times than they would if run alone.


I don't run MTR alongside Pingplotter, I just want the source code of MTR inspected and make the UDP engine mirror exactly the MTR behaviour of handling ICMP port unreachable packets (type 3, code 3) for proper consistency.
I've already expanded the UDP port range by a lot, even 33000-65535 but it's unreliable nonetheless.
The code is opensource, some links:
https://github.com/traviscross/mtr - for the UNIX mtr
https://github.com/White-Tiger/WinMTR/releases - for the WinMTR (both original appnor version and the redux fork edition)


Originally Posted By: Poe

1. If you are not already on the latest version of PingPlotter (5.15.7), we recommend that you update as there have been numerous enhancements around resource usage recently. The latest version can be found here: https://pingplotter.com/download
2. If you are not running as a service, we recommend you install PingPlotter with that option enabled as users generally have better performance as a service. In that same vein -- as a temporary measure -- you can set up a Window Task Schedule to automatically restart the service for you whenever you like (presumably overnight) to fix the resource usage growth. We have a knowledge base article on how to accomplish this here http://www.pingman.com/kb/60
3. Last thing I promise — It would be most helpful if you could send in a support ticket (Help -> Email PingPlotter Support). This will help us dive into these issues further to be able to address them more thoroughly.


I'm on the latest version.
The automatic restart of the service is a great idea, will make use of that.
Also, in service mode the performance is greatly improved with icmp.dll engine mode, it's almost as fast as using raw icmp mode. Thanks for the tip!
I will also make use of support tickets once I spot abnormal behaviours.
Thanks for your time.


Edited by angelclaw (04/30/20 07:25 AM)

Top

Search

Who's Online
0 registered (), 22 Guests and 0 Spiders online.
Key: Admin, Global Mod, Mod