Memory usage
-
Stephen - Employee
- Posts: 974
- Joined: Sun Dec 24, 2017 8:56 pm
- Has thanked: 77 times
- Been thanked: 170 times
Re: Memory usage
Well, I was hoping the output from ps would tell me something about the switches processes that I didn't already know. Unfortunately, it was not very enlightening - it all checks out normal except of course the current memory load.
Is there anything else you can tell me about the topology of your network? Any unique or unusual devices or configuration that might be present? Are there any non-standard protocols in the works that the switch might be switching? Anything will help because obviously the simulation I built to try and fix this issue did not cover the uniqueness of your network. So I now have to build a better simulator and I need to know more about what is happening. We may have to go as far as me acquiring whatever discoverable devices are on your network.
With discovery turned off, do you still see this issue?
In the mean time, unless I am able to simulate your environment somehow (which is appearing less and less likely) I think the only way to solve this will be to have the process dump it's discovery data in it's entirety once the memory starts to go to this high. I will try to get that on the next release and of course, if that doesn't fix it. Then it means we've been chasing a ghost the whole time and will have to do more troubleshooting to figure out what part of the system in your switches is accumulating memory.
Is there anything else you can tell me about the topology of your network? Any unique or unusual devices or configuration that might be present? Are there any non-standard protocols in the works that the switch might be switching? Anything will help because obviously the simulation I built to try and fix this issue did not cover the uniqueness of your network. So I now have to build a better simulator and I need to know more about what is happening. We may have to go as far as me acquiring whatever discoverable devices are on your network.
With discovery turned off, do you still see this issue?
In the mean time, unless I am able to simulate your environment somehow (which is appearing less and less likely) I think the only way to solve this will be to have the process dump it's discovery data in it's entirety once the memory starts to go to this high. I will try to get that on the next release and of course, if that doesn't fix it. Then it means we've been chasing a ghost the whole time and will have to do more troubleshooting to figure out what part of the system in your switches is accumulating memory.
- Ludvik
- Experienced Member
- Posts: 105
- Joined: Tue Nov 08, 2016 1:50 pm
- Has thanked: 15 times
- Been thanked: 15 times
Re: Memory usage
I don't think our network is somehow non-standard. In this case it is a mikrotik as the main router (uplink is directly on its port), on the second port is netonix switch.
There are 5 Ubiquiti Rocket 5AC-PRISM Gen2 access points (firmware 8.5.8) in the switch and one Siklu 710TX. It's not part of a large L2 domain (the MAC table has 181 records), our network is routed. But every UBNT has its own VLAN, which is then part of the bridge in the mikrotik. DHCP snooping is on.
I do not use STP, MSTP, LAG, ERPS, IGMP snooping.
Client radios are in bridge mode, mostly TP-LINK wifi routers at home. Sometimes mikrotik.
We do not use IPv6 for radio clients. The router has IPv6 enabled, but does not route. So it only has link-local addresses.
Management uses SNMP (I have both versions turned on), once a day ssh.
I tried to turn off the discovery on another piece, it didn't have a significant effect.
A detailed process dump might be a good idea. It just means about a month waiting for the result.
---
Where I watch free RAM, SNMP is used very often.
https://forum.netonix.com/viewtopic.php?f=17&t=5384But on switch where I don't have RAM problem (resp. usage grow slower) has this log file 30MB.
There are 5 Ubiquiti Rocket 5AC-PRISM Gen2 access points (firmware 8.5.8) in the switch and one Siklu 710TX. It's not part of a large L2 domain (the MAC table has 181 records), our network is routed. But every UBNT has its own VLAN, which is then part of the bridge in the mikrotik. DHCP snooping is on.
I do not use STP, MSTP, LAG, ERPS, IGMP snooping.
Client radios are in bridge mode, mostly TP-LINK wifi routers at home. Sometimes mikrotik.
We do not use IPv6 for radio clients. The router has IPv6 enabled, but does not route. So it only has link-local addresses.
Management uses SNMP (I have both versions turned on), once a day ssh.
I tried to turn off the discovery on another piece, it didn't have a significant effect.
A detailed process dump might be a good idea. It just means about a month waiting for the result.
---
Where I watch free RAM, SNMP is used very often.
https://forum.netonix.com/viewtopic.php?f=17&t=5384But on switch where I don't have RAM problem (resp. usage grow slower) has this log file 30MB.
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
Re: Memory usage
Why are you not using v1.5.2 or v1.5.3rcX which helps the developers as your seeing an issue on a thread of code in development
But I still do not see this memory usage issue on any of my switches in my WISP, see below.
CLICK IMAGES BELOW TO VIEW FULL SIZE
But I still do not see this memory usage issue on any of my switches in my WISP, see below.
CLICK IMAGES BELOW TO VIEW FULL SIZE
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
- Ludvik
- Experienced Member
- Posts: 105
- Joined: Tue Nov 08, 2016 1:50 pm
- Has thanked: 15 times
- Been thanked: 15 times
Re: Memory usage
You are lucky man ... i have problem (sometimes smaller, sometimes bigger) with ALL OF NETONIX. But I don't have WS-24/26, only WS-12 and smaller.
Software version doesn't matter! 1.5.2rc6 is same as 1.5.2.
I can't experiment on a network with hundreds of users. You have a software problem. I have the right to buy a functional switch from you. If you're not going to do anything about it, tell it in time. I move to a different supplier to prevent major financial losses. I don't like it anymore.
Please don't write me about your network. I know you don't have problems.
Software version doesn't matter! 1.5.2rc6 is same as 1.5.2.
I can't experiment on a network with hundreds of users. You have a software problem. I have the right to buy a functional switch from you. If you're not going to do anything about it, tell it in time. I move to a different supplier to prevent major financial losses. I don't like it anymore.
Please don't write me about your network. I know you don't have problems.
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
Re: Memory usage
All the switches are the same, same SOC, same basic design, same firmware. I have just about all models running in my WISP and do not see the memory issue. Has to be something different about your setup then most people as there are tens of thousands of these out there running with no issue.
There have been about 4 or 6 people who have run into this memory issue and we have no idea why and can not recreate it.
If you want to assist us then load the latest firmware and make the switch accessible to our programmers (Stephen or Eric) and then maybe they can see what is causing this for you and few other people.
It helps to try and narrow it down.
Disable everything (services) that you can on the switch and see if the issue still occurs.
-SNMP
-NTP
-SMTP
-STP/RSTP/MSTP
-Loop Protection
-Discovery
-Radius
-Everything you can
If the memory issue does not occur then enable services one at a time and try and narrow it down.
There have been about 4 or 6 people who have run into this memory issue and we have no idea why and can not recreate it.
If you want to assist us then load the latest firmware and make the switch accessible to our programmers (Stephen or Eric) and then maybe they can see what is causing this for you and few other people.
It helps to try and narrow it down.
Disable everything (services) that you can on the switch and see if the issue still occurs.
-SNMP
-NTP
-SMTP
-STP/RSTP/MSTP
-Loop Protection
-Discovery
-Radius
-Everything you can
If the memory issue does not occur then enable services one at a time and try and narrow it down.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
Stephen - Employee
- Posts: 974
- Joined: Sun Dec 24, 2017 8:56 pm
- Has thanked: 77 times
- Been thanked: 170 times
Re: Memory usage
Ludvick, I do appreciate your situation and we have been trying to help you.
If you re-call back to this post:
viewtopic.php?f=17&t=5384&p=29171#p29171
We had determined that whatever tool you are using for querying snmpd for an OID that does not exist because of the version of the kernel we must use on our switches.
And I concluded based off the experiments you ran that it was likely the factor we didn't know about before when I rebuilt how our switches handled Discovery protocols. I posted that conclusion on the 30th which was one day after the last report you made on this thread about the continued memory issue.
So I concluded that after that post you had disabled the query to that particular OID with the tool you are using on your network on the switches and thus solved the problem.
Apparently that's not what happened.
Please try disabling the query to OID .1.3.6.1.4.1.2021.4.6.0 to see if that helps.
If it does then we will know we are on finally track with this issue.
If you re-call back to this post:
viewtopic.php?f=17&t=5384&p=29171#p29171
We had determined that whatever tool you are using for querying snmpd for an OID that does not exist because of the version of the kernel we must use on our switches.
And I concluded based off the experiments you ran that it was likely the factor we didn't know about before when I rebuilt how our switches handled Discovery protocols. I posted that conclusion on the 30th which was one day after the last report you made on this thread about the continued memory issue.
So I concluded that after that post you had disabled the query to that particular OID with the tool you are using on your network on the switches and thus solved the problem.
Apparently that's not what happened.
Please try disabling the query to OID .1.3.6.1.4.1.2021.4.6.0 to see if that helps.
If it does then we will know we are on finally track with this issue.
- Ludvik
- Experienced Member
- Posts: 105
- Joined: Tue Nov 08, 2016 1:50 pm
- Has thanked: 15 times
- Been thanked: 15 times
Re: Memory usage
This snmp query (get .1.3.6.1.4.1.2021.4.6.0, not walk!) is for graphing memory usage ... and graphing is as proof for you :-( Memory problem is not based on this.
I started using this OID only after I discovered a memory usage problem.
But you must resolve this :-) Disabling logging, or logrotate, or something else ...
I started using this OID only after I discovered a memory usage problem.
But you must resolve this :-) Disabling logging, or logrotate, or something else ...
-
Stephen - Employee
- Posts: 974
- Joined: Sun Dec 24, 2017 8:56 pm
- Has thanked: 77 times
- Been thanked: 170 times
Re: Memory usage
Ludvick, I have a new test firmware I'd like you to try. You may still use the OID you are using. I have run some test's and I was able to replicate a memory leak and the change's I made seemed to fix it. But obviously, we need to run a long term test on your network to be sure.
May I PM you the firmware?
May I PM you the firmware?
Who is online
Users browsing this forum: No registered users and 75 guests