v1.5.1rcX wrote:FIXED/CHANGED
- Fixed voltage input calibration for the WS-8-150-DC - RC1
- Fixed services running on IPv6 - RC2
- Fixed being able to bypass console login by hitting CTRL-C - RC2
- Fixed MSTP setting instance forwarding state - RC2
- Fixed UI on ERPS tab to prevent footer from from being mangled - RC3
- Fixed ERPS MEPs issue that triggered a Signal Fail on a ring when port not in the ring is disconnected/disabled - RC3
- Fixed ERPS issue that prevented access to Management VLAN when added to the protected VLAN list - RC3
- Fixed ERPS protected VLANs would not forward packets outside of the ring - RC3
- Fixed SNMP (Broken in v1.5.1rc3) - RC4
- Fixed Access Controls for a confirmed security vulnerability - RC5 (for more details check this thread: viewtopic.php?f=12&t=4103 )
- Fixed UI bug in Access Control List - RC6
- Fixed DHCP failure when switch has a static ip assigned to a vlan - RC7
- Increased time buffer for warning voltage email trigger - RC7
- MAC Table search works using colons and dashes - RC8
- update for new fans - RC9/RC10/RC11
- fixes for PoE sensor balloon performance issue and data display - RC12
- fix for LACP LAG breaking during config changes - RC12
ENHANCEMENTS
KNOWN ISSUES
- WEB UI issues when not at 100% Zoom on browser especially on VLAN TAB
- Some language templates need help - please private message Eric Stern to help
- IGMP snooping over VLANS, MSTP, and ERPS are still being developed
RC1 Released 6/20/2018
RC2 Released 7/10/2018
RC3 Released 8/2/2018
RC4 Released 8/11/2018
RC5 Released 9/17/2018
RC6 Released 9/19/2018
RC7 Released 10/1/2018
RC8 Released 11/5/2018
RC9 Released 12/23/2018
RC10 Released 12/25/2018
RC11 Released 12/27/2018
RC12 Released 1/17/2019
v1.5.1rcX Bug Reports and Comments
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
v1.5.1rcX Bug Reports and Comments
Please post bug reports and comments on v1.5.1rcX ONLY in this thread.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
- gah789
- Member
- Posts: 6
- Joined: Fri Oct 09, 2015 3:17 am
- Has thanked: 0 time
- Been thanked: 0 time
Re: v1.5.1rcX Bug Reports and Comments
We have run into a serious problem/bug with 1.5.1rc1.
Context: Voltage measurement at a number of off-grid sites was seriously wrong with prospect of partial close down due to voltage < 11V even though the separately measured voltage was > 12.5V. Existing firmware was 1.4.9. Some of the Netonix units were WS-8-150-DC so we installed 1.5.1rc1 at 5 sites to avoid bug with voltage calibration in earlier 1.5.0.
Problems: (1) On 3 occasions after about 90 mins a WS-12-250-DC stopped passing traffic and/or responding to any attempts at access from outside. The problem is apparently caused by Mikrotik SFP modules on port 14 - both copper & optical - which connect to a Mikrotik hEX router. The only way of restoring the Netonix was to power off and restart. (2) For safety we attempted to revert to 1.4.9 on all 5 units. That did not end well either. For 3 out of the 5 units the firmware upgrade/downgrade appeared to crash right at the end of writing the .img file - the units were still powered up but unresponsive. After a long delay, 2 of them restarted but the final one had to be powered down and restarted. Fortunately, the downgrade had been applied.
Overall, a lot of network downtime combined with a lot of worry and wasted time as we had to make 7-8 visits to inacccessible sites (up to an hour's hike) to deal with the sequence of problems. A bug of this nature (??? due to a buffer overflow) combined with ungraceful and failed firmware downgrades is not good for confidence in the hardware. I have no idea why a problem with Mikrotik SFP modules would suddenly turn up when upgrading from 1.4.9 to 1.5.1rc1 but going back to 1.4.9 has clearly demonstrated that the problem is provoked by the most recent firmware release(s).
Context: Voltage measurement at a number of off-grid sites was seriously wrong with prospect of partial close down due to voltage < 11V even though the separately measured voltage was > 12.5V. Existing firmware was 1.4.9. Some of the Netonix units were WS-8-150-DC so we installed 1.5.1rc1 at 5 sites to avoid bug with voltage calibration in earlier 1.5.0.
Problems: (1) On 3 occasions after about 90 mins a WS-12-250-DC stopped passing traffic and/or responding to any attempts at access from outside. The problem is apparently caused by Mikrotik SFP modules on port 14 - both copper & optical - which connect to a Mikrotik hEX router. The only way of restoring the Netonix was to power off and restart. (2) For safety we attempted to revert to 1.4.9 on all 5 units. That did not end well either. For 3 out of the 5 units the firmware upgrade/downgrade appeared to crash right at the end of writing the .img file - the units were still powered up but unresponsive. After a long delay, 2 of them restarted but the final one had to be powered down and restarted. Fortunately, the downgrade had been applied.
Overall, a lot of network downtime combined with a lot of worry and wasted time as we had to make 7-8 visits to inacccessible sites (up to an hour's hike) to deal with the sequence of problems. A bug of this nature (??? due to a buffer overflow) combined with ungraceful and failed firmware downgrades is not good for confidence in the hardware. I have no idea why a problem with Mikrotik SFP modules would suddenly turn up when upgrading from 1.4.9 to 1.5.1rc1 but going back to 1.4.9 has clearly demonstrated that the problem is provoked by the most recent firmware release(s).
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
Re: v1.5.1rcX Bug Reports and Comments
We have tested v1.5.1rc1 extensively and did not see this reported issue?
As well hundreds of people are running this version and this is the first report of an issue so a little help on how to recreate it would be beneficial.
If you could come up with a way to demonstrate this issue.
Take a switch in your office on a bench of each model you are having an issue with.
Factory default it.
Upgrade to to v1.5.1rc1.
Test it out see if everything is fine.
Then install your MT SFP modules and test out again and see if it is the issue.
Find out exactly how we can create your issue.
I feel bad about your experience but we do warn people on good practice when upgrading firmware on the page where you downloaded the firmware.
As well hundreds of people are running this version and this is the first report of an issue so a little help on how to recreate it would be beneficial.
If you could come up with a way to demonstrate this issue.
Take a switch in your office on a bench of each model you are having an issue with.
Factory default it.
Upgrade to to v1.5.1rc1.
Test it out see if everything is fine.
Then install your MT SFP modules and test out again and see if it is the issue.
Find out exactly how we can create your issue.
I feel bad about your experience but we do warn people on good practice when upgrading firmware on the page where you downloaded the firmware.
As always be careful to roll out new versions, upgrade switches that are easily accessible until you verify that the new firmware works with your network design and desired feature set.
We try to test as much as possible but it is impossible to test every possible network configuration.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
- gah789
- Member
- Posts: 6
- Joined: Fri Oct 09, 2015 3:17 am
- Has thanked: 0 time
- Been thanked: 0 time
Re: v1.5.1rcX Bug Reports and Comments
Your response earns Netonix little credit. The notion that a smallish WISP has the time and resources to go through the process that you describe either before installing new firmware or after a problem of this nature has arisen is ridiculous. We can - and do - test new firmware on easily accessible sites. But, in this case, it would not have made the slightest difference. Easily accessible sites are, more or less by definition, on-grid. Our problems were confined to off-grid sites, which have different equipment and are generally more critical because we will only accept the risks of relying on off-grid power are warranted by necessity and locational advantages.
I can't just take core equipment out of service and we don't have spare versions of every Netonix model in use sitting around. The configuration of every unit is different. Following your advice would require me to take 5 core masts out of service and effectively take down our network in order to carry out the bench testing.
A major part of my problem was caused by the ungraceful outcome of the downgrade from 1.5.1rc1 to 1.4.9. This is something that Netonix is much better placed to test than we are.
With respect to the SFP modules, I am aware that they are irritatingly prone to incompatibilities. Mikrotik's cheap & cheerful philosophy is particularly prone to problems of this kind which is why we have tended to prefer their modules for use with their routers. Searching this forum gives limited guidance on what modules Netonix tests. The MT ones that we use are on that list - the S-RJ01 copper module, the S-31DLC20D singlemode single frequency module, and the S-35/53LC20D singlemode pair of bi-directional modules. In this case the problem occurred with all three types on the connection between the WS-12-250-DC and the hEX. I can, of course, send you a copy of the configuration file if you provide an off-forum email address.
You may take the view that it is too difficult for Netonix (as a small company itself) to test and support a wide range of SFP modules. If so, you should be absolutely clear about exactly which ones you test and support. It is no good just saying Cisco. We use Flexoptix as our supplier of compatible modules and they list hundreds (even thousands) of Cisco, Cisco (ex Meraki) & Cisco (ex Linksys) compatible modules. Since it seems likely that many of your customers will use Mikrotik routers, it would be odd for the list of supported & tested modules not to include at least a small number of standard MT SFPs.
In general we don't upgrade Netonix firmware unless it is necessary to deal with a problem. This episode illustrates why, but the prospect of switches going off-line because of incorrect battery voltage reporting left us with limited choice or time. I can accept that the incompatibility between the MT SFP modules and our particular configuration of the WISP switches was rare bad luck. Still, the problems caused by the downgrades is, I believe, a more serious problem that should be addressed by Netonix.
I can't just take core equipment out of service and we don't have spare versions of every Netonix model in use sitting around. The configuration of every unit is different. Following your advice would require me to take 5 core masts out of service and effectively take down our network in order to carry out the bench testing.
A major part of my problem was caused by the ungraceful outcome of the downgrade from 1.5.1rc1 to 1.4.9. This is something that Netonix is much better placed to test than we are.
With respect to the SFP modules, I am aware that they are irritatingly prone to incompatibilities. Mikrotik's cheap & cheerful philosophy is particularly prone to problems of this kind which is why we have tended to prefer their modules for use with their routers. Searching this forum gives limited guidance on what modules Netonix tests. The MT ones that we use are on that list - the S-RJ01 copper module, the S-31DLC20D singlemode single frequency module, and the S-35/53LC20D singlemode pair of bi-directional modules. In this case the problem occurred with all three types on the connection between the WS-12-250-DC and the hEX. I can, of course, send you a copy of the configuration file if you provide an off-forum email address.
You may take the view that it is too difficult for Netonix (as a small company itself) to test and support a wide range of SFP modules. If so, you should be absolutely clear about exactly which ones you test and support. It is no good just saying Cisco. We use Flexoptix as our supplier of compatible modules and they list hundreds (even thousands) of Cisco, Cisco (ex Meraki) & Cisco (ex Linksys) compatible modules. Since it seems likely that many of your customers will use Mikrotik routers, it would be odd for the list of supported & tested modules not to include at least a small number of standard MT SFPs.
In general we don't upgrade Netonix firmware unless it is necessary to deal with a problem. This episode illustrates why, but the prospect of switches going off-line because of incorrect battery voltage reporting left us with limited choice or time. I can accept that the incompatibility between the MT SFP modules and our particular configuration of the WISP switches was rare bad luck. Still, the problems caused by the downgrades is, I believe, a more serious problem that should be addressed by Netonix.
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
Re: v1.5.1rcX Bug Reports and Comments
Our compatibility is not just from us or the switch core. SFP manufacturers do not always follow the standards. Then there are I2C timing issues to get complicated.
SFP incompatibility is not something that only affects Netonix. All equipment that uses SFP modules have compatibility issues.
We also can not test ALL sfp modules on the market as we would have to try and buy every module on the market, not practical.
We "should" work with any module if it follows the Cisco compatibility standards.
We do have a forum where users try and share their results with SFP modules, see here: viewtopic.php?f=6&t=293#p1239
In your original post you referred to having to walk a great distance to get to the site.
I simply said I would try it first on a "single" site that is not a 1 hour "hike" to get to it, you rolled it out to several sites?
Keep in mind there are over 60,000 of these in the field so I would assume more people are not having any issues. Plus I roll it all firmware to my WISP before I release it.
SFP incompatibility is not something that only affects Netonix. All equipment that uses SFP modules have compatibility issues.
We also can not test ALL sfp modules on the market as we would have to try and buy every module on the market, not practical.
We "should" work with any module if it follows the Cisco compatibility standards.
We do have a forum where users try and share their results with SFP modules, see here: viewtopic.php?f=6&t=293#p1239
In your original post you referred to having to walk a great distance to get to the site.
Overall, a lot of network downtime combined with a lot of worry and wasted time as we had to make 7-8 visits to inaccessible sites (up to an hour's hike) to deal with the sequence of problems.
I simply said I would try it first on a "single" site that is not a 1 hour "hike" to get to it, you rolled it out to several sites?
Keep in mind there are over 60,000 of these in the field so I would assume more people are not having any issues. Plus I roll it all firmware to my WISP before I release it.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
- wirelessblue
- Member
- Posts: 34
- Joined: Wed Apr 22, 2015 5:50 pm
- Has thanked: 9 times
- Been thanked: 1 time
Re: v1.5.1rcX Bug Reports and Comments
I am seeing 99-100% CPU usage on a WS-26-400-AC. It is my core switch and I did not notice any issues until I installed 1.5.0. I installed 1.5.1rc1 thinking it would help. I have STP disabled, a number of VLANs, and I am powering an AF-24HD and two EP-R6 units where one is also powering an AF-5X. Nothing is too exciting or out of the ordinary.
I have 16 VLANs now after I deleted about 10 extra ones. It is down to about 96% CPU usage now. I noticed the CPU issue because Netonix Manager can no longer access the switch when the CPU usage is too high. Any idea where the issue might be? Thanks again for the great products.
Also, saving the config seems to use 100% CPU and sometimes traffic stops for a few seconds. A reboot does solve the CPU problem but obviously I can't reboot easily.
I have 16 VLANs now after I deleted about 10 extra ones. It is down to about 96% CPU usage now. I noticed the CPU issue because Netonix Manager can no longer access the switch when the CPU usage is too high. Any idea where the issue might be? Thanks again for the great products.
Also, saving the config seems to use 100% CPU and sometimes traffic stops for a few seconds. A reboot does solve the CPU problem but obviously I can't reboot easily.
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
Re: v1.5.1rcX Bug Reports and Comments
wirelessblue wrote:I am seeing 99-100% CPU usage on a WS-26-400-AC. It is my core switch and I did not notice any issues until I installed 1.5.0. I installed 1.5.1rc1 thinking it would help. I have STP disabled, a number of VLANs, and I am powering an AF-24HD and two EP-R6 units where one is also powering an AF-5X. Nothing is too exciting or out of the ordinary.
I have 16 VLANs now after I deleted about 10 extra ones. It is down to about 96% CPU usage now. I noticed the CPU issue because Netonix Manager can no longer access the switch when the CPU usage is too high. Any idea where the issue might be? Thanks again for the great products.
Also, saving the config seems to use 100% CPU and sometimes traffic stops for a few seconds. A reboot does solve the CPU problem but obviously I can't reboot easily.
Do you by chance have more than 1 Manager monitoring the switch, if so THAT IS BAD and can cause this.
If you are monitoring it with SNMP do not poll it every second, the more you poll it the more CPU it uses. Every 30 to 60 seconds is OK.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
- wirelessblue
- Member
- Posts: 34
- Joined: Wed Apr 22, 2015 5:50 pm
- Has thanked: 9 times
- Been thanked: 1 time
Re: v1.5.1rcX Bug Reports and Comments
Only a single Manager instance. Also, SNMP is disabled entirely.
-
sirhc - Employee
- Posts: 7347
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1597 times
- Been thanked: 1318 times
Re: v1.5.1rcX Bug Reports and Comments
wirelessblue wrote:Only a single Manager instance. Also, SNMP is disabled entirely.
No clue?
I do know that more than 1 Manager monitoring a switch will cause this, beyond that I am not there so I am going to throw out some guesses???
Misconfigured SMTP or other service, try disabling services on the Device/Configuration TAB like SMTP, NTP and such?
Possibly a corrupted configuration file, factory default it and manually set it back up?
If you have a public accessible IP someone could be trying to hack it, use the Access Control List to limit who can attempt to gain access to the switch?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
- wirelessblue
- Member
- Posts: 34
- Joined: Wed Apr 22, 2015 5:50 pm
- Has thanked: 9 times
- Been thanked: 1 time
Re: v1.5.1rcX Bug Reports and Comments
sirhc wrote:No clue?
I do know that more than 1 Manager monitoring a switch will cause this, beyond that I am not there so I am going to throw out some guesses???
Misconfigured SMTP or other service, try disabling services on the Device/Configuration TAB like SMTP, NTP and such?
Possibly a corrupted configuration file, factory default it and manually set it back up?
If you have a public accessible IP someone could be trying to hack it, use the Access Control List to limit who can attempt to gain access to the switch?
I only have a single Manager instance but perhaps something is problematic there since my Manager instance is behind an OSPF link that has been a little troublesome lately.
I have no SMTP or any other service except for NTP enabled. It is also on a private and secured management network without outside access.
I could factory default it and set everything up again if I have to in the middle of the night. I will reboot it again over the weekend and make sure my Manager instance is not problematic. Thank you.
Who is online
Users browsing this forum: No registered users and 79 guests