v1.5.22 Bug Reports and Comments

[oeyre]

Also, did any of your other VLANs that weren't exhibiting the problem have IP's assigned too them?

One more question to assist with one of my own working theories, is DHCP Snooping enabled on any of your ports?

To the best of my knowledge this is the first time we've added an extra IP to a VLAN and have only been using the main management VLAN/IP settings up to now. I later removed the extra IP and things kept working.

We don't use DHCP and none of the ports in question have DHCP snooping enabled. Some have isolation enabled if that means anything.

Edit: we don't use LACP either, and this issue seems to only have manifested on a unit with a SFP (specifically on the SFP port).

Edit 2: has the cause of the SNMP counter lockup issue been identified yet? We're seeing approx 10% of our units develop this problem either on bootup or sometime thereafter.

[sirhc]

Edit 2: has the cause of the SNMP counter lockup issue been identified yet? We're seeing approx 10% of our units develop this problem either on bootup or sometime thereafter.

How many SNMP servers do you have querying the switch, the one guy had two servers each polling I think originally 30 seconds then backed down to 1 minute each.

If 1 server how fast are you polling the switch? 1 minute is the absolute fastest we recommend.

[oeyre]

Unfortunately we've also noticed that OSPF is also not working on through this same switch. Regular ping works fine, however side A router does not see any adjacency for BFD/OSPF. Side B router has adjacency for BFD/OSPF stuck in incomplete state. BFD counters indicate Tx packets but no Rx packets. VLAN is tagged both sides.

Side A ---- (p13) Netonix (p1) ---- Side B

Have tried adding & removing IP from VLAN, enabling & disabling DHCP snooping on the port, deleting and adding the VLAN. All to no success. Ports are not LACP.

Edit: also tried disabling and enabling multicast on the SFP port.

Anyone got any tips?

How many SNMP servers do you have querying the switch, the one guy had two servers each polling I think originally 30 seconds then backed down to 1 minute each.

If 1 server how fast are you polling the switch? 1 minute is the absolute fastest we recommend.

2 different stations, should be once per 5 mins but I'll need to confirm

[Dawizman]

We use PRTG to monitor our infrastructure. This switch is being polled every 30 seconds (as is every one of the 200+ other netonix on our network). It seems other OIDs have stopped updating as well, since the same time yesterday (I simply didn't notice). Power and temperature related OIDs are not updating, so far only system uptime seems to still be responsive.

A list of other OID's we monitor, all of which are reporting the same values since the interface counters stopped working;
1.3.6.1.4.1.46242.3.1.3.2
1.3.6.1.4.1.46242.3.1.3.1
1.3.6.1.4.1.46242.3.1.3.3
1.3.6.1.4.1.46242.3.1.3.4
1.3.6.1.4.1.46242.3.1.3.5
1.3.6.1.4.1.46242.3.1.3.6
1.3.6.1.4.1.46242.3.1.3.7
1.3.6.1.4.1.46242.2.1.2.1
1.3.6.1.4.1.46242.4.1.3.4
1.3.6.1.4.1.46242.4.1.3.5
1.3.6.1.4.1.46242.7.0
1.3.6.1.4.1.46242.8
1.3.6.1.4.1.46242.4.1.3.1
1.3.6.1.4.1.46242.4.1.3.2
1.3.6.1.4.1.46242.4.1.3.3
1.3.6.1.4.1.46242.6.0

For the time being I would try to disable and re-enable snmp on that switch, if it still isn't starting back up after that you may need to reboot the switch to get it back.

However, we recommend avoiding frequent SNMP queries, as this has historically led to issues. Usually, we say that one query a minute should be considered maximum rate.

On our end I have a test that is scheduled to continue for 5 days that is querying these OID's. We will see if that can replicate what you're seeing.

In the mean time, if you could reset snmp on the switch and slow down the queries a bit - just watch and let us know if it happens again.

I misspoke previously. We ping the switches at 30 second intervals, but SNMP queries are 180 seconds.

I have restarted SNMP on this switch, and will advise if it stalls again.

[yahel]

Apologies for being slow on this - long day.

I didn't got a chance to speak with Vivek about this, but I have read his report.
I can't say with 100% certainty, but it seems like disabling the watchdog IPs did not solve the problem.

Also, on the same switch, the problem started also on the 2nd SFP port, which has a VERY different SFP-module (some junk SFP-to-RJ45 thingy, while as the first SFP port that gave us the problem, has a real fiber).

To clarify - the problem we are seeing is with Multicasts (OSPF-multicasts in our case), not passing via these ports (but sometimes does, or at least did).
We can't find a way to make it work, but at least one of the SFP ports did work on that switch after upgrade to 1.5.22 and now stopped.

We do not use DHCP snooping on any port.
We have Multicast checked on all ports.

Next, I think I'll try to test if Multicast work on these SFP ports when they are not trunks.
Anything else to test?
This is a production switch that is giving us this hard time (we are holding off the upgrades on all other switches that uses the SFP ports) - so things are risky and take time to test...

Yahel.

[Stephen]

For the time being just hang tight.

We have a new release coming out sometime later today.

[KBrownConsulting]

There is still bug in SMTP but different one... curretnly it uses field "From Address" in "helo" command and the reply from SMTP is then

[<-] 501 5.5.4 Invalid domain name

...because it expects some FQDN domain, not an "email" format. Previously you ignored this field and used name "localhost" and then the same as "Mail From:" command and that was a bug. So the solution is to use "helo" only and nothing else (or the Local Host Name field) and "Form Address" field use in "Mail From:" command only. Thanks for fix.

I too am having problems with email.

Worked fine in 1.5.17rc2 but is broken in 1.5.22.

Below is the error I get if I fill in the "From Address" and run the SMTP test from the Configuration page:

[<-] 220 <redacted mail server> ESMTP [c]
[->] EHLO <redacted my email account>
[<-] 501 5.0.0 Invalid domain name
SSL_connect: Success
ssmtp: Cannot open <redacted mail server>


If I leave the "From Address" blank I get this error:

[<-] 220 ESMTP [c]
[->] EHLO localhost
[<-] 250 HELP
[->] STARTTLS
[<-] 220 2.0.0 Ready to start TLS
[->] EHLO localhost
[<-] 250 HELP
[->] AUTH LOGIN
[<-] 334
[->]
[<-] 334
[<-] 235 2.0.0 OK Authenticated
[->] MAIL FROM:<admin@localhost>
[<-] 553 5.5.4 <admin@localhost>... Real domain name required for sender address
ssmtp: 553 5.5.4 <admin@localhost>... Real domain name required for sender address


I have tried putting a bunch of different things in the "From Address" and the "Local Hostname" option boxes but so far have been unable to find anything that works.


Same behavior on 2 different model switches: WS-8-150-AC & WS-24-400A

If I downgrade back to 1.5.17rc2 it works again. (Have upgraded & downgraded several times to test. Also fully defaulted switch after one of the upgrades & manually reconfigured just to make sure config wasn't corrupted and it still didn't work.)

Hoping this can be looked into and addressed as SMTP notification is completely broken for me in 1.5.22

Another related issue: If you put the wrong port into the SMTP settings and run the SMTP test, the switch GUI becomes completely unresponsive for a long time. After about 2min & 30sec you get the "Attempting to reestablish communication with the device ..." banner and then shortly after that it reconnects and you get a "SMTP Test Results" error that claims the test "Timed out after 60 seconds", even though in reality it took several minutes. And when I say the GUI becomes completely unresponsive, I mean completely to the degree that if you open a separate Private/Incognito browser and try to log into the switch it shows you the Login page but won't actually load anything until the test has completed.

[nextit]

do you have any updates about the pppoe/vlan trunk problem?

[Stephen]

do you have any updates about the pppoe/vlan trunk problem?

The firmware coming out today has a fix in it that we think may be responsible. But testing this locally has proven difficult so we are relying on the community for feedback. However, we were able to confirm a bug that was related to multicast that in theory could be the cause. We'll be relying on everyone to let us know if the behavior improves after it's out.

[Stephen]

There is still bug in SMTP but different one... curretnly it uses field "From Address" in "helo" command and the reply from SMTP is then

[<-] 501 5.5.4 Invalid domain name

...because it expects some FQDN domain, not an "email" format. Previously you ignored this field and used name "localhost" and then the same as "Mail From:" command and that was a bug. So the solution is to use "helo" only and nothing else (or the Local Host Name field) and "Form Address" field use in "Mail From:" command only. Thanks for fix.

I too am having problems with email.

Worked fine in 1.5.17rc2 but is broken in 1.5.22.

Below is the error I get if I fill in the "From Address" and run the SMTP test from the Configuration page:

[<-] 220 <redacted mail server> ESMTP [c]
[->] EHLO <redacted my email account>
[<-] 501 5.0.0 Invalid domain name
SSL_connect: Success
ssmtp: Cannot open <redacted mail server>


If I leave the "From Address" blank I get this error:

[<-] 220 ESMTP [c]
[->] EHLO localhost
[<-] 250 HELP
[->] STARTTLS
[<-] 220 2.0.0 Ready to start TLS
[->] EHLO localhost
[<-] 250 HELP
[->] AUTH LOGIN
[<-] 334
[->]
[<-] 334
[<-] 235 2.0.0 OK Authenticated
[->] MAIL FROM:<admin@localhost>
[<-] 553 5.5.4 <admin@localhost>... Real domain name required for sender address
ssmtp: 553 5.5.4 <admin@localhost>... Real domain name required for sender address


I have tried putting a bunch of different things in the "From Address" and the "Local Hostname" option boxes but so far have been unable to find anything that works.


Same behavior on 2 different model switches: WS-8-150-AC & WS-24-400A

If I downgrade back to 1.5.17rc2 it works again. (Have upgraded & downgraded several times to test. Also fully defaulted switch after one of the upgrades & manually reconfigured just to make sure config wasn't corrupted and it still didn't work.)

Hoping this can be looked into and addressed as SMTP notification is completely broken for me in 1.5.22

Another related issue: If you put the wrong port into the SMTP settings and run the SMTP test, the switch GUI becomes completely unresponsive for a long time. After about 2min & 30sec you get the "Attempting to reestablish communication with the device ..." banner and then shortly after that it reconnects and you get a "SMTP Test Results" error that claims the test "Timed out after 60 seconds", even though in reality it took several minutes. And when I say the GUI becomes completely unresponsive, I mean completely to the degree that if you open a separate Private/Incognito browser and try to log into the switch it shows you the Login page but won't actually load anything until the test has completed.

In the FromAddress input, does it work if you just use the domain itself?

For example if you want the switch's email to contain the from address admin@thisswitch.com then the FromAddress input box should be "thisswitch.com"

Doing it that way, does it work better?