v1.5.22 Bug Reports and Comments

[sirhc]

We are committed to finishing off the WS firmware at the cost of further delay of WS3, but would greatly help if more people who install it with no problems and describe what features they are using. This prevents us from wasting time going down rabbit holes.

[Dawizman]

I didn't notice if it's known in my quick read through. Upgraded one switch at a new tower site in order to test before going network wide. WS-12-250-DC. It's been running for a few days no issues. This morning we moved customers over to it, and the SNMP interface traffic counters seem to have stopped working shortly after, despite SNMP still responding, and other OIDs working just fine such as power and thermal related sensors, and uptime.

How can I help diag?

Here's an example of what we are seeing. Using SNMP v2c. It worked from initial boot, and then shortly after we started passing traffic it just went to 0.

I have not yet rebooted the switch in case there's an diagnostic info I can get you guys.



https://imgur.com/a/D2yQEPn

[Stephen]

I didn't notice if it's known in my quick read through. Upgraded one switch at a new tower site in order to test before going network wide. WS-12-250-DC. It's been running for a few days no issues. This morning we moved customers over to it, and the SNMP interface traffic counters seem to have stopped working shortly after, despite SNMP still responding, and other OIDs working just fine such as power and thermal related sensors, and uptime.

How can I help diag?

Here's an example of what we are seeing. Using SNMP v2c. It worked from initial boot, and then shortly after we started passing traffic it just went to 0.

I have not yet rebooted the switch in case there's an diagnostic info I can get you guys.



https://imgur.com/a/D2yQEPn

This is an odd one, I have a test running on one of our medium bandwidth live switch's to see if we can replicate the issue.

In the mean time can you send the result's of the following commands:

Code: Select all

ps aux | grep snmp


Also, on the same switch, please post a screenshot of the total throughput (the bottom right graph on the Status tab)

Wait maybe 10 minutes, and get another screenshot of the total throughput.

Also, seeing the logs on that switch could be helpful.

[Stephen]

Hi.

VLANs stopped working after upgrade to v1.5.22. I have about 100 VLANs, all of them stopped working. I use 1528 MTU on the ethernet and 1524 MTU on the VLAN. Downgrade to v1.5.16 made them work again.


Thanks for your work.

Can you PM me a config backup of the switch with about 100 VLANs that failed on v1.5.22?

[Dawizman]

I didn't notice if it's known in my quick read through. Upgraded one switch at a new tower site in order to test before going network wide. WS-12-250-DC. It's been running for a few days no issues. This morning we moved customers over to it, and the SNMP interface traffic counters seem to have stopped working shortly after, despite SNMP still responding, and other OIDs working just fine such as power and thermal related sensors, and uptime.

How can I help diag?

Here's an example of what we are seeing. Using SNMP v2c. It worked from initial boot, and then shortly after we started passing traffic it just went to 0.

I have not yet rebooted the switch in case there's an diagnostic info I can get you guys.



https://imgur.com/a/D2yQEPn

This is an odd one, I have a test running on one of our medium bandwidth live switch's to see if we can replicate the issue.

In the mean time can you send the result's of the following commands:

Code: Select all

ps aux | grep snmp


Also, on the same switch, please post a screenshot of the total throughput (the bottom right graph on the Status tab)

Wait maybe 10 minutes, and get another screenshot of the total throughput.

Also, seeing the logs on that switch could be helpful.

Code: Select all

admin@OpenWrt:/www# ps aux |grep snmp
 907 admin 5324 S /usr/sbin/snmpd -Lf /dev/null -p /var/run/snmpd.pid
 1637 admin 2680 S grep snmp


I also did an SNMP walk just to confirm, the counters have stopped. These were run ~ 5 minutes apart

Code: Select all

C:\Users\Justin>C:\usr\bin\snmpbulkwalk.exe -v 2c -c <hidden> 10.21.2.10 .1.3.6.1.2.1.31.1.1.1.6
IF-MIB::ifHCInOctets.1 = Counter64: 13836532
IF-MIB::ifHCInOctets.2 = Counter64: 3887397789
IF-MIB::ifHCInOctets.3 = Counter64: 0
IF-MIB::ifHCInOctets.4 = Counter64: 0
IF-MIB::ifHCInOctets.5 = Counter64: 209658652
IF-MIB::ifHCInOctets.6 = Counter64: 102408443
IF-MIB::ifHCInOctets.7 = Counter64: 94443216
IF-MIB::ifHCInOctets.8 = Counter64: 102612383
IF-MIB::ifHCInOctets.9 = Counter64: 103104744
IF-MIB::ifHCInOctets.10 = Counter64: 72973599
IF-MIB::ifHCInOctets.11 = Counter64: 0
IF-MIB::ifHCInOctets.12 = Counter64: 0
IF-MIB::ifHCInOctets.13 = Counter64: 0
IF-MIB::ifHCInOctets.14 = Counter64: 0

C:\Users\Justin>C:\usr\bin\snmpbulkwalk.exe -v 2c -c <hidden> 10.21.2.10 .1.3.6.1.2.1.31.1.1.1.6
IF-MIB::ifHCInOctets.1 = Counter64: 13836532
IF-MIB::ifHCInOctets.2 = Counter64: 3887397789
IF-MIB::ifHCInOctets.3 = Counter64: 0
IF-MIB::ifHCInOctets.4 = Counter64: 0
IF-MIB::ifHCInOctets.5 = Counter64: 209658652
IF-MIB::ifHCInOctets.6 = Counter64: 102408443
IF-MIB::ifHCInOctets.7 = Counter64: 94443216
IF-MIB::ifHCInOctets.8 = Counter64: 102612383
IF-MIB::ifHCInOctets.9 = Counter64: 103104744
IF-MIB::ifHCInOctets.10 = Counter64: 72973599
IF-MIB::ifHCInOctets.11 = Counter64: 0
IF-MIB::ifHCInOctets.12 = Counter64: 0
IF-MIB::ifHCInOctets.13 = Counter64: 0


Throughput screenshots run about 20 minutes apart:
First: https://imgur.com/a/lHsFZzs
Second: https://imgur.com/a/FmjYxEt

Current uptime is ~4 days 1 hour. I would ballpark the counters stopped at almost exactly 3 days of uptime.

[Stephen]

...

Code: Select all

admin@OpenWrt:/www# ps aux |grep snmp
 907 admin 5324 S /usr/sbin/snmpd -Lf /dev/null -p /var/run/snmpd.pid
 1637 admin 2680 S grep snmp


I also did an SNMP walk just to confirm, the counters have stopped. These were run ~ 5 minutes apart

Code: Select all

C:\Users\Justin>C:\usr\bin\snmpbulkwalk.exe -v 2c -c <hidden> 10.21.2.10 .1.3.6.1.2.1.31.1.1.1.6
IF-MIB::ifHCInOctets.1 = Counter64: 13836532
IF-MIB::ifHCInOctets.2 = Counter64: 3887397789
IF-MIB::ifHCInOctets.3 = Counter64: 0
IF-MIB::ifHCInOctets.4 = Counter64: 0
IF-MIB::ifHCInOctets.5 = Counter64: 209658652
IF-MIB::ifHCInOctets.6 = Counter64: 102408443
IF-MIB::ifHCInOctets.7 = Counter64: 94443216
IF-MIB::ifHCInOctets.8 = Counter64: 102612383
IF-MIB::ifHCInOctets.9 = Counter64: 103104744
IF-MIB::ifHCInOctets.10 = Counter64: 72973599
IF-MIB::ifHCInOctets.11 = Counter64: 0
IF-MIB::ifHCInOctets.12 = Counter64: 0
IF-MIB::ifHCInOctets.13 = Counter64: 0
IF-MIB::ifHCInOctets.14 = Counter64: 0

C:\Users\Justin>C:\usr\bin\snmpbulkwalk.exe -v 2c -c <hidden> 10.21.2.10 .1.3.6.1.2.1.31.1.1.1.6
IF-MIB::ifHCInOctets.1 = Counter64: 13836532
IF-MIB::ifHCInOctets.2 = Counter64: 3887397789
IF-MIB::ifHCInOctets.3 = Counter64: 0
IF-MIB::ifHCInOctets.4 = Counter64: 0
IF-MIB::ifHCInOctets.5 = Counter64: 209658652
IF-MIB::ifHCInOctets.6 = Counter64: 102408443
IF-MIB::ifHCInOctets.7 = Counter64: 94443216
IF-MIB::ifHCInOctets.8 = Counter64: 102612383
IF-MIB::ifHCInOctets.9 = Counter64: 103104744
IF-MIB::ifHCInOctets.10 = Counter64: 72973599
IF-MIB::ifHCInOctets.11 = Counter64: 0
IF-MIB::ifHCInOctets.12 = Counter64: 0
IF-MIB::ifHCInOctets.13 = Counter64: 0


Throughput screenshots run about 20 minutes apart:
First: https://imgur.com/a/lHsFZzs
Second: https://imgur.com/a/FmjYxEt

Current uptime is ~4 days 1 hour. I would ballpark the counters stopped at almost exactly 3 days of uptime.

I currently have a test running that is at about 20 hours uptime. Only difference is that I'm querying 1.3.6.1.2.1.2.2.1.10 aka IF-MIB::ifInOctets

Would you mind also checking this OID to make sure it's counters have stopped as well?

Also how often are you querying the system?

[JeffreyS]

I noticed two issues.
Some switches - not all - lost the ability to connect to our NTP server. The switch had no problems pinging the NTP server.
Found issue on both WS-6 mini and WS-12-250-DC "Time not set"

I had this on a WS-8-150-AC that I deployed last week running on 1.5.21 and initially after upgrading yesterday morning to 1.5.22 it did the same. Now, the time is in sync.

I had another switch take a bit for the time to sync with the NTP server after boot after upgrading to 1.5.22. I think it may be that I had a time delay set on the uplink device to power up. I since removed this from the uplink devices and the time sync during boot is more reliable. This used to not be the the case from my experience. *shrug*

[sirhc]

I noticed two issues.
Some switches - not all - lost the ability to connect to our NTP server. The switch had no problems pinging the NTP server.
Found issue on both WS-6 mini and WS-12-250-DC "Time not set"

I had this on a WS-8-150-AC that I deployed last week running on 1.5.21 and initially after upgrading yesterday morning to 1.5.22 it did the same. Now, the time is in sync.

I had another switch take a bit for the time to sync with the NTP server after boot after upgrading to 1.5.22. I think it may be that I had a time delay set on the uplink device to power up. I since removed this from the uplink devices and the time sync during boot is more reliable. This used to not be the the case from my experience. *shrug*

The old firmware would keep trying indefinitely, which was not a great idea.

We limited it to 10 times. After that it will try every 24 hours.

You can always force it to try and sync by disabling save enabling save NTP

[Flo]

We are still struggling with the issue having all PPPoE discovery packets (MAC protocol: 8863) being filtered on SFP ports.

I did some more debugging with the following findings on Friday:
- PPPoE client PADI packets send out the network are no longer received from PPPoE server
- changing SFP module brand does not resolve this issue
- unplug / re-plug SFP modules does not resolve this issue
- soft-reboot WS-250-AC does not resolve this issue
- hard-reboot WS-250-AC does not resolve this issue
- increasing the port MTU from 1528 to 1556 does not resolve this issue
- disable "Storm Control": Broadcast, Multicast, Unicast filters to "None" does not resolve this issue
- disable Discovery protocols does not resolve this issue

Downgrading to v1.5.16 does immediately resolve the issue.

If you have PPPoE discovery working on latest v1.5.22 via SFP links please share details on your setup.

Update: As promised, I did some more debugging using a brand new spare WS-12-250-AC and I was able to narrow down this issue.

PPPoE discovery is working on latest v1.5.22 only, if you are running PPPoE untagged (Vlan: U) via a SFP uplink port.

In detail:
I have unpacked a brand new WS-12-250-AC running default configuration. First I have upgraded the device to v1.5.16, then further to v1.5.22.
On SFP-port 13 I have connected a MikroTik RB5009UP... configured as PPPoE server, on Ether-port 9 a MikroTik hEX ... configured as PPPoE client.

The configuration works fine on both v1.5.16 and v1.5.22.

Next step:
Adding Vlan (eg: 123) to the WS-12-250-AC and both MikroTik devices; adding second PPPoE server on Vlan 123.
This setup works fine running v1.5.16, but not with v1.5.22.
PPPoE discovery packets from the hEX to the RB5009 connected on the SFP-port are only received for the untagged default network; but are filtered on the Vlan.

Other packets (ping test) did pass between both devices on the same Vlan.

Hint:
As a last step I tried out, if enabling the "Trunk port / Allowed VLANs" checkbox does change anything. It does not, but one more thing: I am unable to uncheck this checkbox running latest v1.5.22.

[Flo]

Hi.

VLANs stopped working after upgrade to v1.5.22. I have about 100 VLANs, all of them stopped working. I use 1528 MTU on the ethernet and 1524 MTU on the VLAN. Downgrade to v1.5.16 made them work again.


Thanks for your work.

Jorge,

do you have services connected on an uplink SFP port using Vlans?
Doing some more testing, it seems to me that all multicast packets (PPPoE discover, ARP) are filtered on uplink SFP ports using tagged VLANs.