v1.5.21 Bug Reports and Comments

[sirhc]

Ok, did some further debugging on this issue. Hopefully the following results can help you to resolve it:

-> The issue only occurs, if the PPPoE server is connected on a SFP-port running latest v1.5.2x firmware series.
-> replacing SFP with other brand: same error
-> disabling STP on switch: same error
-> downgrade to v1.5.16: issue resolved

@sirhc: Do you have another idea, what may cause PPPoE discovery packets being filtered on SFP uplink ports running latest v1.5.2x firmware series?

Thank you for performing tests I asked.

Was asked above if changes had been done to STP/RSTP, not really but the way SFPs are handled was completely redone to fix a lot of issues where soft boots would cause some SFPs to not come up and a lot of SFP vendors do not follow the standards. For instance Cisco copper (GENUINE) never had the issue with soft boots as they handle the way the SFP controller handles the PHY, they do it the correct way. This can be replicated by connecting a device to a non Cisco SFP in our switch then under ports disable the SFP port. The laptop will still show a link as they do NOT properly shut down the PHY leaving it running.

Stephen and I will have to work further on this with you in the near future, sorry for the pain until then.

[sirhc]

just one more bug:
ssh to v1.5.21 and run:

Code: Select all

 {switch-name}# web_conf_check
sh: php-cli: not found

This one is Stephen's, but should be very simple fix in the near future.

[sirhc]

Have I answered all the questions?

Yes thank you, trying to prevent going down useless rabbit holes, happens all the time sadly.

Why the attitude?

Sorry, burned out, 3 months straight, even harder on Stephen as he has to juggle a 3 year old that's teething and he pulled the straw for night duty as they both work from home.

Sorry I have not seen this note within minutes of you posting -- it's sometimes hard to follow these threads -- I'm sure harder for you.

YOU HAVE NO IDEA but Thanks!

Good news is we are 99.9% sure we know what this is and Stephen has family issues today but will be working on it tonight provided his 3 year old daughter cooperates.

I know you said this bug started around 1.5.16 but in reality it has been around since the inception but only became more noticeable the more features we added.

To make a long story short its all about timing. We tell the switch to take readings every second or so it's supposed to. But graphing is a LOW priority service and sometimes the switch gets busy with higher priority stuff like STP, LACP, and so on delaying the readings. When the reading gets reported we just assumed 1 second since last reading but if delayed say 3 seconds then it averages out the reading based on 1 second. This anomaly shows up in all graphs as seen in picture below.

But like I said we are 99% sure we know what it is and how to fix it but nothing is guaranteed until you implement and test.

[sirhc]

There is a bug in SMTP email configuration. The "From Address" field is ignored and the sender is always "switch@localhost".

Simple fix (should be).

Will be in next release

[Flo]

@sirhc: is it possible to reset a SFP module using SSH without having to reboot the WS?
It seems to me that a SFP module port is ignoring the (configured) port-MTU setting after soft- disabling and re-enabling the SFP port, resulting in a port MTU of 1526.

All incoming packets being larger than 1526 are dropped on the receiving SFP port with "Port Detail" -> "Receive Error Counters" -> "Rx Oversize: ..." counting up. I expect being able to resolve this issue un-plugging and re-plugging the SFP module(s).

For further testing I have ordered a couple of other SFP modules using the brands from the compatibility list.

[sirhc]

@sirhc: is it possible to reset a SFP module using SSH without having to reboot the WS?
It seems to me that a SFP module port is ignoring the (configured) port-MTU setting after soft- disabling and re-enabling the SFP port, resulting in a port MTU of 1526.

All incoming packets being larger than 1526 are dropped on the receiving SFP port with "Port Detail" -> "Receive Error Counters" -> "Rx Oversize: ..." counting up. I expect being able to resolve this issue un-plugging and re-plugging the SFP module(s).

For further testing I have ordered a couple of other SFP modules using the brands from the compatibility list.

Well yea, upgrade to v1.5.21 and that problem goes away, well 99.9% anyway. You can also find a damaged or BAD SFP and nothing you can do but replace.

But your answer is NO, if there was a way to do that we would have incorporated that in the firmware and or published the command so as to save a truck roll for thousands of people.

I have not come across any of my old SFP modules that did not come up after warm boot with v1.5.21. I am sure there are some out there as not all vendors follow the spec.


MUST use Cisco compatible, MOST work fine with v1.5.21

Cisco Genuine brand never has that issue as they follow the standard.

But again v1.5.21 fixes 99.9% of all SFP issues, why not upgrade as older versions can also be hacked.

[sirhc]

Another test of a new SFP is to install on bench linked to something.
Do a couple software warm reboots, maybe a firmware upgrade even to same version (v1.5.21).
Then disable SFP under port Tab save apply.
Then Enable SFP under port Tab save apply. - If it comes up your good, if not or it flaps (link up down) use another SFP module.

All people did was complain about the hack, the SFP issue, the memory leak, and a few other things. We drop everything and work 7 days a week for 3 months and come up with our best code ever with only these THREE known issues:

(1) Occasional anomaly in graphs - not a huge deal but will be fixed soon. And it was always there from v1.0.0 just not as apparent.

(2) SMTP alert emails does not send proper sender in email, easy fix and will be fixed soon.

(3) ONE person reporting multicast issue on SFP - Might take a bit as we can not recreate and no one else is reporting it.

SO WHY NOT USE IT??????

[RTGLW]

Hi team,

We believe we've found an issue with DHCP on 1.5.21. TLDR: the dhcp service does not appear to run when the switch boots.

We discovered this when upgrading a nearly default WS-6-MINI from 1.5.14 to 1.5.21. The only configuration changes it had were; DHCP enabled, host name changed, port names changed, and NTP server specified. Everything else was set as it comes on a default switch. We've replicated this issue with a 100% default WS-8-150-DC, upgrading that one from 1.5.15rc3 to 1.5.21. Both switches were configured for DHCP before the upgrades.


After the 1.5.21 upgrade completes and the host reboots it no longer grabs a lease, but it's suddenly reachable on it's 192.168.1.20 default IP. Setting the switch to static and then back to DHCP does not resolve the issue, nor do reboots. We see the switch asking who has it's own static IP on a TCP dump in this state:

Code: Select all

16:56:04.547119 ec:13:b2:92:9f:4a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Announcement who-has 192.168.1.20 (ff:ff:ff:ff:ff:ff) tell 192.168.1.20, length 46


In the host we can see there is no udhcpc client running:

Code: Select all

admin@Netonix_Switch:/www# ps aux | grep dhcp
2026 admin 2680 S grep dhcp


Running "udhcpc -b -i eth0 -H Netonix -S" in the cmdline interface will restore some functionality after starting the udhcpc client. We see the switch now grab a lease, but the switch is now completely unreachable on that DHCP IP. The switch again shows that it's asking who has it's own IP:

Code: Select all

17:15:08.943673 ec:13:b2:92:9f:4a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Announcement who-has 192.168.50.9 (ff:ff:ff:ff:ff:ff) tell 192.168.50.9, length 46


Devices behind the switch are still routing traffic fine, so this appears to only affect management. @Stephen / @sirhc - if you need configuration backups let me know, otherwise I am hoping this one is easily replicated on your end given our lab results.

[sirhc]

Hi team,

We believe we've found an issue with DHCP on 1.5.21. TLDR: the dhcp service does not appear to run when the switch boots.

Easy fix, well check and put in next release. To be honest that is something we hardly ever use or test so our bad.

Thanks but for now just static assign it I WOULD NOT USE OLDER FIRMWARE

[Flo]

Another test of a new SFP is to install on bench linked to something.
Do a couple software warm reboots, maybe a firmware upgrade even to same version (v1.5.21).
Then disable SFP under port Tab save apply.
Then Enable SFP under port Tab save apply. - If it comes up your good, if not or it flaps (link up down) use another SFP module.

It does come up without an issue but seems to block oversize packets larger than 1506 in size. Like oversize MTU packages using: Vlan + MPLS or Vlan + PPPoE

All people did was complain about the hack, the SFP issue, the memory leak, and a few other things. We drop everything and work 7 days a week for 3 months and come up with our best code ever.

Thank you for your excellent work.
Both in hardware and software.

(3) ONE person reporting multicast issue on SFP - Might take a bit as we can not recreate and no one else is reporting it.

That's me and the bug is about PPPoE discovery does not work if the upstream PPPoE server is connected via a WS- SFP-port.

Doing some more debugging on this one, I am still unsure if the Netonix port MTU configuration of (default value) 1528 does include the FCS field or not.

Official documentation about ethernet standards is as follow:

The frame size of a standard Ethernet frame (defined by RFC 894) is the sum of the Ethernet header (14 bytes), the payload (IP packet, usually 1,500 bytes), and the Frame Check Sequence (FCS) field (4 bytes)

Seems to me that the Netonix WS series do not count the FCS field (4 bytes) to the MTU for Ethernet ports, but maybe it does for SFP ports.