v1.5.21 Bug Reports and Comments

[sirhc]

Ok - I've upgraded two switches to 1.5.21 and sadly - I can report that the voltage surges from 1.5.16 remain !! (See my post above in this same thread).
(To clarify - these same switches do not have voltage surges in 1.5.14).
Please do a deep dive at what must have been introduced between 1.5.14 and 1.5.16 --- I'm guessing some I2C bus related code.

Thanks!!

Yahel.

We will investigate, obviously it is a false reading and the improvements in v1.5.21 outway this one false reading so I would continue to use v1.5.21 until addressed.

[CrackerRiley]

At the risk of not posting in the right place, we noticed a new issue that was introduced when upgrading from 1.5.14 to 1.5.16.
The issue is voltage graphs on DC switches (specifically WS-12-250-DC ).
The graph looks like that - essentially making it useless.

https://photos.app.goo.gl/bEkn2w9etwtHm5Ye7
We had to downgrade multiple switches back to 1.5.14 to avoid this issue.
I would love to know if this was reported/addressed in later firmwares.

To further clarify - these are NOT REAL surges - this is happening consistently on multiple switches (>40) and is not happening when downgrading to 1.5.14

Thanks,

Yahel.

I can confirm our 1.5.16 switches do the same thing.
I have a switch now on 1.5.19 that also shows the spikes.
Don't have a DC switch at 1.5.21 yet to see if it's still happening there.

[sirhc]

Yahel,

Question:
1) How are you powering these switches, power supply, solar chargers, AC charger, on or off grid...
Chargers kicking in could cause spikes.

I also notice an occasional anomaly on data graphs which I ignore. For instance last night after fixing my back link which is only capable of a couple hundred megs I waited till 3AM and forced the backup link into service and traffic was averaging 100-200 Mbps then on a couple occasions over an hour or so period I saw a blip like yours to 600+ Mbps which was impossible so I discarded it.

All the graphs use the same routine so this gives credence to what you and I both saw. But this has nothing to to do with I2C, most likely a simple error in the code that creates the data arrays used for the graphs which is why I sometimes see it in data graphs as they share the same logic and method.

This could be an ANOMALY in the graphs and could be ignored for now. Downgrading to such an old version that can be hacked, has memory leaks, SFP incompatibilities, to name a few does not seem like a good idea. Version 1.5.21 is the most stable secure version to date.

IntL-Daniel

There is a bug in SMTP email configuration. The "From Address" field is ignored and the sender is always "switch@localhost".

We will look at both these issues as soon as possible but right now unless something BAD shows up we have shifted our main goal to finishing up WS3-14E-600-AC/DC as that model was already supposed to be on the market months ago.

But to be clear using older firmware is dangerous and much less stable. v1.5.21 has so far 2 small fix requests:
1: A graph anomaly
2: The SMTP alert email from address

Not saying we will ignore them, they will get fixed, but are on low priority as we NEED to get WS3 code finished. We spent almost 3 months getting to v1.5.21 to eliminate the hack and stabilize many long running issues like SFP, memory leaks, and so on.

[yahel]

I completely understand how the voltage graph anomaly might seem as a minor issue and low priority for you.
Where it comes to our network, we just finished downgrading 47 switches (WS-12-250-DC) from 1.5.16 to 1.5.14 because this graph issue is a deal breaker for us.
We rely on the graphs to understand battery charge condition, and during these stormy times this is more critical for us than whatever security issues (probably less relevant due to our topology) or stability issues which we have not observed with 1.5.14.
The storms affect both our solar sites (no sun for days) as well as other sites due to prolonged power outages.

It's been long on my list to graph the voltages using SNMP and add more automation to battery state-of-charge into our monitoring system - this would take time, and until we do, we've learned to rely on the Netonix's voltage graphs, which we cannot do without (the surges make the graphs unusable).

Thanks for understanding - I hope you can spare a bit of time on this (I hope it's just a little minor thing to fix and would not detract you much from your WS3 focus).

Thanks!

Yahel.

We have over 200 Netonix switches on this network alone (I consult and manage other operations as well).
Partial list from Netonix-Manager of units we recently downgraded (about 3 more pages of that - won't fit in a screen).

[sirhc]

Yahel,

You did not describe your power solution.

Are you using solar chargers or such, is it possible this is a spike from the charger?

If your switches get hacked (latest hack was malicious requiring a re-flash or possibly a console recovery sometimes) or if a reboot requires you to roll a truck from an SFP not comping back up requiring it to be pulled and re-inserted, or reboot loop requiring a power cycle.

Just a friendly warning at what your risking. If a hacker finds 1 he might find all. Please at least use access control list which will provide a shield provided one of the computers in the access control list is NOT their hole into your network.

Thanks,
Chris

[yahel]

Thanks Chris - don't get me wrong, I am VERY worried about the security issue -- however, I do have access controls and isolated VLANs to prevent access to the switches without VPN..

The voltage surges are NOT REAL -- with 1.5.14 there are no surges and all is working fine.
Some sites are solar, some are not - we have hundreds of them, and 1.5.16 reliably make the surges appear, while 1.5.14 reliably eliminate the surges in the graph.
This was verified across about 70 switches (now almost all rolled back to 1.5.14).

I would LOVE to go up to 1.5.21 - I hear you, and the memory leak does get us sometimes, I'm sure other issues do occasionally show up - but I just can't give up on the voltage graphs at this point.
We are working on it, but our software development and sysadmins teams are busy with other stuff, so unlikely we'll be able to do without the graphs until after the winter.

Your help in fixing this problem would be highly appreciated.

Thanks,

Yahel.

[Flo]

Hi sirhc,

thanks for the awesome performance of the new v1.5.2x firmware series.

I have noticed the following serious bug on v1.5.21:

SFP ports (at least on WS-12-250-AC) block multicast transmissions of PPPoE-Scan (PPPoE-Discovery, MAC proto 8863) packets, even though the MC option is enabled on both ends of the fiber link. Non multicast packets (like PPPoE, MAC proto 8864) do pass.

It does work correctly on RJ45 copper ports, so it seems to me that the issue is related to the latest SPF-interface updates.

I will have to downgrade the affected devices back to v1.5.16, except if you have another idea how to fix this issue.

-Flo

[sirhc]

Hi sirhc,

thanks for the awesome performance of the new v1.5.2x firmware series.

I have noticed the following serious bug on v1.5.21:

SFP ports (at least on WS-12-250-AC) block multicast transmissions of PPPoE-Scan (PPPoE-Discovery, MAC proto 8863) packets, even though the MC option is enabled on both ends of the fiber link. Non multicast packets (like PPPoE, MAC proto 8864) do pass.

It does work correctly on RJ45 copper ports, so it seems to me that the issue is related to the latest SPF-interface updates.

I will have to downgrade the affected devices back to v1.5.16, except if you have another idea how to fix this issue.

-Flo

Could you please try multiple SFP brands, espiciall if you have genuine Cisco modules and let us know. And if you find others work please send pic and info on module that fails. Also try disabling MSTP on that port, helps us out a lot.

[Flo]

...
I have noticed the following serious bug on v1.5.21:
SFP ports (at least on WS-12-250-AC) block multicast transmissions of PPPoE-Scan (PPPoE-Discovery, MAC proto 8863) packets, even though the MC option is enabled on both ends of the fiber link. Non multicast packets (like PPPoE, MAC proto 8864) do pass.
...

Could you please try multiple SFP brands, espiciall if you have genuine Cisco modules and let us know. And if you find others work please send pic and info on module that fails. Also try disabling MSTP on that port, helps us out a lot.

SFP modules are generic FS which did work without any issues on prior versions up to v1.5.16.
RSTP is obligatory due to an active/backup configuration. Is there a way to debug why the new firmware may block multicast packets on SFP port(s)?
Hint: Device -> Configuration -> Options: IGMP snooping is disabled.
ARP works without any issue.

[sirhc]

The reason I am asking to try another brand or 2 of SFPs is we re-wrote how SFP modules are handled to fix previous issues.

We now support far more SFP modules without issues, like some times people would warm reboot switch from UI or firmware upgrade and SFP would not come back and user would have to go to switch and pull and re-insert SFP to bring back up or cold boot switch. Many people had reported this issue including I saw it at my WISP, I would usually just switch brand test to work and throw SFPs in pile of don't trust, After re-write of how SFP modules are controlled I was able to9 go back through that pile and they all work like a charm.

If you found a brand that caused issues then we could order same module and look why.

I meant RSTP not MSTP but again you could temporarily disable backup link then disable RSTP on main link and see if multicast works. Again there are reasons Im asking for these tests as RSTP is not aware of some special packets like MSTP is, I am NOT making up useless tests to jerk you around it actually helps us and could prevent us from looking in areas that have been eliminated.

We are trying to help but am also asking you to try and help eliminate rabbit hole as we have limited time and NEED to get WS3 back on track or pack it up as industry WANTS 10G SFPs and 2.5G copper.

And WS3 line WILL have routing abilities, not right away but within months of pre-release including static, OSPF, and BGP but not world route capable. Also very impressive bandwidth limiting, up to 4096 rules per switch based on ingress /egress / IP / MAC / Port or range / timeand more.