v2.0.9rc2 Bug Reports and Comments - WS3 Firmware

[Julian]

OCP timing window, and whether or not it trips, is hard set by discrete components; waiting for a software fix may be unproductive

will be extremely so, to be clear, there are very precise limits requiring a lot of math to avoid; an analog approach is best, a supplemental FPGA to handle the rigid timings required digitally would bring unnecessary cost and development time.

Not what we usually see with these, though. I can envision more than one responsible underlying gremlin, but would need to take an actual physical look in order to know enough specifics to be able to confirm precisely which. An AC component on your DC feed will not particularly bother a 12-250-DC, but can look similar on the 3-14E, if you want to take a look for that on your end, but other things can lurk, too.

Happy to take a look at the switch, as well, they do occasionally develop a personality. The two I -have- seen are both back in service, and *checks* outside the warranty extension a trip to the bench gives you, too. rma.netonix.com is that way, sir <--

[klosey]

Appreciate the suggestions, to be fair, we purchased 5 from MBSI back in 09/23 and probably had them sit on our shelves for 6 months or so before trying to deploy. We only had 2 out in the field when we discovered the problems. Unfortunately they have been removed from our live networks but we will see if we can set them back up in a test environment and see if we can recreate the problem on all the units if necessary.

We were not using any surge protectors, all radios are direct into switch with no ethernet surge protectors used.

[sirhc]

It's possible there is a hardware issue but I would be more prone to think OCP is seeing AC ground current which it does not differentiate from DC current.

It could be exposing AC ground current travellings through your switch.

The NEW WS3-14E will measure AC ground current which will tell people grounding issues.

The older WS unit will pass AC current and do so until something blows.

Are rgese off grid or on grid?

If on grid is AC service ground rods in GOOD condition and bonded to tower grounding system?

If you bench test these units and they work fine powering up radios on bench I would lean towards ground current.

[elcano89]

Hi!
I just got a new WS3-14-600-AC today to replace an older netonix that is saturated going over 1G over the 2 sfp ports. It came with v2.0.9rc2 factory installed. I configured it with a static ip on our tagged management vlan, but I couldn't get it to respond from our production server or from anywhere else for that matter. We noticed it only responds to packets originating from the same subnet, which points to a missing default route on the netonix's table. I had to do a src-nat on our router, to mask any request going into the netonix, with an IP of the same subnet as the management interface, and only then could access it. I tried factory resetting, reinstalling v2.0.9rc2, but had the same outcome. I decided to downgrade to v2.0.8, and problem solved, now it seems to generate a default route correctly and will answer to any requests outside the subnet.

I can only speculate that it is probably a default route issue, but I don't know how to drop into the unix shell and check out the routing table. We have over 80 Netonixs deployed, all legacy WS series, this is our first WS3, and we've never seen this issue before. Only issue we have come across in all these years was a batch of switches that would not forward packets with the factory flashed firmware (oh boy... that was a real PITA to figure out! ). But otherwise, knock on wood, pretty solid on the WS series.

Anyone else encountered this issue? Or did I get a bad factory flashed FW?

Thanks!

[elcano89]

Hi!
I just got a new WS3-14-600-AC today to replace an older netonix that is saturated going over 1G over the 2 sfp ports. It came with v2.0.9rc2 factory installed. I configured it with a static ip on our tagged management vlan, but I couldn't get it to respond from our production server or from anywhere else for that matter. We noticed it only responds to packets originating from the same subnet, which points to a missing default route on the netonix's table. I had to do a src-nat on our router, to mask any request going into the netonix, with an IP of the same subnet as the management interface, and only then could access it. I tried factory resetting, reinstalling v2.0.9rc2, but had the same outcome. I decided to downgrade to v2.0.8, and problem solved, now it seems to generate a default route correctly and will answer to any requests outside the subnet.

I can only speculate that it is probably a default route issue, but I don't know how to drop into the unix shell and check out the routing table. We have over 80 Netonixs deployed, all legacy WS series, this is our first WS3, and we've never seen this issue before. Only issue we have come across in all these years was a batch of switches that would not forward packets with the factory flashed firmware (oh boy... that was a real PITA to figure out! ). But otherwise, knock on wood, pretty solid on the WS series.

Anyone else encountered this issue? Or did I get a bad factory flashed FW?

Thanks!

Ok... We just finished migrating all the sites from the old switch, and... It's bad... There is a lot of weirdness going on. I'll list what we've encountered so far:

1) The web UI locks up from time to time, randomly, and when it does, you can see the total throughput spike to over 10Gbit/s and it stops responding. The CPU also goes up to 100% and stays there. It seems to be an internal loop or some weirdness going on inside the switch chip or even a phantom report to the UI, we cannot observe that traffic over the 10g SFP link on the uplink router, and traffic seems to keep flowing "normally" or at least appears so. I have to test some more and see if latency is affected.

2) When you reboot the switch or anything that cycles down/up the uplink (sfp+ port14), something happens, like a storm, that it even locks up the router (Mikrotik CCR1072).

3) The lockup and the 10g phantom traffic weirdness started at around 30 vlans added and total traffic was over 700Gbit/s.

4) As it sits, it's handling right now about ~1.2Gbit/s, but will go over 1.8Gbit this night on peak hours. We stopped migrating the last 3 sites because we aren't sure if it will take a crap with the 3Gbit/s total traffic it could see over the night.

5) Other less critical stuff we notice; The total power draw graph is stuck at 15.2 watts, but you can see the correct watts draw over each individual interfaces. But if you select the interface, it will display 0.0w on the upper right hand report.

The migrated config is a mirror of what was running on the other WS-12-250-AC switches, and they've been working perfectly. We are migrating to WS3 for the 10g ports and better performance.

Any insight is appreciate, we'll keep it as is on production and is available if data for testing is needed (as long as it doesn't go boom and we have to rollback).

Cesar

[elcano89]

*Next day update*

The switch hasn't crashed, nor have I noted any significant degradation in latency, but it doesn't want to be touched at all, any change locks the CPU at 100%, the interface stops refreshing, and traffic spikes at over 20-25Gbit/s. We finished migrating everything from the older WS, so it is seeing around ~1.8-2.0Gbit/s now, but the CPU is over 75-80% all the time, the UI is very laggy, and it is still seeing random traffic spikes, at over 20Gbit/s. The incorrect watts report and small bugs are not of much importance to us, but the switch locking up and bringing the uplink interface down is of concern.

To recap, this replaced a WS-12-250-AC running 2 sfp 1Gbit/s uplinks to our agg router, which was coming close to 2Gbit/s throughput. The config remained exactly the same on the router and the new Netonix. We are not running LAG, STP, or any particular function. The old WS-12-250-AC ran this exact config with absolutely no issues, no spikes, or any CPU or UI issues. I found another thread in here with a tech reporting the same spikes and CPU issue. We invested in various of these as upgrades, so any help is appreciated.

Cesar

[sirhc]

Now that Stephen is back on WS3 since we finally got WS code done and the best code we ever has for WS.

The WS3 firmware is being built on a much newer SDK and should be a great improvement.

Sadly we are 6+ weeks out but will be whole new platform like WS v1.5.25

[elcano89]

Thanks for the reply!

I'll just let it be for the time being, it seems to be "working" with bugs and all, and there's no need to work any config changes as it sits. If anything is needed for testing or debugging, I'm glad to be of help.

Thanks
Cesar

[sirhc]

I think you'll like the new code when it comes out.

We started this with the new version at the time 6 months ago.

Onecwe get it out we will upgradecto an even new version before we start implementing routing.

New WS3-E will have newdest version SOC and use NAN flash which is much much faster, much larger cpu. Boot time should be like 10 to 30 seconds.

Isolated power supplies
AX Ground current pass through to help you spot poor grounding and fix it.

Within the year routing so 1 box is all you need at a tower or use it as just a switch.

But it's a core router not a software router so line speed.

[robgnet]

Updated a switch from 1.5.16 to 1.5.24 noticed that the MAC Aging field was blank, tried to put in a value of 300, and click save, and it says Invalid Fields, then I saw 1.5.25 so I updated to that, still stuck, cannot save anything to make that field go away.