New WS3 Similar Issues

[Garnet]

Hello again,

Since my last time posting we have acquired a new WS3 and it is showing similar problems as our original unit. The unit was bench tested as per Netonix instructions found here: https://forum.netonix.com/viewtopic.php?f=6&t=2780 and passed all tests.

The unit could not be reached at the IP it acquired over DHCP and did not appear to revert to its static IP on its own. The switch was then power cycled (remove power cable, wait 20 seconds, repower). After which it was accessible at its static IP address. Next we tried changing the IP to a different static in this case from 192.168.1.20 to 192.168.1.40. This was successful.

Next I made two configuration changes at the same time. Changed the switch name to 'HT-WS3' and enabled all discovery options. The web UI did not refresh on its own but after a manual page refresh the web UI reported a successful configuration change (this same refresh issue happened with the previous unit). Finally I rebooted the unit. After 6 minutes the switch was not accessible by the web UI or SSH. However during this time the switch acquired two DHCP leases (even though DHCP is disabled on this unit) with one lease bearing the name 'Netonix_Switch' and the other 'estax-71-04-20'. An image of the DHCP leases is attached.

Thus I so far must conclude that our brand new WS3 has the same issues as the WS3 discussed in the thread here: viewtopic.php?f=17&t=7233&p=35804#p35804

We will be doing more testing to see if this switch can be made stable enough for production but until then I would like to once again ask if any other WS3 units have had these issues?

[sirhc]

We have had reports of issues with the WS3 switches using DHCP instead of static.

I would assume since most people do not put core equipment on DHCP very often is why there have only been a few people report this.

For now can you please use a static address as we do not have the time to address this issue.

If you can narrow down specific features that cause issue and document them we will address them as soon as we can.

[Garnet]

When did you start receiving reports about this issue? I have been back and forth on this forum for the better part of a year and one of my counterparts before that and repeatedly were told that the issue was not reproducible.

I have mentioned many times this exact DHCP issue both with the switch being inaccessible as well as acquiring multiple leases and was told that no one had seen this problem (or the issue was just ignored). Also if it wasn't clear above this switch had DHCP disabled (it was using a static IP) and STILL acquired two DHCP leases!

You can read through the linked thread and the threads that are linked there if you want specifics as I have been documenting the step by step procedures that have lead to a crashed or locked up WS3 for some time now. If you need more specifics after that then let me know.

I've been told to make it clear that because you failed to acknowledge we were not alone in this problem you have set back our network upgrades by almost a full year.

[sirhc]

You may be the person I am thinking of that reported it and was discussing with Stephen.

And no we could not reproduce it but we are not discounting what your saying.

If your the one I was thinking of that reported this issue then you are using mikrotik router as the DHCP server, correct?

For now a work around for you is to static assign the switch.

[Garnet]

You are correct that the test setup is using a Mikrotik router as the DHCP server. However you have ignored what I said about the switch acquiring two IP addresses over DHCP when it was set to use a static IP, and this is the second WS3 that has displayed this strange behaviour. On the positive side our new unit seems to at least be able to backup and restore configurations.

[Garnet]

We're going to move forward with testing this WS3. For reference the easiest way to recreate the crashing issue we have experience with both of our WS3's is to do the following:

Using a factory defaulted (via DEF button) WS3 (AC version):
1. Connect WS3 ground lug to a proper grounding point
2. Connect a RS232 to USB adapter to the WS3 and to a laptop
3. Connect a Cat5e patch cable between port 1 of the WS3 and any machine (in this case a laptop via USB to Ethernet adapter)
4. Open a VT100 emulator program (e.g. PuTTy or TerraTerm)
5. Connect the WS3 to AC power
6. Observe serial terminal readout for any boot errors (at this point there should be none)
7. When switch has booted sign in via web UI
8. Go to the configuration options, change the switch name (e.g. HT-WS3) and enable all discovery options
9. Save the configuration (At this point the web UI will usually not refresh itself and needs a manual refresh)
10. Web UI should report that configuration change was successful
11. Reboot the WS3 via the web UI
12. Observe as the serial terminal readout shows you a crashing WS3 (refer to linked thread in previous messages for images of error messages)

[Garnet]

Update: We were configuring this switch to mirror the config of the switch it will be replacing and the following happened.
1. The tech configuring stated that every time they would press save/apply they would get an error that the changes could not be saved, then the browser would refresh and the "changes successful" message would appear.

2. The switch was rebooted to make sure that the new changes 'stuck' in spite of the error.
3. The switch locked up (see previous messages and threads) and upon booting did not have the correct static IP address instead having only a loopback address.

We can accept using static IP's as a workaround for the bugs we've encountered but how does Netonix expect us to workaround not being able to reboot a switch?

[Garnet]

To jog everyone's memory's I've attached an image of what occurs following the roughly 10 minute period that the WS3 units lock up for. This particular image was captured today after a reboot.

[Garnet]

Another update:
It appears that the crashing and lockups are due to improper handling of the switch name field. As my previous posts note we have been using a switch name that contains hyphens. If the switch name contains hyphens it is guaranteed to lockup and require a factory default to function again. There also appears to be an issue if LLDP is enabled which can cause the lockup.

So now my question is, did nobody use the same switch name that I had listed in my prior posts when trying to reproduce this bug? I re-checked my original thread troubleshooting the original WS3 I had, and Stephen even acknowledged that this may be the issue that the WS series switches had.

[sirhc]

We are working on a new firmware for WS3
The error on saving a config change is due to a corrupted config file. The exact cause of the corrupted config that was on the units when shipped is under investigation.
Defaulting the switch and setting it back up fixes that error. Obviously do not export and import the config as it is corrupted so you have to default and set back up.
Defaulting the config does fix a few issues and seems to extend the system reboot. For instance the WS3 I am using and watching took 22 days to reboot after the default previously was rebooting more often.
But since we resolved the TLS issue and a couple other issues needed to resume production of WS - NCS versions as we still can not get current sensors due to semiconductor shortages and we do not have code done to accept the NEW style current sensors developers have gone back to working on the WS3 code. We hope to have a new version soon.
Things being addressed in the WS3 code:
- Stability as to why the unit does reboot for unknown reason
- Fixing the reported issue where the switch ibtains 2 DHCP addresses even when a static address is assigned.
- Fixing the SFP+ port issue but some people who reported on this issue could not have been using latest code as AUTO was removed prior to latest version and the need to change port speed save change port speed save to work properly might not have been an issues if using latest version of WS3 firmware where AUTO was removed abde defaulting it.
We will release a new firmware as soon as possible.