v1.5.24 Bug Reports and Comments

[mayheart]

Yea i gave it to Stephen and suggested a plan of attack he will do on Thursday.

He did work today but tackled a couple small issues as he had family holiday plans and is off tomorrow and the 1st.

So the plan is to set your configuration up Thursday in a lab. We brain stormed on this today and have agreed on a stragey that SHOULD make clear where the issue is.

Let me know if you need anything tested. I do have a VLAN built out to a few locations that I can break anytime.

[wirelessblue]

I just spent several days troubleshooting the SFP issue and ruling out other causes. We had an issue with a cable that stopped working and I even replaced the WS-12-250-DC with a WS-8-150-DC switch. I am connecting via single mode fiber from a Ubiquiti ES-16-XG with a TRENDnet TEG-MGBS10 SFP set to 1G and that connects to another TRENDnet TEG-MGBS10 SFP also set to 1G in, at first, my WS-12-250-DC switch and now my WS-8-150-DC (the behavior is unchanged when setting the SFP port to auto or 1G). The behavior I am seeing is that some traffic passes like cameras on VLAN 4 and other traffic like VLAN 20, VLAN 110, and VLAN 111 will not pass. All traffic is currently tagged to this switch but I also tried running several VLANs in different configs so there was an untagged VLAN over the wireless link but that made no difference. From the WS-8-150-DC, there is a 60GHz Ubiquiti GigaBeam powered via 24V that links about 330 meters over to another GigaBeam directly connected and powered via 24V from a WS-12-250-DC on that side which has no fiber involved and could not get traffic on most VLANs except for the cameras. All of these Netonix switches had 1.5.16 previously but I upgraded all of them to 1.5.24. The issue disappeared when I downgraded to 1.5.14 on the WS-8-150-DC which is connected via fiber. All other switches are still on 1.5.24 and are now working fine. Is there anything else I can provide that would be helpful? Thank you.

[sirhc]

I just spent several days troubleshooting the SFP issue and ruling out other causes. We had an issue with a cable that stopped working and I even replaced the WS-12-250-DC with a WS-8-150-DC switch. I am connecting via single mode fiber from a Ubiquiti ES-16-XG with a TRENDnet TEG-MGBS10 SFP set to 1G and that connects to another TRENDnet TEG-MGBS10 SFP also set to 1G in, at first, my WS-12-250-DC switch and now my WS-8-150-DC (the behavior is unchanged when setting the SFP port to auto or 1G). The behavior I am seeing is that some traffic passes like cameras on VLAN 4 and other traffic like VLAN 20, VLAN 110, and VLAN 111 will not pass. All traffic is currently tagged to this switch but I also tried running several VLANs in different configs so there was an untagged VLAN over the wireless link but that made no difference. From the WS-8-150-DC, there is a 60GHz Ubiquiti GigaBeam powered via 24V that links about 330 meters over to another GigaBeam directly connected and powered via 24V from a WS-12-250-DC on that side which has no fiber involved and could not get traffic on most VLANs except for the cameras. All of these Netonix switches had 1.5.16 previously but I upgraded all of them to 1.5.24. The issue disappeared when I downgraded to 1.5.14 on the WS-8-150-DC which is connected via fiber. All other switches are still on 1.5.24 and are now working fine. Is there anything else I can provide that would be helpful? Thank you.

v1.5.25 is due out soon and we hope to finish this all off.

THANK YOU FOR YOUR HELP AND PATIENCE.

[marcbou]

we just went through hell after an "upgrade" to 1.5.24 of several switches broke simple DHCP propagation over VLANs (DHCP discover broadcast packets no longer passed).

Downgrade to 1.5.14 fixed it.

Why are you putting customers through this crap ? You'd think that after all these years "stable" firmware would be available from Netonix, but alas, evidently no..

This smells utter incompetence and amateurism ! You deserve bankruptcy or a CLASS ACTION LAWSUIT !

Alpha quality software like 1.5.24 (and the various releases before it in recent times) SHOULD NEVER HAVE BEEN MADE AVAILABLE ON YOUR SITE AS "STABLE SOFTWARE" !

Your testing and QA procedures (if any exist ?) NEED A SERIOUS REVAMP.

[sirhc]

we just went through hell after an "upgrade" to 1.5.24 of several switches broke simple DHCP propagation over VLANs (DHCP discover broadcast packets no longer passed).

Downgrade to 1.5.14 fixed it.

Why are you putting customers through this crap ? You'd think that after all these years "stable" firmware would be available from Netonix, but alas, evidently no..

This smells utter incompetence and amateurism ! You deserve bankruptcy or a CLASS ACTION LAWSUIT !

Alpha quality software like 1.5.24 (and the various releases before it in recent times) SHOULD NEVER HAVE BEEN MADE AVAILABLE ON YOUR SITE AS "STABLE SOFTWARE" !

Your testing and QA procedures (if any exist ?) NEED A SERIOUS REVAMP.

Sorry you feel that way.
But one you should read through the release notes.

Then read through the comments thread and see if any reported issues could effect your setup.

Then as recommended you you should lab it and or try on an easy accessible switch. Then try another and confirm your configuration is compatible with the firmware.

If you blindly rolled it out on multiple switches or your entire network then you chose to roll the dice.

There are literally thousands of possible configurations and mo way to lab everything.

The majority of the people have had good results from this firmware.

I don't know of any company that has never had issues with firmware roll outs.

Just recently a multibillion dollar company rolled out a firmware that bricked almost 100% of their devices.

Read the release notes.
Read the comments thread.
Lab if possible or roll out 1 switch at a time on easy accessible units then move slowly until comfortable firmware works with your application.

If you chose mass roll out then well badvthings can happen.

Known issues are possibly trunked ports, multicast but some have issues some don't.

READ, LAB, TEST ON 1 DEVICE AT TIME BEFORE YOU BLINDLY PUSH.

There are literally hundreds of thousands of units working better then ever and fine and a handful of people having issues and we are working hard to tackle those issues as fast as possible.

So I ask you did you read notes, thread, review known issues, lab, test roll out on 1 unit at time or blindly mass push?

If you did not do all of the above.....

[michaeln416]

Your testing and QA procedures (if any exist ?) NEED A SERIOUS REVAMP.

I second that statement.

There are many free resources online regarding Software Quality Assurance Processes. I recommend starting with IEEE as an entry-level introduction: https://www.computer.org/resources/what-is-software-quality

I would love to see Netonix post its policies and procedures for testing and signing off on a new release before making it publicly available. What are their milestones for migrating from Beta to Release Candidate to Stable?

For the record, I'm not blaming the programmer or questioning their skills. This is a corporate policy issue (or lack thereof) regarding maintaining, testing, and releasing software.

[sirhc]

We have temporarily dropped rc versions as we had to close a security hole / hack.

So many people refused to use RC versions.

We do our best to test code but there are so many possible configurations there will almost always be something missed.

Tell me a company that never has issues.

Most new firmware versions for any manufacturer repair bugs found in previous versions.

This code has a few reported issues we are correcting Hopefully soon.

I test the firmware extensively but even I cautiously roll it out. I start with office switches. Then a close tower and either myself or an employee is notified to be ready or on call. I then monitor that switch / tower before doing another. After a few cautious upgrades I then begin a roll out.

Over the years in IT I have seen small and major bugs for just about every manufacturer I have ever used.

[michaeln416]

Over the years in IT I have seen small and major bugs for just about every manufacturer I have ever used.

Of course, there will always be bugs. It is, therefore, all about how quickly & efficiently those bugs are patched and the quality of each incremental release of the software.

While our other vendors' software has bugs, they are infrequent. More importantly, their processes and policies around reporting, responding, patching, updating, and the quality controls for each new release follow industry best practices and instill confidence.

[mtngoat]

I am thankful for the efforts your small business takes to provide good hardware and software. The behind the scenes hours and hours and hours with lots of anxiety and sweat goes unnoticed. Didn't go home at 5 or even get to go to sleep because you were trying to provide updates that wouldn't be appreciated.

The post covid world can be intolerant, selfish, and plain rude.

Thank you sirhc and Stephen and anyone else behind the scenes. Me, my customers, and my small business thanks you for providing the hardware and software to simplify powering antennas and passing traffic.

[sirhc]

Over the years in IT I have seen small and major bugs for just about every manufacturer I have ever used.

Of course, there will always be bugs. It is, therefore, all about how quickly & efficiently those bugs are patched and the quality of each incremental release of the software.

While our other vendors' software has bugs, they are infrequent. More importantly, their processes and policies around reporting, responding, patching, updating, and the quality controls for each new release follow industry best practices and instill confidence.

Considering since we were hacked and had to re-write 60+ lines of code to remove PHP and have released a new version almost weekly and while we were at it we decided to fix long standing minor bugs along the way.

We had been working 7 days a week for 4+ months.

WE are not a BILLION dollar company let alone multi BILLION and have limited resources.

All I can say is we are doing our best.

Most of the programing was Stephen who often worked through the night.

I have tried my best to lab and test as many things as possible but there are so many possibly configurations it it impossible to test everything.

For 95%+ of customers the new firmware has been great but sadly and expectedly there are some using features and configurations we simply missed.

Stephen is now working on the last major known / reported issue with SFPs using multicast and PPPoE and DHCP. All of us brain storming we have a plan to find the issue. Originally we were betting on my solution to find it but Stephen came up with a possible cause of the issue so internally we are placing the odds at 60% vs 40% in favor of Stephen's idea.
Several smaller issues have been fixed for v1.5.25 but are waiting to release it until we fix this last major issue.

Hopefully this week we put this behind us and move back onto the WS3 product line.

Other people in the company such as engineering have been working on improving other issues in the new new WS3 to make them more harden against ground current so in away everyone is working on what they can and not wasting time.

Your option is to use v1.5.16 until we fix the remaining issues making sure to use ACCESS CONTROL LIST which will protect you from being hacked unitless a device in the access list is compromised but thats on you.