RSTP bug

[sirhc]

We would have a fix sooner but apparently Eric expects to have a life and go on dates.

Next thing you know he will expect food and bathroom breaks or even time to sleep. Hell someone already poisoned his mind and convinced him he needs a pay check?

As I understand the situation this RSTP software bug should not effect a simple RSTP configuration where only 2 switches are involved but manifests itself when there is intermediary switches like in your lap setup? Now this is not to say we are not going to fix it but I just wanted to express to those with simple RSTP configuration without a dumb switch in the middle should not be affected until we resolve this bug.

This is more then likely why it slip through our testing as we were not testing this type of implementation/topology.

As Rory said though we are sorry for the inconvenience and will work hard to resolve this soon!

[LRL]

No worries. I appreciate you guys getting right on it. My reason for testing with the intermediary switch is because that mimics the failure of the remote side of a backhaul.

I'm not sure if you guys looked at the issue with having RSTP on while the ports were assigned to different vlans. From my packet captures it was passing the BPDU packets across vlans as a result. It seems to me this because you guys haven't implemented PVRSTP or sometimes refereed to as MSTP. Which isn't an issue but it might be a good idea to build in a check to prevent RSTP/STP on the ports untagged on different vlans.

[sirhc]

Well yes and no, using a "switch", even a "dumb" SWITCH to simulate an airFIBER wireless Ethernet BRIDGE is not entirely an accurate lab as a switch handles the BPDU frames/packets used to bring RSTP port status to forwarding or blocking is differently.

The way the WISP Switch RSTP would behave under "most" normal applications is not accurately represented in your lab because you used a switch to represent a BRIDGE. Certain types of special frames/packets are not forwarded by switched the same way as they would be through an Ethernet bridge. A much more accurate representation would be to use an old HUB not a switch to represent the airFIBER wireless Ethernet Bridge. If you had used a HUB to represent the wireless BRIDGE your lab would function perfectly fine.

The RSTP as currently implemented in the firmware absolutely would work fine to fail over between two switches or towers using airFIBER and airMAX radios as 2 wireless Ethernet BRIDGES.

Your lab setup is closer to MSTP then RSTP with the use of that 3rd switch. The switch core we use is capable of MSTP we are just not implementing it yet in the UI but MSTP is absolutely running in the core which is where I think this problem is manifesting itself as we are not configuring some facet of MSTP which is a shared routine with RSTP.

Now with all that being said though there is still a bug here which we fully intend to correct, BUT using the switches as the firmware is using RSTP to control fail-over between 2 towers with 2 wireless Ethernet bridge links would absolutely work fine, it is that 3rd SWITCH that is messing things up for our firmware the way we currently have it implemented.

I think the issue is with the BPDU frames or packets which are not normal packets/frames and are not supposed to transgress beyond a switch, they are designed to carry information between 2 switches using RSTP. You have to forgive me here as my knowledge is a bit weak in this arena and I probably should go read more about it but that is what I have Rory and Eric for so I will leave it up to them.

[LRL]

As I understand BPDU packets, the switch/bridge sends them out all interfaces as multicast and if they're received on another interface it knows there is a loop and blocks one port according to cost and priority.

I may be wrong, it's been 14 years since I learned about this and the mind tends to disappoint me sometimes. Anyways, that was the logic I approached this with.

[sirhc]

As I understand BPDU packets, the switch/bridge sends them out all interfaces as multicast and if they're received on another interface it knows there is a loop and blocks one port according to cost and priority. Inside that BPDU frame is information that RSTP switches share or pass on to their neighbors to each other such as switch and port priority which is used by all the switches involved to determine how to converge in the event of a loop of path failure in about 2 seconds.

I may be wrong, it's been 14 years since I learned about this and the mind tends to disappoint me sometimes. Anyways, that was the logic I approached this with.

BPDU frames/packets are sent out every port that R/STP is configured on with their unique switch port interface mac address but they are destined to 01:80:C2:00:00:00 which when received by an adjoining switch are not passed through anywhere.

The BPDU frame contains information about the R/STP configuration and is used to determine if port(s) need to be set to discarding to prevent a loop or set some ports to forwarding to recover and converge from a path failure. R/STP does path convergence similar to higher level routing protocols.

Sending packets out one port and looking for them on another port is how loop protection works which is totally different.

Loop Protetion is a SIMPLE non-intelligent Boolean type solution where as R/STP makes decisions based on metrics such as priorities or costs if expanding from RSTP to MSTP.

With RSTP when multiple switches are involved they sort of communicate and decide which ports to set to discarding based on priority values set on switches and ports.

The fact that you have 2 RTSP configured switches and a dumb switch in the middle complicates issues and as I stated earlier is more like an MSTP LAB than a simple RSTP fail-over

And as I stated earlier if used as a simple fail-over for wireless links with an RSTP switch at both ends or just a SINGLE RSTP switch on one end and no dumb switch in the middle it would work fine.

The LAB you set up where as it does show a bug on our part is not an accurate LAB as to what your intended use would be and it would work just fine as you intend to use it but not in the configuration you set up which I want to stress again is not a "typical" WISP deployment for a fail over link between towers. Change out that 3rd switch with a HUB which more accurately represents a wireless BRIDGE and it works fine! Switches handle special packets like BPDU differently than HUB's or Transparent wireless Ethernet Bridges.

HOWEVER with that said again I do consed that your lab setup does indeed show a software bug that should not be and we will correct it as soon as possible but in a simple link fail-over it will work just fine as intended.

[LRL]

My apologies. Thanks for setting me straight.

[sirhc]

Well good news, Eric is sending me a new firmware which I will post up for you to try.

Now Rory has not had a chance to LAB test it but Eric feels it is fixed but Rory will still test tomorrow but I will post it up shortly for those that are eager to lab it.

I DECIDED TO LAB BEFORE RELEASE - STAND BY PLEASE

[Rory]

I just wanted to reach out to the folks watching this thread. In our testing of this STP bug we found a bug in how the STP code was applying its settings. From my testing (and Eric's debugging) the port priority settings were being applied immediately upon hitting the 'Apply' button. However, most if not all of the system wide settings (switch priority, timers, whatnot) would not be applied until a reboot.

I haven't heard of anyone reporting this bug, and I'm not sure which firmware versions are affected. I would say its a safe bet that if you are using STP you should update ASAP. We should have a new version out that fixes both these issues as well as modifying some of the interactions between STP and Loop Protection so as to avoid possible error conditions as a result of using both technologies under a very specific set of conditions, or in the event that a user misconfigures STP. I really took some time to try and break the protocol. I am reasonably certain that this update will bring the stability and performance of these features where we need it to be.

Barring major issues the firmware should be out today. If I hear any different I will pass the information on here.

[LRL]

I ran it through my lab and only experienced one issue, but I was unable to reproduce it after.

I ran through several cycles of creating and breaking the loop between the dumb switches. One time the WS never transitioned the port back to a forwarding state. I let several minutes go by but the port remained in the discarding state. I reconnected the dumb switch interconnect and then disconnected and port transitioned fine.

I run this through a few dozen more cycles and was unable to recreate. I'm leaning more towards a issue with the dumb switch, but for what that's worth.

Thank you guys for the fix! I very much appreciate the incredible response time.

On another note, the lack of power cycling of the radios vastly improved my outage window for our production unit. Love it!

[sirhc]

Yea we know there is still a bug in there.

We have a NEW version coming out later today which should finish this issue off as well as a few other ones people reported with ping watch dog and CLI.

The problem is it is not applying changes to RSTP and ping watch dog every time or only after reboots.