MSTP does not handle VLAN changes

[JustJoe]

Had a network with multiple switches having multiple VLANs sometimes running in parallel. The only way to make this work is disable STP/RSTP, per port, otherwise they start getting flagged as alternates and traffic gets blocked. In this network, routers happily use layer3 OSPF for fallback Internet access, because I was NOT wanting to play layer2 loop STP Russian-roulette. ;)

So, have been testing MSTP. The specific Instance/VLAN lists are not that bad for us, because our Internet clients all use router mode CPEs with PPPoE to completely isolate them, so don't need 1 VLAN per client. Instead, the VLANs separate the AP & PtP radio infrastructure subnets to something easy to manage all the way back to the access routers.

When configured from scratch, MSTP is GREAT !!! Any accidental loops are VERY quickly sensed and managed (which is what I really want STP for) and nicely logged.

But POOP !!! I can't change a VLAN port membership and have MSTP be aware after a config apply.

At first this was very confusing because my Netonix MSTP learning curve was on 3 lab switches where the lost links seemed random. But eventually (and somewhat surprisingly) I was able to recreate a big part of this on one switch.

Setup:
1) Management host PC: 3 separate NICs with IPs: 192.168.1.133, 192.168.32.133, 192.168.34.133 each cabled to (eventual) matching VLAN port on WS.
2) WS-12-250-AC v1.5.6 reset to defaults (that means RSTP is enabled)
3) Set main IP: 192.168.32.222/24
4) Set management VID = 32 'EEEEEEUUUUTTTT'
5) Added VID = 34 'EEUUUUEEEETTTT' Set IP: 192.168.34.222/24
6) Added VID = 11 'UUEEEEEEEETTTT' Set IP: 192.168.1.222/24

All the applies went smoothly

7) Tested every VID32 & VID34 port, cabling to the appropriate host PC and all pings either replied when correctly matched or timed-out on a mismatch.

8) Configured to use MSTP:
MST Instance 1 VLAN List 32
MST Instance 2 VLAN List 34
MST Instance 3 VLAN List 11
All the applies went smoothly

9) Repeated #7, all good.
10) POR, repeated #7, all good.

11) Changed Prt6 from VLAN34 to VLAN32, config seemed happy:
UI: VLAN 32 PortSettings: changed from 'EEEEEEUUUUTTTT' to 'EEEEEUUUUUTTTT'
UI: VLAN 34 PortSettings: changed from 'EEUUUUEEEETTTT' to 'EEUUUEEEEETTTT'

12) BUT, Prt6 became un-pingable as either ..32.222 or ..34.222, probably because it thought it was still in Instance 2:
Port: link state changed to 'up' (100M-F) on port 6
STP: msti 0 set port 6 to discarding
STP: msti 2 set port 6 to discarding
STP: msti 0 set port 6 to learning
STP: msti 0 set port 6 to forwarding
STP: msti 2 set port 6 to learning (Should be msti 1 I think)
STP: msti 2 set port 6 to forwarding

-------OK in this test case, if I power cycle reboot the switch, it starts to work correctly. But that is just not acceptable in a production remote location. And if I revert the action to Prt6 and put it back in VLAN34, once again it is locked out from pinging.

This is a bug that needs to be fixed first, BUT ... the original bug on the multi switch network still failed *sometimes* even after POR ???
...but that needs to wait for a retest when I get a version with this fix in it.

I am going to do a second post and try to include at least one WS config and one log. (If you need other trial points, please ask, I may have already saved them.) I have a suspicion the configs are going to look fine, and instead this is going to be a mismatch on what gets applied to vtss/MSTP during that GUI config update.

Also, IF you should find there is a bypass to accomplish the VLAN port re-assignment without requiring a POR, that would really help!

(Edit: corrected acronym spelling)

[JustJoe]

192.168.32.222_3rd MSTP w-VLAN chg Prt6_WS-12-250-AC_Netonix-Switch.log.txt

Console log

(6.85 KiB) Downloaded 413 times

[JustJoe]

Additional info:

I suspected that the act of changing then applying the config to move Prt6 from one VLAN to another was missing the step of re-scanning and updating the MSTP part to vtss.

So I tested a hunch:

1) Reassign Prt6 to the other VLAN, apply, then ping to confirm it is unreachable.
2) Find a "dummy" change to make on the STP tab ... I chose to change the revision number from "1" to "2".
3) Then apply and ping ... and the ping works !

Now this is a bit of a problem as a bypass, because now this switch's rev number no longer matches other switches in the same region. I don't have enough experience to know how much chaos this might cause on a production network, but ....

4) Going back and this time resetting the revision back to "1" and applying ... and the ping still works.

[Stephen]

Thank you for the information JustJoe I'm looking into it.

[JustJoe]

Even more info...

On the STP tab / MSTP / CIST instance / There is the enable checkbox column per port ...

If a VLAN port has not been enabled under MSTP, then has its box checked, then applied, a similar situation occurs as above where it shows up correctly on the MSTP status page as for example "forwarding", but in fact it will not pass traffic until I again try the bypass/workaround of changing the revision number, then changing it back.

Also, I wish I could test making some of these changes from CLI to see if I can get around the bug that way. But in this "port enable" checkbox case, there seems to be no obvious CLI configuration command I could find.


*********Is anyone out there actually using MSTP on Netonix successfully ????

[JustJoe]

Thank you for the information JustJoe I'm looking into it.

Any update on at least confirming a recreate on this?

Not even a download of the config I provided?

[Stephen]

I've been working on some background things for awhile, I get to start addressing issue's on the WS this week. I'll be starting with what you've mentioned here and the TLS problem.

[david.sovereen@mercury.net]

I hesitate to post this here because this is not an MSTP-specific observation and I certainly do not want to hijack JustJoe's post as I am eager to see this MSTP issue fixed. I post this here, only because it seems closely related/similar.

Using RSTP, we have seen multiple times where a Device on a Port/VLAN cannot be reached. Other Devices (typically Mikrotiks in our network) on the same VLAN can sometimes partially "see" the Device and under IP -> ARP, will show the MAC and IP of the unreachable device with the flag D(ynamic) but missing the C(omplete) flag. Rebooting the Netonix has no effect, but changing the Port VLAN value to E(mpty) and Save/Apply, and changing the Port VLAN value back to T(agged) fixes it and makes the device reachable. IP -> ARP on those Mikrotiks will now have the (C)omplete flag. I saw this yesterday happen on a brand new Netonix switch that had 1.5.5 on it (and could not be downgraded to 1.5.2, a version that seems more reliably in our network).

The similarity I see to JustJoe's situation is that one can change an STP setting and the web interface and log suggest the setting change was made, but for some reason it wasn't really "under the hood" and that one needs to change the setting again, Save/Apply, and then change it back and Save/Apply to really get the setting change made "under the hood."

Hope this only helps and does not detract from the issue JustJoe reported. My apologies if you would have preferred this posted in a new thread.

Dave

[JustJoe]

I don't have any problem with you posting here. I have already posted a couple of times to add observations to the problem.

A question for what you are describing ... Have you read the "devices shipped at v1.5.5 not passing packets"?

viewtopic.php?f=17&t=6243

That actually seemed to be some sort of manufacturing firmware initialization issue, where just reinstalling the same firmware was supposed to totally fix it.

In contrast, the switches I am using in my lab tests probably started at v1.4.x and were incrementally updated over the years to v1.5.6.

[Stephen]

JustJoe, please take a look at the release 1.5.7rc1

I was able to replicate the issue and resolve it.

This is a bug that needs to be fixed first, BUT ... the original bug on the multi switch network still failed *sometimes* even after POR ???
...but that needs to wait for a retest when I get a version with this fix in it.

I'll keep an eye on this thread. If you still see failures after testing this release, I'll work on preparing another firmware update. First let's see how far this one takes us.