Is it possible to log MAC flap events?

[oeyre]

Hi all,

I am wondering if it is possible currently, or with some development to log instances where a MAC address rapidly changes its learned port (aka a MAC flap)? I've had a few issues now where one particular customer on a multipoint radio managed to cause a loop inside their home network and send our frames back to us, causing all kinds of network instability. The only way that we're currently able to track down when this happens is to manually watch and refresh the MAC table on the netonix and note if it changes ports. It would be much easier if the switch could log/alarm on these listing which ports a certain address was flapping between.

[Stephen]

This is an interesting idea. I'm not sure presently if our system could track this but to investigate, need a bit more information.

As I understand it, you are defining a "Mac Flap" as an instance where a mac address is consistently being discovered on a different port.

So for example.
Event 1:
<some mac address> on port 1
Event 2:
<same mac address> on port 2
Event 3:
<same mac address> on port 1
...etc

Is this the general idea of what you're seeing?

How quickly do these flips happen that indicate this to you that there is an error?

Does my basic sketch above capture the general idea?

[oeyre]

Hi Stephen, thanks for looking at this.

Just to clarify on the scenario: in a typical Layer 2 segment we can reasonably expect that any particular MAC address once learned by the switch will remain on the same port in normal circumstances. Some notable exceptions to this would be a STP reconvergence, or hosts behind a radio device that jumped from one AP to another. It's not really a big deal for this to happen at a low scale, however if the MAC address flaps fast enough between ports then the CPU will take a hit trying to keep up with the learning, not to mention packet loss caused from frames temporarily being sent to the incorrect port.

Here is a diagram with my example topology, imagine the BNG with MAC aa:aa:aa:11:11:11 is terminating customer connections from all over the network. Suddenly a loop happens in part of the network and now every single frame that this faulty segment sees is sent back onto the network. So from the Netonix POV, (and really any other MAC learning device in the path) it will constantly be having to re-learn the BNG MAC and any others flapping between port 24 and port 1.

As an example, this is how Cisco switches log this behaviour:

Code: Select all

12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Te1/49 to port Po5
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Po5 to port Te1/49
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Te1/49 to port Po5
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Po5 to port Te1/49
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Te1/49 to port Po5
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Po5 to port Te1/49
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Te1/49 to port Po5
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Po5 to port Te1/49
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Te1/49 to port Po5
12:05:00: %C4K_EBM-4-HOSTFLAPPING: Host 74:8E:F8:62:4C:41 in vlan 4057 is moving from port Po5 to port Te1/49


I'm not sure what initial threshold (if any) was passed to start logging the flapping, however you can see all these events happened within the same second! Possibly you might also want to set some kind of upper rate limit of messages to log as to not thrash the CPU/disk?

[Stephen]

Noted, thanks for the information.

We have an rc release planned to address a few bugs that have been reported in 1.5.16 - the bugs have to be priority and some of them are somewhat involved. However, I will experiment with this as well. If I have developments, will post them in this thread.

That being said my biggest concern is to determine if this functionality is available in the switchcore or if I will have to do it with application logic handled on the CPU. If that's the case, it might be too strenuous for the switch to be usable on even a medium sized network.

But it's worth a try.

Also, yes if events are recorded I would probably want to add a tally so that it doesn't log these events constantly if it happens.

[oeyre]

So I have just noticed that there is a "Loop Protection" setting already available, however from the looks it just disables the entire port.

As an alternative, or possibly just as a first step, I am wondering if its possible to add an option for this feature to log only, instead of outright disabling the port? That might be "good enough" to identify the offending port.