Flow Control..... take 2, 3, 4?

[mhoppes]

Do you have a 2951 I can borrow to buy if it fixes it?

[sirhc]

As a packet is sent from 1 router to the next router that router acknowledges the packet was received OK and sends a small packet back requesting the next packet. This is the normal TCP mechanism that deals with how many packets from a fat pipe can fit into a small pipe.

Now when you have packets from 100's or even 1000's of sources from all over the web all converging on a 100M-FD that is really say 60M-HD those packets from all those sources reach the ports waiting room (buffer) and fill it up QUICK then as more packets are sent into the waiting room from 1000's of sources to different destinations that is already full packets are tossed out and disappear (dropped) causing the sending device to time out and send duplicates. Since the switch is a Layer 2 transparent bridge it is not sending replys back as it receives the packets, that is the responsibility of the destination (the CPE) to send that reply to it's gateway that packet was received send me the next one or group of packets.

This is why when you test an AP with no or few clients or low traffic times everything seems fine but add 10 maybe 20 clients visiting web site each with 100's of data streams these packets cause the port buffer to fill up and drop packets.

If it is just you on the AP your packets will not cause this overflow because your packets are usually from a small number of sources and your packets easily fit into the waiting room and your CPE receives the packet and sends the request for another. It is not until MULTIPLE client radios are requesting packets from 100's of sites all screaming into the waiting room at once and your AP is struggling with a wireless retry to a poor connection and the packets just keep coming.

With Flow control the switch is able to determine via an algorithm (all happening in microseconds) that the buffer will be full soon so it sends a pause frame to the sender (the router) to pause for 50 micro-seconds giving the port time to clear the waiting room out.

I understand and can see this in my head I just think I am not explaining it very well.

[sirhc]

Do you have a 2951 I can borrow to buy if it fixes it?

LOL - actually no, I need more, expanding and upgrading old tower routers that used to be 2941

I think you could use 2941 Matt and they are CHEAP on eBAY, like $200.

You might need to add an HWIC for additional ports as the 28 and 29 series only have 3 built in ports.

[mhoppes]

BTW, from Chuck @ UBNT AirFiber:

Hi,

While I don't doubt that there are configurations for which flow control slows the effective throughput, that is not our experience here, and we have still not been able to produce those results in the lab. If we get details of a configuration that exhibits this, we will try to reproduce and address it. As of right now, we still believe that turning flow control on for both airFiber and the attached switch/router is the proper thing to do.

As to pause frames passing across a network, they do not. Pause frames are link local MAC frames (not ethernet packets) that are always terminated on the ethernet link from which they were generated. If all of your layer 2 switches have flow control enabled, each switch will generate it's own link local Pause frames on an as-needed basis.

Chuck

[sirhc]

That is correct what Chuck said

As to pause frames passing across a network, they do not. Pause frames are link local MAC frames (not ethernet packets) that are always terminated on the ethernet link from which they were generated. If all of your layer 2 switches have flow control enabled, each switch will generate it's own link local Pause frames on an as-needed basis.

But keep in mind if you have a switch with an AP on port 12 and port 1 is the uplink from your router/backhaul the switch will issue the pause frame to the router/backhaul from port 1 when port 12's buffers are full.

Now that then causes a cascade effect if everything across your Layer 2 segment is Flow Control aware and enabled meaning as port buffers fill up back the chain pause frames will get issued further and further back the Layer 2 chain until it reaches a router. Once it reaches a routed way point in the source destination path the router can control how fast it will receive and forward packets for individual streams because if a router is unable to send a packet fast enough to a destination it igors the packets coming to it waiting in it's buffers (waiting room) not sending the needed reply for the next packet in sequence for that stream and by default TCP mechanisms will kick in. Now this too is not a perfect system and there are issues but go much less noticed, the main one being that a packet being ignored by a router can expire and the sender can issue a replacement but when the router turns around to process that packet it inspects the packet's TTL and if it is expired it knows there is a duplicate packet following and will discard the packet which is better then passing it along as it is now an invalid packet so why waste resources transmitting it any further.

If there is enough packets getting retransmitted then the sender usually times out and stops sending packets to that destination and an error usually appears on your browser screen saying something like timed out.

[mhoppes]

Turned on FC on both netonix switches and both airfibers... remember, this was a simple setup.

Netonix--->AF24~~~~AF24--->Netonix--->EdgeRouter

Throughput literally went from 13 down / 40 up to 40 down / 40 up!

WOAH.... I bow before you Chris.

[sirhc]

But you still need to get the EdgeMAX Router to listen to Pause Frames from Flow Control

[sbyrd]

I have a few questions on when to enable FC or not.

1. I have a router at a tower. Each port has 1 AP (RocketM) directly attached to it. Should FC be on or off on the router interface port for each directly attached AP?

2. The uplink port of that tower router routes through a AF-5X bridge to a Netonix switch that then trunks into a router at my backhaul tower on the other end. I am correct to turn on FC on the uplink interfaces of the router on each side and the ports on the Netonix?

3. I have a router at a tower. The APs at this tower are powered by a PoE switch (TS-8 Pro). Each AP put into its own vlan on the TS and are then trunked to an interface on the tower router. I am correct to turn on FC on the TS and the trunk port of the router?

[mhoppes]

You should be enabling flow control on everything, everywhere. But definitely on the links going to the APs and your backhauls.

You will also want to go into the airFiber and enable flow control there, the airMax gear does not have a "FC On" option, but you should still enable it on the router ports if your router supports it.

[sbyrd]

One other question.

4. I have a tower router that routes through a SAF bridged backhaul and connects directly to a router interface at a backhaul tower. Should FC be on or off on the routers interfaces at each end?