Flow Control..... take 2, 3, 4?

[mhoppes]

So I've been seeing a "ToughSwitchy" issue on some parts of my network where download speeds will be 16meg while upload speeds will be 46 meg. All of the network is Netonix. Talking with Chris in some detail he (and I) believe this could be a flow control issue as the packets from the Gigabit Internet are attempted to be crammed down the little 25megabit Internet connection the end user has over a lossy wireless connection.

So... enable FC on all switches... back to head-end... only to discover.... the EdgeRouter doesn't support FC?!?! What the? Looking for comments from anyone else here who's done this. I'd like to stay with the ER line of products for my routers, but if they don't support FC and that actually is my issue then the brings up more concerns.

The odd part about this setup is the configuration is:

Router--->AF24~~~AF24--->Netonix
With several PTMP APs and some PtP hanging off that Netonix. The PtP links seem to function fine (50/50) but the PtMP has just started to get really bad like 16 down and 46up and it seems to be affecting all of the PtMP sectors. The sectors are running great around 80-90% AMC and AMQ and no huge utilization change. This morning there is plenty excess capacity on the network and that's still the only speeds I could pull down.

Because I'm seeing this on the first switch in the network I'm hesitant to say it's a FC issue 100%, but it might be?

[wayneorack]

I know you have done a ton of testing on this issue. One thing I haven't seen tested is the use of APs with Gigabit ports. Have you done a comparison of Rocket m5 vs.Rocket Ti. I wouldn't normally use a Rocket Ti if somebody put a gun to my head. But I do wonder if the gigabit port would help in this situation.

[sirhc]

Flow control needs to be enabled from the "Router" to the AP(s).

Does not matter if this is the first switch or the last switch.

That first switch is just as susceptible of a port buffer overflow as the last.

Once Flow Control can hit the router or a way point in the network the router can then slow down individual streams at the source using normal TCP mechanisms.

I can not believe that the edgeMAX ROUTER does not support Flow Control, I am sure you just do not know where to turn it on (yet).

Flow Control by the way is a function of "Auto" detect meaning if statically set the port speed/duplex Flow Control is disabled regardless of what you have set anywhere else.

[mhoppes]

No.. not the SWITCH.. The EdgeRouter does not support flow control as of February... this is very bad.

https://community.ubnt.com/t5/EdgeMAX/e ... d-p/942875

https://community.ubnt.com/t5/EdgeMAX/E ... rue#M58168

[sirhc]

No.. not the SWITCH.. The EdgeRouter does not support flow control as of February... this is very bad.

https://community.ubnt.com/t5/EdgeMAX/e ... d-p/942875

https://community.ubnt.com/t5/EdgeMAX/E ... rue#M58168

Sorry, I meant ROUTER so I fixed my post

REALLY?! The EdgeMAX Routers do NOT support Flow Control? - That is ludicrous!

The only reason I did not switch to the EdgeMAX Routers because they did not support BFD.

Now I am SUPER HAPPY I did not sell my Cisco Routers and install the EdgeMAX because I ALMOST did just to support UBNT and was going to give up my super fast BFD convergence time hoping someday UBNT would support BFD.... SHEW did I dodge a bullet or what because without Flow Control my service would suck!

[mhoppes]

Yeah, now I'm beginning to question this.... but I can't afford to drop $$ on a Cisco hoping that's my issue.... hmmmmm...

Which router do you run at your head end, Chris?

[wayneorack]

Chris,
What model(s) of Cisco routers are you using?

[sirhc]

Chris,
What model(s) of Cisco routers are you using?

2951

[mhoppes]

Cool. Still not 100% convinced this is a flow control issue, but going to be doing more testing this week.

[sirhc]

Cool. Still not 100% convinced this is a flow control issue, but going to be doing more testing this week.

Well I have Flow Control ENABLED in this test. SEE PICTURE AT BOTTOM OF THIS POST

If I DISABLE Flow Control and there is a decent amount of traffic on my AP (port) my download blew.

I have tested the crap out of this theory.

There are several posts I put up awhile back showing the effects of enabled and disabled of Flow Control at my house.

You need to have Flow Control enabled on the AP port and the up link port on you switch from your router at the tower as well your router needs to be able to listen to pause frames.

If you do not have a router there you need to have flow control all the way back to a routed point in your network and the back haul links need to be able to pass pause frames back.

Yes, sadly if you run a flat network all APs will have the ability to pause "all" traffic" on pause frames but it is not as bad as you think until your network gets big and then at that point you should convert to routed anyway.

This is why if you look at my original tower configuration under THIS post we have the back haul AF24 links in the VLAN trunk aggregation group but later we broke the back haul links out to mid-spans as seen in THIS post of our configuration. Now to be honest "we" did see the effects of the pause frames when we had the back hauls in the VLAN truck but it was not that bad but we decided to take them out.

We wanted to then create 2 port STATIC LAG trunks, 1 for AP's at the tower and one for back hauls but we ran into trouble with two VLANs Trunks while configuring the Cisco so being in a hurry so we just went to 1 VLAN trunk for tower AP's and made the back hauls mid-span but we will go back when we have time convert to 2 VLAN LAGs,.
By separating out the tower AP's from te back hauls those pause frames did not effect traffic just passing through the tower to the next tower.

I have made tons of posts about Flow Control
viewtopic.php?f=14&t=470&p=3098&hilit=flow+control#p3098
viewtopic.php?f=30&t=452&p=2961&hilit=flow+control#p2961

Just do a Search for Flow Control and posts I wrote and you will find some screen grabs of speed tests with and without Flow Control enabled. If there was other traffic on my AP without Flow Control download speeds were as low as 6 Mbps and with it on depending on AP traffic load from 40 to 80 Mbps. Upload speeds were always OK but even better with the Flow control enabled.


The bad part is the EdgeMAX router will not use Flow Control according to Matt so Flow Control in the switches in not used correctly as you need to issue pause frames back to a routed point in your network that has the ability to slow down source streams.

If the EdgeMAX does not support Flow Control then I would raise hell with UBNT to put it in because you need it. Layer 2 devices like switches do not have any mechanisms at their disposal to slow down streams back to the source where as Layer 3 devices like routers (if they are a way point in the route) do which is why you need Flow Control back to a point that is routed.

When you have multiple people on an AP and some go to a web page that has multimedia and adds from maybe a 100 sources and those sources are in Data Centers with FAT pipes and they will all try to push those packets out as fast as they can. As those packets flow through the Net (routed) they are slowed down by TCP mechanisms telling the sender to stop shoving too many packets towards the destination (your client) but still probably a 1G connection into your net. Once inside your net if you are Flat there are no abilities on Layer 2 to slow down packets except Flow Control, either the packet gets though or it gets delayed which is worse then being thrown away as the TTL will expire and the sender thinks the packet was lost so it sends a duplicate, or the packet is simply dropped and the same thing happens.

With Ethernet communications it is all in the timing and Layer 2 has no mechanisms to deal with a port that is advertised as being 100M-FD actually being 60M-HD and that speed can vary from moment to moment. Throwing an even larger buffer at a port is not the answer as then you run the risk of TTL expiring and the sender re-sending a duplicate packet anyway and now your wireless AP will be forced to transmit that packet twice and that is the bottle neck.

Look Flow Control SUCKS but it is NEEDED, at least until something better comes along *cough*

CLICK IMAGE BELOW TO SEE FULL SIZE

flow control.jpg (505.25 KiB) Viewed 15058 times