WS-12-400-AC Dropping several ports simultaneously

[Dawizman]

Just replaced a couple tough switched with a WS-12-400-AC on Monday. I started noticing dropouts on most of the connected radios yesterday (Tuesday). The switch is running 1.4.6 (I don't like running rc firmware on my production network), the devices that are rebooting are all Airmax and Airmax AC. The sole AF5X is not having issue, it has been consistent since I first booted it up 2 days ago. I will post the log since the last switch reboot 2 hours ago. We only rebooted in an attempt to correct the issue. Any ideas?

[sirhc]

You should be running v1.4.7rc18

We do firmware differently than most companies.

If we finalized v1.4.7 tomorrow then the very next day a bug was found we would release v1.4.8rc1

So the major difference between v1.4.6 and v1.4.7rc18 is that v1.4.7rc18 has all the reported bugs found and fixed since v1.4.6 was released along with security patches.

We do have release notes where you download and if you look at the release notes it shows what bugs were fixed in rc1,rc2 and so on.
Disable Flow Control on any port facing airFIBER radios as they have a Flow Control issue which is widely known on this forum and UBNT's Forum.
Beyond that there are 30,000 WISP switches running out in the wild and if this was a common "problem" of the switch you would hear about it so it has to be something with your config or setup.

I would start by doing the following:
1) Upgrade to v1.4.7rc18 unless you want to run firmware with known Bugs and security holes.
2) Disable Flow Control on any port facing airFIBER radios and airMAX AC radios (they have a Flow Control Bug / Issue).

There are also other radios/Device with Flow Control bug / Issues such as a suspected issue with some epmp radios so you can always disable Flow Control on all port then each day enable Flow Control on a port and watch the Port Stats for Rx Pause Frames.
Beyond this you would need to investigate more but I can tell you there is nothing wrong with the switch but there is something which you have to discover.
Also it helps to post up screen captures of any switch UI TABs with config changes from default settings.

[Dawizman]

I've read about all sorts of issues with flow control, but I didn't think this was related, as the radios are actually power cycling. In fact, every 24V device is experiencing issues, but the single 48VH device has not had any problem yet.

As for the firmware versions... Would it not make more sense to drop the rc from the name then, if you are so confident in the software? We're used to beta & rc software introducing bugs while fixing others.

[sirhc]

hmm power cycling.....didn't catch that.

And yes we are considering changing our firmware numbering scheme.

Please post up screen grabs of all your Tabs.

Also please describe your site, rower, box, power, grounding.

[Dawizman]

The power is fed from a panel 250ft away via underground direct bury 12awg cable. The tower foundation consists of 3 6" x 12ft steel helical piles screwed in to the ground 10ft. These work better than any ground rod. The piles are all bonded together with heavy wire. The electrical is bonded directly to the foundation via a ground buss where it comes in to our enclosure. The switch is fed from an APC UPS which is bonded to the ground buss. The enclosure & switch are both tied to the ground buss. All cat5e going up the tower is shielded outdoor Belden, terminated with shielded connectors. The tower is only 96ft to the top.

Here are the screenshots:

[Julian]

worth checking into what your red exclamation point is, i think.

Can you post a view of your log tab with any red exclamation entries in view?

[sirhc]

The power is fed from a panel 250ft away via underground direct bury 12awg cable.

OK lets first address ground current, read this post: viewtopic.php?f=17&t=1786&start=30#p13447

If your pulling power from a distance which prevents you from bonding the AC service ground rods with the tower ground rods you should NOT use the AC service ground. - VERY BAD. If your tower grounds are better than your service grounds then current will seek out your tower grounds and the current will pass through the switch and across your cables.

While you are at it please read these grounding posts as well:
viewtopic.php?f=30&t=1816
viewtopic.php?f=30&t=188
viewtopic.php?f=30&t=1429
viewtopic.php?f=17&t=1786&start=30#p13447
https://community.ubnt.com/t5/airFiber/ ... rue#M31070

Next verify you did not have any ping watchdog running on your radios.

But I can not stress enough the importance to understand that ground is not ground and there is what is called ground potential difference and that means current will flow between the ground potentials and if your switch, cable, and radios are the path for this current to flow since you do not have a HEAVY dedicated cable with a shorter path bonding the 2 grounding systems (rods) you will burn up equipment.

Ground current flowing on Ethernet cables will eventually destroy equipment but before it does this current can cause radios/equipment to make a buzzing sounds, and can cause Ethernet errors and other weird issues.

Ground potential differences can cause minor issues like errors and buzzing until it rains which can cause the potential to widen and then cause damage. Or an Electrical surge comes in the AC service into the building you get power from and surge protectors in that building do what they are supposed to and clamp the excess voltage to ground which then shoots across towards your tower through the switch and wires to the radios and if the current is too high you get damage.

Ethernet surge protectors and a UPS or electrical surge protector does absolutely nothing to prevent ground current from passing. Ground current is the #1 cause of damage to switches we get back for RMA.

[lligetfa]

The power is fed from a panel 250ft away via underground direct bury 12awg cable. The tower foundation consists of 3 6" x 12ft steel helical piles screwed in to the ground 10ft. These work better than any ground rod. The piles are all bonded together with heavy wire. The electrical is bonded directly to the foundation via a ground buss where it comes in to our enclosure.

Should we assume that it is a single leg not a 240V split? A 250 foot run of 12/2 with ground under current means that voltage will drop on the hot lead and rise equally on the neutral lead. Only a well balanced 240V split would not carry current on the neutral and so not have a potential rise on neutral.

Bonding the mains protection ground to the tower ground does nothing. Bonding the tower ground to the mains service ground through the 250 foot long lighter gauge bare wire does no good either.

[sirhc]

I agree with you Les.

In my opinion I would do as I suggest in the one post linked above where he does not bring or use the electrical service Earth Ground at all and only uses the tower Earth Ground.

So he only brings the Hot (Black wire) and Neutral (White wire) from the remote service into his box/receptacle and uses a ground wire from the tower ground bus to connect to the receptacle Earth Ground Lug.

This way all of his equipment in his box and the tower are all using the same Earth Ground and there is no possibility of an Earth Ground Potential difference which will cause excessive ground current to flow through his equipment which Ethernet/Electrical Surge Protectors do nothing to alleviate as they clamp excessive or high voltage to ground and do nothing to stop ground current.

THis ground current flow can cause weird things to happen even before it reaches a level to damage things. This current causes noise on the equipment PCB or Board which can cause weird things to happen even things like reboots.

If certain cables between the switch and radio are shorter than others those radios will take the brunt of the current as they are the shorter paths.

Also radios like NanoStations and PowerBeams and such which do not provide a path to tower ground as they are insulated inside a plastic non conductive chassis. It is radios like Rockets which connect to an antenna which clamps to steel that provide a path from the Ethernet cable to the tower that creates a path and becomes bond and the issue.

[lligetfa]

I don't wish to drag this off topic but the NEC required minimum for service ground is intended to draw enough current on a ground fault to throw your average 15A breaker but there is no actual test to verify intent. In some cases the resistance on the service ground added to the resistance on the circuit under fault might not actually trip a breaker. Remember that the code intent is not to protect sensitive electronic equipment but rather to save lives and buildings.

Also, code requirement for bonding is #6 wire, not that puny bare copper in your 12/2 feed.