PROBLEMS with DC switches at or near FM tower locations

[sirhc]

You say "crashing" what does crashing mean?

[wisp-wand]

It disappears for about a minute, (doesn't respond to ping or SNMP) all customers lose their connection. When its returns, the log is as follows:-

Jan 1 00:00:08 kernel: vtss_port: Loaded port module on board Luton26, type 5
Jan 1 00:00:09 system: Setting MAC address from flash configuration: EC:13:B2:XX:XX:XX
Jan 1 00:00:11 system: starting ntpclient
Jan 1 00:00:11 admin: adding lan (eth0.770) to firewall zone lan
Jan 1 00:00:12 admin: Unable to query power supply
Jan 1 00:00:21 dropbear[769]: Running in background
Jan 1 00:00:24 mini_snmpd[792]: started, listening on port 161/udp and 161/tcp
Jan 1 00:00:25 system: starting ntpclient
Jan 27 15:34:47 switch[784]: Error reading temp sensor
Jan 27 15:34:49 switch[837]: Detected cold boot
Jan 27 15:34:50 switch[837]: PoE enabled on port 2, PoE Smart is starting cable check
Jan 27 15:34:50 switch[837]: PoE enabled on port 3, PoE Smart is starting cable check
Jan 27 15:34:55 switch[837]: Port 2 cable check results: Open, Open, Open, Open
Jan 27 15:34:55 switch[837]: Port 2 passed PoE Smart cable check, turning on power
Jan 27 15:34:55 switch[837]: Port 3 cable check results: Open, Open, Open, Open
Jan 27 15:34:55 switch[837]: Port 3 passed PoE Smart cable check, turning on power

almost seems like a warm restart, but it says cold reboot.

Regards

Wayne

[sirhc]

OK so what you're calling a crash is in reality the switch rebooting.

The question is why?

Possibilities in order of likelihood (in my opinion):
1) Power disruption, switch loses power
2) Defective power supply
3) Defective switch board
4) Some type of interference on the I2C similar to the FM tower sites (noting your substation proximity comment) - DOUBTFUL

Would be nice if you had a spare unit to swap out to see if the symptom remains which would eliminate 2 and 3 as the odds of you having (2) defective units extremely unlikely.

But let's first see what happens with v1.3.9

[wisp-wand]

OK I have another WS-12-250DC here. If it reboots again I can restore the config onto this spare unit and swap them over. At the same time I can install another device that runs on the same power, and that way we'll know if it reboots again, whether the power did fail. The PSU unit is simple and the battery float is the output, you can see the input to the switch is the float voltage of around 26.6, so I can't see it going off, but never say never, certainly learnt to expect the unexpected over the years.

[sirhc]

OK I have another WS-12-250DC here. If it reboots again I can restore the config onto this spare unit and swap them over. At the same time I can install another device that runs on the same power, and that way we'll know if it reboots again, whether the power did fail. The PSU unit is simple and the battery float is the output, you can see the input to the switch is the float voltage of around 26.6, so I can't see it going off, but never say never, certainly learnt to expect the unexpected over the years.

If it reboots and you want to try the other unit lets make it pure meaning if it is not too much of a bother do not export and import config but rather set it up manually to match the one in service.

[wisp-wand]

OK we have managed 20 day uptime since upgrading to version 1.3.9 however noticed that the backup link had a low uptime. Seems we have some more spurious PSU telemetry, causing the switch to shutdown POE, and it shutdown the backup link POE.

Here is the log, looks like it has done it 3 times:-

Feb 14 02:35:06 switch[1403]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)

Feb 14 02:35:06 kernel: link state changed to 'down' on port 2

Feb 14 02:35:11 switch[1403]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification

Feb 14 02:35:14 kernel: link state changed to 'up' on port 2

Feb 14 02:35:18 kernel: link state changed to 'down' on port 2

Feb 14 02:35:20 kernel: link state changed to 'up' on port 2

Feb 14 02:35:37 kernel: link state changed to 'down' on port 2

Feb 14 02:35:39 kernel: link state changed to 'up' on port 2

Feb 14 02:35:42 kernel: link state changed to 'down' on port 2

Feb 14 02:35:45 kernel: link state changed to 'up' on port 2

Feb 19 15:32:48 switch[1403]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 32 C)

Feb 19 15:32:49 kernel: link state changed to 'down' on port 2

Feb 19 15:32:53 switch[1403]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification

Feb 19 15:32:56 kernel: link state changed to 'up' on port 2

Feb 19 15:33:00 kernel: link state changed to 'down' on port 2

Feb 19 15:33:02 kernel: link state changed to 'up' on port 2

Feb 19 15:33:19 kernel: link state changed to 'down' on port 2

Feb 19 15:33:21 kernel: link state changed to 'up' on port 2

Feb 19 15:33:25 kernel: link state changed to 'down' on port 2

Feb 19 15:33:27 kernel: link state changed to 'up' on port 2

Feb 20 03:38:54 switch[1403]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (heatsink 2 is 118 C)

Feb 20 03:38:55 kernel: link state changed to 'down' on port 2

Feb 20 03:38:59 switch[1403]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification

Feb 20 03:39:02 kernel: link state changed to 'up' on port 2

Feb 20 03:39:06 kernel: link state changed to 'down' on port 2

Feb 20 03:39:08 kernel: link state changed to 'up' on port 2

Feb 20 03:39:25 kernel: link state changed to 'down' on port 2

Feb 20 03:39:27 kernel: link state changed to 'up' on port 2

Feb 20 03:39:30 kernel: link state changed to 'down' on port 2

Feb 20 03:39:33 kernel: link state changed to 'up' on port 2

[wisp-wand]

Can I have a reply on the above please. I am keen to roll out more of these switches, I have posted elsewhere that they can solve some issue for me, as well as simplify my sites, but not at the expense of reliability, I need some pointers, or perhaps the firmware can be modified to be more resilient to the spurious PSU communication.

Thanks

W

[sirhc]

OK I have another WS-12-250DC here. If it reboots again I can restore the config onto this spare unit and swap them over. At the same time I can install another device that runs on the same power, and that way we'll know if it reboots again, whether the power did fail. The PSU unit is simple and the battery float is the output, you can see the input to the switch is the float voltage of around 26.6, so I can't see it going off, but never say never, certainly learnt to expect the unexpected over the years.

If it reboots and you want to try the other unit lets make it pure meaning if it is not too much of a bother do not export and import config but rather set it up manually to match the one in service.

We are at a loss but discussing internally.


Lets make sure it is not a faulty unit and indeed is something in the firmware, you mentioned you have a spare unit, did you ever swap the unit out and see if the problem stops?

If not can you?

[sirhc]

We are reviewing code as well at this time but if you can swap the unit and see if same problem exists that pretty much tells us we have bug that only you are seeing.

If the problem goes away then we know the unit is probably faulty.

Or that FM tower that is several hundred feet away is causing the issue? ( I thought you mentioned there was an FM tower somewhere near you but I may be mistaken?)

[wisp-wand]

Hi Sirhc,
as recommended I retyped the config into my spare unit, and installed today. It seemed fine up until I noticed some flapping and the below events (this is the complete log since cold start)

So I guess I don't need to RMA the previous unit, as they both have the same symptoms.
To answer your earlier question, this is not on an FM tower. It is in a lift motor room. There is a cell tower and large sub station about 500metres away, but no one else is on this roof.

Jan 1 00:00:05 netonix: 1.3.9 on WS-12-250-DC
Jan 1 00:00:09 system: Setting MAC address from flash configuration: EC:13:B2:81:xx:xx
Jan 1 00:00:10 kernel: link state changed to 'up' on port 4
Jan 1 00:00:11 system: starting ntpclient
Jan 1 00:00:12 admin: adding lan (eth0.770) to firewall zone lan
Jan 1 00:00:16 kernel: link state changed to 'down' on port 4
Jan 1 00:00:18 kernel: link state changed to 'up' on port 4
Jan 1 00:00:21 dropbear[774]: Running in background
Jan 1 00:00:26 system: starting ntpclient
Mar 19 09:48:13 switch[799]: Detected cold boot
Mar 19 09:48:14 switch[799]: PoE enabled on port 2, PoE Smart is starting cable check
Mar 19 09:48:14 switch[799]: PoE enabled on port 3, PoE Smart is starting cable check
Mar 19 09:48:18 switch[799]: Port 2 cable check results: Open, Open, Open, Open
Mar 19 09:48:18 switch[799]: Port 2 passed PoE Smart cable check, turning on power
Mar 19 09:48:19 switch[799]: Port 3 cable check results: Open, Open, Open, Open
Mar 19 09:48:19 switch[799]: Port 3 passed PoE Smart cable check, turning on power
Mar 19 09:48:21 kernel: link state changed to 'up' on port 2
Mar 19 09:48:22 kernel: link state changed to 'up' on port 3
Mar 19 09:48:26 kernel: link state changed to 'down' on port 2
Mar 19 09:48:28 kernel: link state changed to 'up' on port 2
Mar 19 09:48:43 kernel: link state changed to 'down' on port 3
Mar 19 09:48:45 kernel: link state changed to 'down' on port 2
Mar 19 09:48:46 kernel: link state changed to 'up' on port 3
Mar 19 09:48:47 kernel: link state changed to 'up' on port 2
Mar 19 09:48:50 kernel: link state changed to 'down' on port 2
Mar 19 09:48:52 kernel: link state changed to 'up' on port 2
Mar 19 10:06:30 kernel: link state changed to 'up' on port 1
Mar 19 10:13:07 kernel: link state changed to 'down' on port 1
Mar 19 11:07:42 switch[790]: Power Supply Output Voltage is out of range (70.7V)
Mar 19 12:49:18 switch[790]: Power Supply Output Voltage is out of range (51.2V)
Mar 19 13:10:34 switch[790]: Power Supply Output Voltage is out of range (383.2V)
Mar 19 14:21:50 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 14:21:51 kernel: link state changed to 'down' on port 2
Mar 19 14:21:55 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 14:21:58 kernel: link state changed to 'up' on port 2
Mar 19 14:22:02 kernel: link state changed to 'down' on port 2
Mar 19 14:22:04 kernel: link state changed to 'up' on port 2
Mar 19 14:22:21 kernel: link state changed to 'down' on port 2
Mar 19 14:22:23 kernel: link state changed to 'up' on port 2
Mar 19 14:22:27 kernel: link state changed to 'down' on port 2
Mar 19 14:22:29 kernel: link state changed to 'up' on port 2
Mar 19 15:00:28 switch[790]: Power Supply Output Voltage is out of range (133.2V)
Mar 19 15:47:31 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 15:47:32 kernel: link state changed to 'down' on port 2
Mar 19 15:47:36 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 15:47:39 kernel: link state changed to 'up' on port 2
Mar 19 15:47:43 kernel: link state changed to 'down' on port 2
Mar 19 15:47:46 kernel: link state changed to 'up' on port 2
Mar 19 15:48:02 kernel: link state changed to 'down' on port 2
Mar 19 15:48:05 kernel: link state changed to 'up' on port 2
Mar 19 15:48:08 kernel: link state changed to 'down' on port 2
Mar 19 15:48:10 kernel: link state changed to 'up' on port 2
Mar 19 15:56:19 switch[790]: Power Supply Output Voltage is out of range (51.2V)
Mar 19 16:45:56 switch[790]: Power Supply Output Voltage is out of range (70.7V)
Mar 19 16:59:11 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 16:59:11 kernel: link state changed to 'down' on port 2
Mar 19 16:59:16 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 16:59:19 kernel: link state changed to 'up' on port 2
Mar 19 16:59:24 kernel: link state changed to 'down' on port 2
Mar 19 16:59:26 kernel: link state changed to 'up' on port 2
Mar 19 16:59:42 kernel: link state changed to 'down' on port 2
Mar 19 16:59:44 kernel: link state changed to 'up' on port 2
Mar 19 16:59:48 kernel: link state changed to 'down' on port 2
Mar 19 16:59:50 kernel: link state changed to 'up' on port 2
Mar 19 17:18:51 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 17:18:51 kernel: link state changed to 'down' on port 2
Mar 19 17:18:56 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 17:18:59 kernel: link state changed to 'up' on port 2
Mar 19 17:19:03 kernel: link state changed to 'down' on port 2
Mar 19 17:19:05 kernel: link state changed to 'up' on port 2
Mar 19 17:19:22 kernel: link state changed to 'down' on port 2
Mar 19 17:19:24 kernel: link state changed to 'up' on port 2
Mar 19 17:19:28 kernel: link state changed to 'down' on port 2
Mar 19 17:19:30 kernel: link state changed to 'up' on port 2
Mar 19 18:32:20 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 18:32:21 kernel: link state changed to 'down' on port 2
Mar 19 18:32:25 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 18:32:28 kernel: link state changed to 'up' on port 2
Mar 19 18:32:32 kernel: link state changed to 'down' on port 2
Mar 19 18:32:34 kernel: link state changed to 'up' on port 2
Mar 19 18:32:51 kernel: link state changed to 'down' on port 2
Mar 19 18:32:53 kernel: link state changed to 'up' on port 2
Mar 19 18:32:56 kernel: link state changed to 'down' on port 2
Mar 19 18:32:59 kernel: link state changed to 'up' on port 2
Mar 19 19:56:59 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 19:57:00 kernel: link state changed to 'down' on port 2
Mar 19 19:57:04 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 19:57:07 kernel: link state changed to 'up' on port 2
Mar 19 19:57:11 kernel: link state changed to 'down' on port 2
Mar 19 19:57:13 kernel: link state changed to 'up' on port 2
Mar 19 19:57:30 kernel: link state changed to 'down' on port 2
Mar 19 19:57:32 kernel: link state changed to 'up' on port 2
Mar 19 19:57:35 kernel: link state changed to 'down' on port 2
Mar 19 19:57:38 kernel: link state changed to 'up' on port 2
Mar 19 21:15:58 switch[799]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 19 21:15:58 kernel: link state changed to 'down' on port 2
Mar 19 21:16:03 switch[799]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 19 21:16:05 kernel: link state changed to 'up' on port 2
Mar 19 21:16:10 kernel: link state changed to 'down' on port 2
Mar 19 21:16:12 kernel: link state changed to 'up' on port 2
Mar 19 21:16:29 kernel: link state changed to 'down' on port 2
Mar 19 21:16:31 kernel: link state changed to 'up' on port 2
Mar 19 21:16:34 kernel: link state changed to 'down' on port 2
Mar 19 21:16:36 kernel: link state changed to 'up' on port 2
Copyright 2014-2015 Netonix