i2c Errors on WS-26-500-DC Switches

DOWNLOAD THE LATEST FIRMWARE HERE
aaron@futuretk.com
Member
 
Posts: 20
Joined: Mon Feb 13, 2017 10:43 am
Has thanked: 0 time
Been thanked: 0 time

i2c Errors on WS-26-500-DC Switches

Thu Mar 08, 2018 4:48 pm

We have a number WS-26-500-DC of switches that are occasionally rebooting themselves. The only common thing we are seeing on these switches is a presence of i2c power supply errors in the log. We have upgraded to firmware 1.4.9 on all of them, but the issues are continuing. Here are the relevant log entries from a switch that had this happen today. I have since configured external logging so I hope to catch more detail the next time it happens.


Dec 31 18:00:06 netonix: 1.4.9 on WS-26-500-DC
Dec 31 18:00:15 system: Setting MAC address from flash configuration: EC:13:B2:06:1C:3C
Dec 31 18:00:17 system: starting ntpclient
Dec 31 18:00:18 admin: adding lan (eth0) to firewall zone lan
Dec 31 18:00:19 admin: Unable to query power supply
Mar 8 12:12:07 UI: i2c error setting 0x47 12 110
Mar 8 12:12:28 UI: i2c error setting 0x47 14 147
Mar 8 12:12:31 dropbear[954]: Running in background
Mar 8 12:12:34 switch[984]: Detected cold boot
Mar 8 12:12:37 switch[983]: retrying i2c link to power supply
Mar 8 12:12:40 switch[983]: retrying i2c link to power supply



Mar 8 12:12:43 switch[983]: retrying i2c link to power supply
Mar 8 12:12:43 switch[983]: i2c link down to power supply (0)
Mar 8 12:12:43 switch[983]: i2c link up to power supply (1)

Thank you
-Aaron

aaron@futuretk.com
Member
 
Posts: 20
Joined: Mon Feb 13, 2017 10:43 am
Has thanked: 0 time
Been thanked: 0 time

Re: i2c Errors on WS-26-500-DC Switches

Fri Mar 16, 2018 10:21 am

We are continuing to have this issue. It seems to be impacting most of our new WS-26-500-DC switches all running 1.4.9 firmware.

Here is a log from another one I replaced yesterday which is also showing Read and Write errors:

Dec 31 18:00:04 kernel: 0x00500000-0x00b00000 : "rootfs"

Dec 31 18:00:04 kernel: mtd: partition "rootfs" set to be root filesystem

Dec 31 18:00:04 kernel: 0x00b00000-0x00f40000 : "rootfs_data"

Dec 31 18:00:04 kernel: 0x00f40000-0x00f80000 : "FIS directory"

Dec 31 18:00:04 kernel: 0x00f80000-0x00f81000 : "RedBoot config"

Dec 31 18:00:04 kernel: 0x00fc0000-0x01000000 : "Redundant FIS"

Dec 31 18:00:04 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

Dec 31 18:00:04 kernel: TCP cubic registered

Dec 31 18:00:04 kernel: NET: Registered protocol family 10

Dec 31 18:00:04 kernel: lo: Disabled Privacy Extensions

Dec 31 18:00:04 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team

Dec 31 18:00:04 kernel: NET: Registered protocol family 17

Dec 31 18:00:04 kernel: 802.1Q VLAN Support v1.8 Ben Greear

Dec 31 18:00:04 kernel: All bugs added by David S. Miller

Dec 31 18:00:04 kernel: VFS: Mounted root (squashfs filesystem) readonly.

Dec 31 18:00:04 kernel: Freeing unused kernel memory: 116k freed

Dec 31 18:00:04 kernel: Please be patient, while OpenWrt loads ...

Dec 31 18:00:04 kernel: Algorithmics/MIPS FPU Emulator v1.5

Dec 31 18:00:04 kernel: mini_fo: using base directory: /

Dec 31 18:00:04 kernel: mini_fo: using storage directory: /jffs

Dec 31 18:00:04 sysinit: killall: udhcpc: no process killed

Dec 31 18:00:06 netonix: 1.4.9 on WS-26-500-DC

Dec 31 18:00:07 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.

Dec 31 18:00:07 kernel: switch: 'Luton26' board detected

Dec 31 18:00:10 kernel: vtss_port: Loaded port module on board Luton26, type 5

Dec 31 18:00:11 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)

Dec 31 18:00:11 system: Setting MAC address from flash configuration: EC:13:B2:06:06:CE

Dec 31 18:00:11 kernel: i2c /dev entries driver

Dec 31 18:00:11 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19

Dec 31 18:00:13 sysinit: Loading defaults

Dec 31 18:00:13 system: starting ntpclient

Dec 31 18:00:13 sysinit: Adding custom chains

Dec 31 18:00:14 sysinit: Loading zones

Dec 31 18:00:14 sysinit: Loading forwarding

Dec 31 18:00:14 sysinit: Loading redirects

Dec 31 18:00:14 sysinit: Loading rules

Dec 31 18:00:14 sysinit: Loading includes

Dec 31 18:00:14 admin: adding lan (eth0) to firewall zone lan

Dec 31 18:00:15 sysinit: Error: Read failed

Dec 31 18:00:15 admin: Unable to query power supply

Dec 31 18:00:23 kernel: eth0: no IPv6 routers present

Dec 31 18:00:27 Port: link state changed to 'up' (100M-F) on port 17

Dec 31 18:00:27 sysinit: Error: Write failed

Dec 31 18:00:28 sysinit: Error: Write failed

Dec 31 18:00:28 sysinit: Error: Read failed

Dec 31 18:00:29 sysinit: Error: Read failed

Dec 31 18:00:29 sysinit: Error: Write failed

Dec 31 18:00:30 sysinit: Error: Write failed

Dec 31 18:00:30 sysinit: Error: Read failed

Dec 31 18:00:31 sysinit: Error: Read failed

Dec 31 18:00:31 sysinit: Error: Write failed

Dec 31 18:00:32 sysinit: Error: Write failed

Dec 31 18:00:32 sysinit: Error: Read failed

Dec 31 18:00:33 sysinit: Error: Read failed

Dec 31 18:00:33 sysinit: Error: Write failed

Dec 31 18:00:34 sysinit: Error: Write failed

Dec 31 18:00:34 sysinit: Error: Read failed

Dec 31 18:00:35 sysinit: Error: Read failed

Dec 31 18:00:36 sysinit: Error: Write failed

Dec 31 18:00:36 sysinit: Error: Write failed

Dec 31 18:00:37 sysinit: Error: Read failed

Dec 31 18:00:37 sysinit: Error: Read failed

Dec 31 18:00:38 sysinit: Error: Write failed

Dec 31 18:00:38 sysinit: Error: Write failed

Dec 31 18:00:39 sysinit: Error: Read failed

Dec 31 18:00:39 sysinit: Error: Read failed

Dec 31 18:00:40 sysinit: Error: Write failed

Dec 31 18:00:40 sysinit: Error: Write failed

Dec 31 18:00:41 sysinit: Error: Read failed

Dec 31 18:00:41 sysinit: Error: Read failed

Dec 31 18:00:42 sysinit: Error: Write failed

Dec 31 18:00:42 sysinit: Error: Write failed

Dec 31 18:00:43 sysinit: Error: Read failed

Dec 31 18:00:43 sysinit: Error: Read failed

Mar 15 08:23:33 sysinit: Error: Write failed

Mar 15 08:23:34 sysinit: Error: Write failed

Mar 15 08:23:34 sysinit: Error: Read failed

Mar 15 08:23:35 sysinit: Error: Read failed

Mar 15 08:23:35 sysinit: Error: Write failed

Mar 15 08:23:36 sysinit: Error: Write failed

Mar 15 08:23:36 sysinit: Error: Read failed

Mar 15 08:23:37 sysinit: Error: Read failed

Mar 15 08:23:37 UI: i2c error setting 0x47 12 110

Mar 15 08:23:37 sysinit: Error: Read failed

Mar 15 08:23:38 sysinit: Error: Write failed

Mar 15 08:23:38 sysinit: Error: Write failed

Mar 15 08:23:39 sysinit: Error: Read failed

Mar 15 08:23:39 sysinit: Error: Read failed

Mar 15 08:23:40 sysinit: Error: Write failed

Mar 15 08:23:40 sysinit: Error: Write failed

Mar 15 08:23:41 sysinit: Error: Read failed

Mar 15 08:23:41 sysinit: Error: Read failed

Mar 15 08:23:42 sysinit: Error: Write failed

Mar 15 08:23:42 sysinit: Error: Write failed

Mar 15 08:23:43 sysinit: Error: Read failed

Mar 15 08:23:43 sysinit: Error: Read failed

Mar 15 08:23:44 sysinit: Error: Write failed

Mar 15 08:23:44 sysinit: Error: Write failed

Mar 15 08:23:45 sysinit: Error: Read failed

Mar 15 08:23:45 sysinit: Error: Read failed

Mar 15 08:23:46 sysinit: Error: Write failed

Mar 15 08:23:46 sysinit: Error: Write failed

Mar 15 08:23:47 sysinit: Error: Read failed

Mar 15 08:23:47 sysinit: Error: Read failed

Mar 15 08:23:48 sysinit: Error: Write failed

Mar 15 08:23:48 sysinit: Error: Write failed

Mar 15 08:23:49 sysinit: Error: Read failed

Mar 15 08:23:49 sysinit: Error: Read failed

Mar 15 08:23:50 sysinit: Error: Write failed

Mar 15 08:23:50 sysinit: Error: Write failed

Mar 15 08:23:51 sysinit: Error: Read failed

Mar 15 08:23:51 sysinit: Error: Read failed

Mar 15 08:23:52 sysinit: Error: Write failed

Mar 15 08:23:52 sysinit: Error: Write failed

Mar 15 08:23:53 sysinit: Error: Read failed

Mar 15 08:23:53 sysinit: Error: Read failed

Mar 15 08:23:54 sysinit: Error: Write failed

Mar 15 08:23:54 sysinit: Error: Write failed

Mar 15 08:23:55 sysinit: Error: Read failed

Mar 15 08:23:55 sysinit: Error: Read failed

Mar 15 08:23:56 sysinit: Error: Write failed

Mar 15 08:23:57 sysinit: Error: Write failed

Mar 15 08:23:57 sysinit: Error: Read failed

Mar 15 08:23:58 sysinit: Error: Read failed

Mar 15 08:23:58 UI: i2c error setting 0x47 14 147

Mar 15 08:23:58 sysinit: Error: Read failed

Mar 15 08:23:58 sysinit: killall: telnetd: no process killed

Mar 15 08:23:59 sysinit: 2018-03-15 08:23:59: (log.c.216) server started

Mar 15 08:24:01 dropbear[941]: Running in background

Mar 15 08:24:04 switch[971]: Detected cold boot

Mar 15 08:24:06 Port: link state changed to 'down' on port 2

Mar 15 08:24:06 Port: link state changed to 'down' on port 3

Mar 15 08:24:06 Port: link state changed to 'down' on port 9

Mar 15 08:24:06 switch[971]: PoE enabled on port 1, PoE Smart is starting cable check

Mar 15 08:24:06 Port: link state changed to 'down' on port 10

Mar 15 08:24:06 Port: link state changed to 'down' on port 11

Mar 15 08:24:06 switch[971]: PoE enabled on port 2, PoE Smart is starting cable check

Mar 15 08:24:06 Port: link state changed to 'down' on port 17

Mar 15 08:24:06 Port: link state changed to 'down' on port 13

Mar 15 08:24:06 Port: link state changed to 'down' on port 14

Mar 15 08:24:06 switch[971]: PoE enabled on port 3, PoE Smart is starting cable check

Mar 15 08:24:06 Port: link state changed to 'down' on port 15

Mar 15 08:24:06 Port: link state changed to 'down' on port 16

Mar 15 08:24:06 Port: link state changed to 'down' on port 18

Mar 15 08:24:06 Port: link state changed to 'down' on port 1

Mar 15 08:24:07 Port: link state changed to 'down' on port 4

Mar 15 08:24:07 switch[971]: PoE enabled on port 4, PoE Smart is starting cable check

Mar 15 08:24:07 Port: link state changed to 'down' on port 5

Mar 15 08:24:07 switch[971]: PoE enabled on port 5, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 6, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 7, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 8, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 9, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 10, PoE Smart is starting cable check

Mar 15 08:24:07 switch[970]: retrying i2c link to power supply

Mar 15 08:24:08 switch[971]: PoE enabled on port 11, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 13, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 14, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 15, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 16, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 17, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 18, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 19, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 20, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 21, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 22, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 23, PoE Smart is starting cable check

Mar 15 08:24:11 switch[971]: Port 1 cable check results: Open, Open, Open, Open

Mar 15 08:24:11 switch[971]: Port 1 passed PoE Smart cable check, turning on power

Mar 15 08:24:11 switch[970]: retrying i2c link to power supply

Julian
 

Re: i2c Errors on WS-26-500-DC Switches

Fri Mar 16, 2018 12:01 pm

we are actively pursuing a resolution on this issue, please monitor upcoming firmware releases.

User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: i2c Errors on WS-26-500-DC Switches

Sat Mar 17, 2018 1:23 pm

We have been testing the WS-26-500-DC sent back from Intellipop and so far no reboots up time 1 day 19+ hours.

We have changed the input voltages from 24V to 52V during this test with no reboots but the CPU Utilization is way too HIGH.

We are "suspecting" that this may be the cause. Eric has optimized the code to reduce CPU to 30% +/- which we think will greatly help.

A theory of what might be happening is the CPU is getting bogged down and possibly not responding to the watchdog which forces a reboot or other adverse issues resulting in a reboot.

The fact that it is not rebooting in our LAB even with HIGH CPU is probably that there is nothing else going on so the high CPU is not an issue.

Anyway below you can see Intellipops switch with HIGH CPU usage and under similar POE load as he had deployed it and also another unit with the new optimized code with LOW CPU utilization.

So anyway will be hopefully releasing v1.5.0rcX today or tomorrow.

v1.4.9
26-1.png


v1.5.0rcX
26-2.png
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

aaron@futuretk.com
Member
 
Posts: 20
Joined: Mon Feb 13, 2017 10:43 am
Has thanked: 0 time
Been thanked: 0 time

Re: i2c Errors on WS-26-500-DC Switches

Sun Mar 18, 2018 9:46 am

Good to hear. Keep us posted. I have one switch in particular that is now rebooting several times a day so I'd be happy test the new firmware, and would likely know very soon if it solves our issue.

User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: i2c Errors on WS-26-500-DC Switches

Sun Mar 18, 2018 10:22 am

aaron@futuretk.com wrote:Good to hear. Keep us posted. I have one switch in particular that is now rebooting several times a day so I'd be happy test the new firmware, and would likely know very soon if it solves our issue.


Will be happy to have you try this but your issue is slightly different.

You are seeing I2C errors along with reboots.

The firmware may help with the reboots - hopefully

But the I2C issue may be related to hardware issue, possible causes:
Lose, bad connection on I2C cable between switchboard and power supply
Bad I2C cable between switch and power supply
Bad component(s) on either board.

However complete I2C failure should not cause reboot (I think, need to confirm with Eric) but simply you lose telemetry from power supply and or I2C sensors on switch board.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

aaron@futuretk.com
Member
 
Posts: 20
Joined: Mon Feb 13, 2017 10:43 am
Has thanked: 0 time
Been thanked: 0 time

Re: i2c Errors on WS-26-500-DC Switches

Sun Mar 18, 2018 9:00 pm

Can you email me a copy of this new firmware, or do I need to wait for General release?

User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: i2c Errors on WS-26-500-DC Switches

Mon Mar 19, 2018 2:54 pm

We released a new version to possibly help with issues for the WS-26-500-DC

You can download v1.5.0rcX here: viewtopic.php?f=17&t=240&p=848#p848

Please let us know if it helps.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

aaron@futuretk.com
Member
 
Posts: 20
Joined: Mon Feb 13, 2017 10:43 am
Has thanked: 0 time
Been thanked: 0 time

Re: i2c Errors on WS-26-500-DC Switches

Tue Mar 20, 2018 10:14 am

This did drastically reduce the CPU utilization. So far so good. I'll know more in a few hours.

User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: i2c Errors on WS-26-500-DC Switches

Wed Mar 21, 2018 3:09 pm

aaron@futuretk.com wrote:This did drastically reduce the CPU utilization. So far so good. I'll know more in a few hours.


So Aaron please PM me your cell number since your issue is a little differnt I want to talk to your about some things to look for and some possible fixes for I2C error.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

Return to Hardware and software issues

Who is online

Users browsing this forum: No registered users and 18 guests