Switch loads into BusyBox via Console only

[yahel]

Dear Netonix people...

I have a switch that stopped responding to ping or web-GUI but was passing traffic.
Upon site visit and reboot, it never recovered - no link-state on any ports.

Back in the lab, via console -- it throws many screens of errors and ends up in a BusyBox shell.
It still remembers the name we gave it "PATh-Netonix", but not much else...

Code: Select all

 BusyBox v1.11.2 (2014-09-17 17:18:08 EDT) built-in shell (ash)

Enter 'help' for a list of built-in commands.

 
admin@PATh-Netonix:/# df -h

Filesystem
Size
Used Available Use% Mounted on

rootfs 1.2M 1.2M0 100% /
/dev/root 1.2M 1.2M 0 100% /rom
/dev/mtdblock6 4.3M 1.4M 2.9M 33% /jffs
mini_fo:/jffs 1.2M 1.2M 0 100% /

admin@PATh-Netonix:/# ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

admin@PATh-Netonix:/# ps -aux
PID USER VSZ STAT COMMAND
1 admin 2012 S init
2 admin 0 SW< [kthreadd]
3 admin 0 SW< [ksoftirqd/0]
4 admin 0 SW< [watchdog/0]
5 admin 0 SW< [events/0]
6 admin 0 SW< [khelper]
19 admin 0 SW< [kblockd/0]
43 admin 0 SW [pdflush]
44 admin 0 SW [pdflush]
45 admin 0 SW< [kswapd0]
46 admin 0 SW<[aio/0]
56 admin 0 SW<[mtdblockd]
59 admin 0 SW< [spi-vcoreiii.0]
267 admin 0 SWN [jffs2_gcd_mtd6]
278 admin 2024 R /bin/ash --login
289 admin 2136 S syslogd -S -C128 -L -R 192.168.233.6:514
291 admin 2008 S klogd
303 admin 1132 S /sbin/hotplug2 --override --persistent --max-children
396 admin 1936 S /usr/sbin/dropbear -p 22
401 admin 2008 S /usr/sbin/httpd -p 80 -h /www -r PATh-Netonix
412 admin 2012 R ps -aux

admin@PATh-Netonix:/# lsmod

Module
Size
Used by
Tainted: P


i2c_vcoreiii
6096
0


i2c_dev
5904
0


i2c_core
16336
2 i2c_vcoreiii,i2c_dev

admin@PATh-Netonix:/#

admin@PATh-Netonix:/# cd /lib/modules/2.6.26.8/

admin@PATh-Netonix:/lib/modules/2.6.26.8# ls -al

drwxr-xr-x
2 admin
root
134 Sep 20
2014
[b].[/b]

drwxr-xr-x
3 admin
root
25 Sep 20
2014
[b]..[/b]

-rw-r--r--
1 admin
root
25960 Sep 20
2014 i2c-core.ko

-rw-r--r--
1 admin
root
10132 Sep 20
2014 i2c-dev.ko

-rw-r--r--
1 admin
root
9900 Sep 20
2014 i2c-vcoreiii.ko

-rw-r--r--
1 admin
root
829636 Sep 20
2014 vtss_core.ko

-rw-r--r--
1 admin
root
24176 Sep 20
2014 vtss_ethdrv.ko

-rw-r--r--
1 admin
root
121604 Sep 20
2014 vtss_ioctl.ko

-rw-r--r--
1 admin
root
18708 Sep 20
2014 vtss_port.ko

admin@PATh-Netonix:/lib/modules/2.6.26.8#


admin@PATh-Netonix:/lib/modules/2.6.26.8# insmod vtss_core.ko

switch: 'Luton26' board detected

admin@PATh-Netonix:/lib/modules/2.6.26.8#


admin@PATh-Netonix:/lib/modules/2.6.26.8# insmod vtss_port.ko


vtss_port: vtss_phy_reset failed, port_no: 12

vtss_port: vtss_phy_conf_set failed, port_no: 12

vtss_port: vtss_phy_reset failed, port_no: 13

vtss_port: vtss_phy_conf_set failed, port_no: 13

vtss_port: vtss_phy_reset failed, port_no: 14

vtss_port: vtss_phy_conf_set failed, port_no: 14

vtss_port: vtss_phy_reset failed, port_no: 15

vtss_port: vtss_phy_conf_set failed, port_no: 15

vtss_port: vtss_phy_reset failed, port_no: 16

vtss_port: vtss_phy_conf_set failed, port_no: 16

vtss_port: vtss_phy_reset failed, port_no: 17

vtss_port: vtss_phy_conf_set failed, port_no: 17

vtss_port: vtss_phy_reset failed, port_no: 18

vtss_port: vtss_phy_conf_set failed, port_no: 18

vtss_port: vtss_phy_reset failed, port_no: 19

vtss_port: vtss_phy_conf_set failed, port_no: 19

vtss_port: vtss_phy_reset failed, port_no: 20

vtss_port: vtss_phy_conf_set failed, port_no: 20

vtss_port: vtss_phy_reset failed, port_no: 21

vtss_port: vtss_phy_conf_set failed, port_no: 21

vtss_port: vtss_phy_reset failed, port_no: 22

vtss_port: vtss_phy_conf_set failed, port_no: 22

vtss_port: vtss_phy_reset failed, port_no: 23

vtss_port: vtss_phy_conf_set failed, port_no: 23

admin@PATh-Netonix:/lib/modules/2.6.26.8#


admin@PATh-Netonix:/lib/modules/2.6.26.8# insmod vtss_ioctl.ko

admin@PATh-Netonix:/lib/modules/2.6.26.8# insmod vtss_ethdrv.ko

admin@PATh-Netonix:/lib/modules/2.6.26.8# ifconfig

eth0
Link encap:Ethernet
HWaddr 02:03:04:05:06:07
 
inet addr:172.18.120.2
Bcast:172.18.120.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:85 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:33 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:1000


 
RX bytes:6730 (6.5 KiB)
TX bytes:2738 (2.6 KiB)

 
eth0.1004 Link encap:Ethernet
HWaddr 02:03:04:05:06:07


 
inet addr:10.6.12.254
Bcast:10.6.12.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:398 (398.0 B)

 
eth0.1005 Link encap:Ethernet
HWaddr 02:03:04:05:06:07


 
inet addr:10.6.13.254
Bcast:10.6.13.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:398 (398.0 B)

 
eth0.101
Link encap:Ethernet
HWaddr 02:03:04:05:06:07


 
inet addr:172.18.221.253
Bcast:172.18.221.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:398 (398.0 B)

 
eth0.103
Link encap:Ethernet
HWaddr 02:03:04:05:06:07


 
inet addr:172.18.122.254
Bcast:172.18.122.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:308 (308.0 B)

 
eth0.112
Link encap:Ethernet
HWaddr 02:03:04:05:06:07


 
inet addr:172.18.123.253
Bcast:172.18.123.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:398 (398.0 B)

 
eth0.2222 Link encap:Ethernet
HWaddr 02:03:04:05:06:07


 
inet addr:10.6.11.254
Bcast:10.6.11.255
Mask:255.255.255.0

 
inet6 addr: fe80::3:4ff:fe05:607/64 Scope:Link

 
UP BROADCAST RUNNING MULTICAST
MTU:1500
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:308 (308.0 B)

 
lo
Link encap:Local Loopback


 
inet addr:127.0.0.1
Mask:255.0.0.0

 
inet6 addr: ::1/128 Scope:Host

 
UP LOOPBACK RUNNING
MTU:16436
Metric:1

 
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

 
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

 
collisions:0 txqueuelen:0


 
RX bytes:0 (0.0 B)
TX bytes:0 (0.0 B)

 
admin@PATh-Netonix:/lib/modules/2.6.26.8

Anyways - that's where I stopped...
Apologies for the formatting....

No Ethernet link on any port - even after "ifconfig eth0 up"...


I think the /rom filesystem is smaller than it should be...

Suggestions?

Thanks,

Yahel

[sirhc]

Try power on factory default, doubtful but why not.

Try console cable recovery.

If within 1 year old RMA repair it, if over a year send to Julian for repair. A re-flash may fix it.

[Julian]

This secondary SoC didn't come out of reset properly on power cycle(vtss_phy_* calls in crawl) so switch(our app, not the OS) can't init fully or run.

There are almost an infinite number of reasons this can go on, in my experience it neither occurs nor clears spontaneously. Unchecked guesses don't do much, so will decline to comment in real detail until I gain firsthand knowledge via failure analysis - There is 100% a physical reason here, most often there are symptoms of overcurrent present in units which present thus.

Happy to take a look, let me know, sir! rma.netonix.com is that way <--

[yahel]

1. Not sure what does this means: "power on factory default,"
Assuming it means to hold the reset button while powering on - I did... No change.

2. Obviously I cannot do a console recovery - because there's no Red-boot access...

Is there a way to clear the nvram from the Linux command-line, or anything to get it into Red-boot so I can follow console-recovery?

Otherwise - I would go with sending it for repair... But I'm still waiting for the 26-port switch to come back to us....
Been VERY long (see post: viewtopic.php?f=6&t=5701&start=30#p39434 ).

Thanks,

Yahel.

[sirhc]

Watch console as it boots it prompts to bail the boot process and stop at boot loader.

Yes power on default you described correctly.