Netonix Keeps Rebooting on 1.5.0

DOWNLOAD THE LATEST FIRMWARE HERE
User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: Netonix Keeps Rebooting on 1.5.0

Fri Jan 18, 2019 5:12 pm

mhoppes wrote:My problem is when it happens it's increadibly sporadic.... .I'll see what I can catch next time it happens.


Here are some questions;

1) Does it happen on all your switches?
If the answer is no then probably something local to this Layer 2 segment

2) Did you try disabling all services not 100% essential including SNMP/Discovery/SMTP/SYSLOG and monitor ,manually?
If you have not then do so.

3) You say it did not happen with v1.4.9 and these are OLD switches so did you try downgrading to v1.4.9 leaving all configurations the same and see?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mhoppes
Associate
Associate
 
Posts: 658
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 124 times

Re: Netonix Keeps Rebooting on 1.5.0

Fri Jan 18, 2019 5:14 pm

1) I have very randomly and very very sparcly see it happen elsewhere -- not often enough to comment on or be concerned with. It has happened too many times on this segment.

2) I just disabled everything this afternoon.

3) When I downgrade to 1.4.9 I start having all kinds of major momentary glitches in data flow and spanning tree flaps. On 1.5.0 that doesn't happen, but the switch will reboot from time to time. Now that I think about it... it did happen with the old switch, which was on 1.4.9 from time to time... so it's NOT something purely new on 1.5.0.

I'll keep an eye on it and let you know of any conclusions or further details I figure out.

User avatar
mhoppes
Associate
Associate
 
Posts: 658
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 124 times

Re: Netonix Keeps Rebooting on 1.5.0

Wed Jan 23, 2019 3:16 pm

So it happened again.. same issue. The switch is on a private IP (though not firewalled). I've disabled STP as well as all discovery protocols and enabled administrative controls on IP addresses that can access the switch.

I'm also running a continual top and tail -f /var/log/messages

Eric, where is the syslog kept? In memory? is it ever purged? Is it possible that some kind of error or attack against the switch would overwhelm the memory through syslog messages and ultimately cause the switch to reboot?

User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: Netonix Keeps Rebooting on 1.5.0

Wed Jan 23, 2019 4:13 pm

Eric is out of town. But I am talking to Stephen to see if he can answer your questions regarding syslog but I think it is in memory and if not still lost on a reboot.

I still think this is a bad piece of equipment somewhere spewing packets or something.
Getting an alert of memory usage spiking and then checking port statistics and even looking at TOP would be a big help.

If there are SFP modules in this unit I would swap them out.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mhoppes
Associate
Associate
 
Posts: 658
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 124 times

Re: Netonix Keeps Rebooting on 1.5.0

Wed Jan 23, 2019 4:23 pm

By Job! This issue may have started when we added a MikroTik SFP.

If that’s the issue. And it’s a big if. Why are there so many issues with SFPs with these switches? I have another one where if I even bump the copper SFP everything just goes down.

User avatar
sirhc
Employee
Employee
 
Posts: 7347
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1597 times
Been thanked: 1317 times

Re: Netonix Keeps Rebooting on 1.5.0

Wed Jan 23, 2019 4:27 pm

mhoppes wrote:By Job! This issue may have started when we added a MikroTik SFP.

If that’s the issue. And it’s a big if. Why are there so many issues with SFPs with these switches? I have another one where if I even bump the copper SFP everything just goes down.


If you go back and read the post on Facebook which linked here: viewtopic.php?f=17&t=3623&p=26619&hilit=+sfp+reboot#p26619

It is NOT a compatibility issue it is a BAD SFP module.

I also told you on Facebook that I have a fiber media converter that is bad that I can plug into "any" Layer 2 segment and it will within an few hours lock up or reboot "all" managed devices even Cisco top of the line switches.

DID YOU NOT READ THE POSTS I MADE?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
Stephen
Employee
Employee
 
Posts: 965
Joined: Sun Dec 24, 2017 8:56 pm
Has thanked: 77 times
Been thanked: 169 times

Re: Netonix Keeps Rebooting on 1.5.0

Wed Jan 23, 2019 6:13 pm

Just double checked to be certain.

/var/log/messages is lost on reboot but the file is not allowed to grow past 200K in memory.
After it fills up 200K it starts throwing away oldest log entries. - FIFO

Otherwise the log is lost after reboot.

deerefanatic
Member
 
Posts: 3
Joined: Mon Jan 02, 2017 1:45 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Netonix Keeps Rebooting on 1.5.0

Fri Feb 01, 2019 6:11 pm

I see the same Issues with Rebooting at random... May go months and then BAM reboot.

User avatar
mhoppes
Associate
Associate
 
Posts: 658
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 124 times

Re: Netonix Keeps Rebooting on 1.5.0

Tue Feb 05, 2019 9:46 am

Chris, can I get an answer from Eric on where the log file is kept? Is it kept in memory or somewhere else?

In other words -- some kind of an error or attack against a switch -- would it potentially overwhelm the onboard memory if the log fie was made to grow?

NOW that being said...since disabled STP and adding access controls the switch has not recently rebooted. I'm going to let it continue to go to see what happens. However, this is the longest I've ever seen it stay up without a reboot in recent times. I'll probably let it go to the 60 day mark before I revert any changes to see what happens.

That being said.. for the last two years I have been trying to track down some instability in a network segment that runs through this switch. For no explainable reason we would suddenly lose connectivity for a minute and then it would return. Switch logs would sometimes show nothing, other times it would show an STP hit on a port that definitely has no loop.

Since turning STP off on all three switches on this network segment the segment has run smooth as a baby's bottom without crashing and without going down.

Now riddle me this batman..... if there actually were loops wouldn't I still be seeing packet storms that would crash the network? Why is STP firing off when there are no (apparent) loops.... also loop control isn't firing off either with STP disabled.

User avatar
Eric Stern
Employee
Employee
 
Posts: 532
Joined: Wed Apr 09, 2014 9:41 pm
Location: Toronto, Ontario
Has thanked: 0 time
Been thanked: 130 times

Re: Netonix Keeps Rebooting on 1.5.0

Tue Feb 05, 2019 1:43 pm

mhoppes wrote:Chris, can I get an answer from Eric on where the log file is kept? Is it kept in memory or somewhere else?

In other words -- some kind of an error or attack against a switch -- would it potentially overwhelm the onboard memory if the log fie was made to grow?


Its stored in /tmp/log/messages, which is technically in memory because /tmp is a ramdisk. But the file is rotated once it exceeds 128kB so it will not grow to fill all of memory.

PreviousNext
Return to Hardware and software issues

Who is online

Users browsing this forum: No registered users and 21 guests