Netonix Boot Loop Issue When Powering ePMP 3000L Radios

[Julian]

Oh good, these are the kinds of puzzles I live for, sir. please take no offense to any of my autistic questions, I'm trying to be of use, here, i promise.

I don't have one here currently for pictures but power-wise 3kl's basically a buck converter that puts out 5.5v, with 36v TVS's across the bob smiths for surge suppression. I think just about everyone radio-wise uses a similar circuit, the specific flavors of part can change but just about everyone's center-tap injected with a bob smith network for noise and TVS's for surge suppression.

Just about any part in that *can* go, but the "pick-a-pair sometimesyworks" you're describing I have not seen yet. Personally, that would have me ready to throw my hat.

how's your in-house twidgetry, is there an oscilloscope over there? I can ask around to borrow a few radios tomorrow but it will be next week before I get anything set up at a test site to try and duplicate

[gunther_01]

I think I may have a small circuit board style oscilloscope.. Other than that, no..

I am about at my witts end.. We can't figure out what is making these devices go nuts yet. Other than we now know we had most likely a bad negative power cable for the netonix power.. We think (serious theorize) that with the ground going bad on the netonix. We might have somehow fried the radios. Or maybe even the netonix ports somewhat. Which is now causing us to chase our tail on what is or isn't broken.

So far, two cambium switches, testing and working with 2 netonix, and a couple 3kls on the tower.. But, when we test the 3kls' they may or may not work in a netonix on the ground on the bench. Some ports seem to work better than others. For either radio.. It's just danged odd.. Only thing we do know, is the netonix does a reboot when it doesn't like whatever is happening.. Thus, why I'm trying to figure out why a netonix does reboot. What reasons cause that..? Especially since these ports seem to still be functioning just fine (power up and have 1g links) when they do decide to work.. I am familiar with blown ports that are just dead.. either no power or link, etc...

[Julian]

I sent you an email with some test points and command line stuff to look at; that scope'll work well enough for this, anything that would cause a whole reboot you would be able to see with the $16 DSO138 I started out with and still wire up every now and again, I'm sure if you acquired yours within the last decade it's probably the same as mine.

Yes bad ground will 100% make you think that maxwell's demon has been summoned to haunt you and you alone in perpetuity

[gunther_01]

Just to post a reply to this.

We found some worn dc power cables going up this tower.. Some on the positive, and also on the negative. At some point, we lost the negative I believe..

There was also some cat 5 cables that didn't test out in the Netonix test. They did on a hand held tester. So, we re-terminated those. One tested good at the netonix. The other still showed an imbalance in the length of pairs for some reason though.. Nothing visible we could find wrong.

So far, we have had no problem on this switch since fixing those issues.

[james@kiwiwifi.nz]

Adding to this topic as we have a very similar issue. Recently pulled 4 x 3000L from 2 different towers. T1 1 x WS-25-500DC 1.5.14, T2 1 x WS-12-250DC 1.5.14.

All 4 have been working fine for over 3 years. We replaced them with new ePMP 4500.

I reset one for installation at another tower T3 to replace an ageing Mikrotik sector. When we turned on the 24v the switch went into a reboot loop. Reboot starts at around 5-10s after the save button has been hit. The switch at T3 is a WS-12-250DC 1.5.26.

We returned this 3000L to the bench and I tested all 4 on two spare switches we have. One a WS-12 and the other a WS-25 both DC. I can reliably replicate the rebooting on both switchs each 5-10 sec after you save the settings. Interestingly one of the 3000L doesn't cause the switchs to reboot all the time, but it does sometimes which makes me think this is a electrical tolarance issue. I've tested the following firmware 1.5.12, 1.5.14, 1.5.26 and 1.5.27. The switchs reboot on all firmware aside 1.5.12. They DO NOT reboot with 1.5.12. Strangley I would have thought 1.5.14 would be fine also as our towers T1 & T2 have that firmware, but since the switch uptime is over a year on those switches there could be other variables that have changed.

I've not tested different cambium firmware yet. However, I have found that if you put a Mikrotik GPeR in line with the switch and the 3000L the reboots do not occour on any of the firmwares.

So, there must be somthing in the newer firmwares that affect the tolerance of the switches when they use the 24v power rail and the 3000L. The switches power up the Mikrotik router on 24v just fine. They also power up on 48v 3000 and 4500 sectors just fine as well. So, it's very much a 3000L issue.

We need a solution for this as we want to use the radios to replace our ageing Mikrotik sectors. Is there anything I can provide to help Netonix solve this please let me know.

[Julian]

Well, we got mr. gunther up with some new terminations and cables, really perplexing why 1.5.12 is working and others don't? maybe someone smarter than I will chime in, and we'll have a party.

I think a flight out to NZ is out of the question, but my email's julian@netonix.com and I'll do what I can from here; Same question, is there an oscilloscope handy (or a meter that'll read DC ripple; sometimes they'll say Vpk-pk)?

[james@kiwiwifi.nz]

For a bit of fun I've challenged Googles new Gemini 4 Argon with this problem and this it's response:


This behavior points to an electrical and timing interaction between the **Cambium ePMP 3000L's PoE input stage**, the **Netonix WS-12-250-DC's 24V power rail / PoE Smart architecture**, and how the switch firmware sequences port power-up and I2C telemetry polling. Other WISP operators have reported this exact reboot loop when powering ePMP 3000L radios on Netonix DC switches in the Netonix Community Forums

### 1. Why the 24V Rail Takes Down the Entire Switch
On the [Netonix WS-12-250-DC](https://netonix.com/ws-12-250-dc.html), the internal smart DC-DC power supply generates a 48V rail and a 24V rail (rated up to 6A / 144W max). Crucially, on the Netonix board architecture, **the 24V rail also feeds the step-down regulators for the switch’s own logic, CPU (MIPS SoC), and I2C telemetry bus**.
Your ePMP 3000, ePMP 4500, and RACOM Ray 18GHz backhaul all run off the 48V/48VH rail, so their startup transients do not directly disturb the 24V bus. While the MikroTik RB4011 does use 24V, its input stage has a relatively gentle, linear startup characteristic. By contrast, if anything causes a sharp transient voltage droop, high-frequency ripple, or a momentary fault on the 24V bus when a 3000L port is enabled, it either starves the switch's logic rail (causing an immediate cold hardware brownout reset) or corrupts the internal I2C bus between the switch board and the smart DC-DC PSU microprocessor, triggering a hardware watchdog reboot a few seconds later.

### 2. What Makes the ePMP 3000L Different
Unlike standard passive 24V devices that simply take Pair 3 (pins 4,5) as Positive and Pair 4 (pins 7,8) as Negative directly into a buck converter, the Cambium ePMP 3000L is **polarity-agnostic** (accepting standard or reverse polarity, as well as power on data pairs) and feeds an internal 5.5V buck converter with 36V TVS diodes across the Bob Smith termination network.

To achieve polarity-agnostic Gigabit PoE, the 3000L extracts power from the center-taps of all four Gigabit Ethernet transformer pairs through a full diode bridge rectifier into its input bulk capacitors. When 24V is switched onto the port:

1. **Capacitive Inrush & Buck Converter Start:** The discharged input capacitance behind the bridge rectifier, combined with the negative incremental input impedance of the 3000L's DC-DC buck converter as it begins switching at low voltage, creates a very sharp, brief current spike ($di/dt$) and high-frequency switching noise back onto the 24V PoE pairs.

2. **PoE Smart Detection Conflict:** As documented in [Netonix's PoE Smart guidelines](viewtopic.php?f=6&t=1178#p8809), before applying full passive PoE, the Netonix switch runs a "PoE Smart" cable check across the pairs to look for cross-shorts. Netonix specifically notes that certain Cambium radios have input stages (bridge rectifiers and center-tap impedances) that confuse the PoE Smart check when unpowered.

### 3. Why Firmware 1.5.12 Works While 1.5.14 and 1.5.26 Reboot
When you click "Save/Apply" to turn on 24V in the Netonix UI, the switch does not just flip a dumb mechanical relay. The MIPS CPU runs a sequence: it reconfigures the port MOSFETs, runs the PoE Smart check (unless disabled), enables the 24V MOSFET, negotiates the Gigabit PHY link, and simultaneously polls the per-port current sensors and the
DC-DC power supply over the internal I2C bus.

Between firmware [v1.5.12](viewtopic.php?f=17&t=7371) and [v1.5.14](viewtopic.php?f=17&t=7423) / v1.5.26, subtle changes in how the switch daemon (`vtss` / switch control process) sequences port configuration, PHY reset, and I2C sensor polling alter the exact timing window when the port is turned on. If the switch polls the I2C bus or triggers a PHY state change at the exact millisecond the 3000L's buck converter fires up and injects switching noise/droop onto the 24V rail, the I2C communication hangs or the CPU watchdog trips—explaining why the switch consistently reboots a few seconds after applying the setting.
Additionally, if your existing towers running 1.5.14 have been up for a long time without issue, it is often because those ports were already enabled or staggered cleanly during a cold boot, whereas enabling the 24V port live via the UI (or during a warm reboot loop where all ports initialize under full load) hits the exact timing/inrush window.

### 4. Why 1 Out of 4 Radios Works (and Why It Is Inconsistent)
Because the issue sits right on a hardware electrical threshold (either the 24V regulator's transient response limit, the polyfuse/MOSFET turn-on curve, or the I2C noise margin), normal **component tolerances** make the difference between passing and crashing.
Electrolytic and MLCC input capacitors inside the 3000L radios commonly have $\pm 10\%$ to $\pm 20\%$ capacitance and ESR variations, especially if the radios have seen several years of tower service. A radio with slightly higher ESR or slightly lower capacitance draws a slightly softer inrush peak, staying just underneath the threshold that trips the Netonix 24V rail or scrambles its I2C telemetry. Similarly, the exact millisecond in the PSU's switching cycle when the MOSFET closes determines whether a borderline radio boots cleanly or triggers a reboot.

### 5. Why the MikroTik GPeR Completely Fixes the Issue
Inserting the [MikroTik GPeR](https://help.mikrotik.com/docs/spaces/U ... 22393/GPeR) between the Netonix switch and the ePMP 3000L changes the electrical circuit in three ways that completely isolate the Netonix from the 3000L's startup behavior:
First, **complete galvanic isolation of the Ethernet transformers and PHY**: Instead of the Netonix port's Ethernet magnetics connecting directly to the 3000L's magnetics and bridge rectifier, the Netonix terminates into the GPeR's Gigabit Ethernet transformer. Any common-mode noise, Bob Smith termination interaction, or PHY auto-negotiation surge from the 3000L is blocked by the GPeR's magnetics.

Second, **inrush current limiting and RC filtering**: In passive PoE passthrough mode, the GPeR extracts the 24V power from the center-taps of its "PoE In" transformer, passes it through the GPeR's own internal current-limiting circuitry (rated at 2A max at 24V), jumper pins, and local decoupling capacitors, and re-injects it into the center-taps of the "PoE Out" transformer. The series DC resistance and inductance of two extra sets of transformer windings plus the GPeR's pass-through protection act as an **RLC low-pass filter / soft-start buffer**. It smooths out the sharp $di/dt$ inrush spike of the 3000L's buck converter and blocks high-frequency switching ripple from travelling back into the Netonix's 24V rail.

Third, **staggered load presentation**: When the Netonix enables 24V, it first charges the GPeR's tiny ~1.5W–2W circuit and local capacitance, damping the instantaneous step-load seen by the Netonix port MOSFET.
### Practical Workarounds & Diagnostic Steps to Try
If you want to run the 3000L radios on newer firmware without keeping a GPeR inline on every port, there are a few targeted adjustments worth testing on your bench setup:
1. **Disable "PoE Smart" on the 3000L Ports:** On the Netonix **Ports** tab, uncheck the **PS** (PoE Smart) box for the ports feeding the 3000L radios (after verifying your cable has no cross-shorts). Netonix specifically recommends disabling PoE Smart for Cambium radios whose input circuits interfere with the switch's pre-power cable check.

2. **Check Grounding / Bond Neutral Behavior:** Ensure the Ethernet surge arrestors (if any are inline—Netonix strongly advises *against* using inline gas-tube/clamping Ethernet surge protectors on their passive PoE ports as they leak current across pairs) are removed for testing, and check whether you are using shielded RJ45 ends that bond the radio chassis ground directly to the switch chassis ground, as ground loops on non-isolated DC switches (`WS-12-250-DC` has a non-isolated DC negative) can feed return current back through the Ethernet shield or transformer center-taps.

3. **Measure 24V Ripple:** If you have an oscilloscope (or a multimeter with AC ripple / Peak-to-Peak measurement), probing the 24V rail or viewing the serial console output at 115200 baud during the 5–10 second window after clicking Save will immediately show whether the reboot is an instantaneous voltage drop (no serial log before boot) or an I2C/watchdog timeout.
-------------------------------------------------------------------------------------

I can confirm disabling the PoE Smart does not work. It just turns the port on faster as it doesnt dot the smart check.
Our bench test setup has both the radios and switch earthed.
We don't have the equipment to measure the ripple. However, we're posting away a few Netonix switches to Canada to get repaired and we're sending a 3000L along with them so the repair guy can get to the bottom of the problem.