I’ve been trying to track down some random Throttle Controller resets (rp2040 version). What seems to happen is that the controller just resets – there is no power-up beep from the ESC (so I don’t think it’s losing power), and there are no “quick beeps” from the ESC (so it’s not noticing the lack of control signal). But the Throttle Controller shows the boot screen and goes from armed to disarmed and cuts the throttle command (obviously) so the motor stops.
I’ve had half-a-dozen resets or so at this point, all on the ground (thank goodness)
I’ve had the battery fully charged (2 kWh battery), and also down near storage levels (about 85%)
I had been testing my own firmware, but reverted to the stock v5.8R and had three more cases
I was able to confirm that the reset isn’t being caused by the watchdog
I monitored the input power to the controller, and was able to confirm that it stays near 10V the whole time (actually goes up a little bit while the reset is happening)
The simplest way for me to reproduce is just to power on the system, arm it, throttle up to spin the motor to get the counter started, then release the throttle… and then check on it periodically to see if it’s still armed. I’ve had resets happen in ~5 minutes, and sometimes it takes up to ~35 minutes.
At this point, I don’t know what it could be! I still wonder about power transients… or maybe some sort of fault (NaNs? divide-by-zero?) in the code? Or is my controller just bad?
I think someone else mentioned something like this recently, but I can’t find the thread any more!
Wow, I think I found the issue (or at least an issue) with the throttle controller (TC).
I’ve been wondering if it might have to do with the local electrical/RF environment, because I felt like it was more likely to happen if my cell phone was a WiFi hotspot, or if I was using Bluetooth to listen to music or talk to the BMS. I know this sounds superstitious, but bear with me!
So I got out my trusty little Baofeng, changed to FRS1 channel… and any time I hit the transmit button, it completely powers off the TC (screen goes black, ESC starts beeping due to no connection). This is more extreme than the random resets that I’ve been seeing, where the TC boots right back up.
I’m going to experiment with range to see if I can reproduce the “reset” behavior.
Has anyone else tried using VHF radios near the TC?
(I now have poor-quality videos of all these faults happening, in case that would be helpful)
That is impressive you were able to hopefully narrow it down to a frequency. I was talking on my phone using Bluetooth during take-off when my TC rebooted and have not used Bluetooth since and have had no issues.
Oh, I don’t think it’s any specific frequency – it’s just that the VHF radio is really unequivocal, and I picked a standard channel to be clear about my experimental method
I think my cell phone, or even RF interference from the ESC or motor itself could eventually cause resets… its just that the chances are lower.
A few updates. First a caveat: I am not an electrical engineer, so I might be jumping to the wrong conclusions.
I’ve pared my test setup down to the minimum: the TC mainboard (without display) and an external power supply. RFI still causes all three fault types (quick reset, slow reset, hard off). Leaving the board alone for 10-100 minutes pretty reliably yields a quick reset.
I’ve checked for any obvious solder or connectivity issues and can’t find anything. It’s still possible that my TC is defective in some way – in fact, I hope it is, because then a replacement would address this issue!
From reading about rp2040 hardware design (see https://datasheets.raspberrypi.com/rp2040/hardware-design-with-rp2040.pdf), I’m starting to suspect the distance between the rp2040 and the flash chip might be an issue – it’s called out explicitly in that document. When I used a scope to probe the flash QSPI control lines… it looked pretty messy (again, I am not an expert!):
Great investigation Nfairfield, I can’t give much input here because I do not experience random controller resets. I am running FW V5.4. with PCB rev 3 and the ATMEL chipset. I have 199Hrs on my paramotor and have never experienced any of the random resets you describe.
I did have a few instances of “throttle no power” issue where the controller stayed on, and stayed armed but the throttle would be unresponsive. It happened to me in flight, and would typically happen only after I was on a long glide, over 5 min.
After pressing the throttle once, the controller would make the 6 beeps like it was rebooting, and then within 1-2 seconds the power would be back.
It only happened to me over a certain area ( Gila ) in New Mexico. I attributed the issue to RF interference or alien’s.
I did nothing to fix or address the issue, it simply stopped happening when I stopped flying over the area.
Some good news (fingers crossed): I have made some modifications to my firmware to improve RF interference tolerance somewhat, and also to dramatically reduce (even maybe eliminate) the random resets I was seeing. The changes are good, but not completely satisfactory because there is no smoking gun. But at this point I’ve tested it >10 hours on the ground, and no random resets! Note: I can still cause a reset if I get close with the VHF radio and try hard.
@jsneeb – very interesting. I’m learning some of the quirks of each part of the system. I have definitely seen exactly what you describe caused by RF (using my VHF radio). The system is happy, I hit it with RF, the ESC beeps for a few seconds (as if the throttle PWM signal was interrupted) and then gets happy. Sometimes I have to disarm/rearm.
Another quirk that might be relevant for you is that the BMS seems to sometimes “go to sleep”, and even stop responding to the power switch on the battery. It wakes back up if I unplug/replug the battery… or if I briefly spin the prop – at which point it seems to wake up, realize its supposed to be off, and shuts off! Of course you had the switch on the whole time… but I wonder if the “sleep” was doing something.
I have two of the v1 batteries without smart BMS, so no issues with BMS sleeping or waking and no switch.
I like the features of seeing your battery cell voltages and temps, but I think I prefer the v1 batteries they just work and have no quirks like you described.
I have the rp2040 throttle running v5.8R and see the same behavior. Holding the radio 6 inches from the throttle or any part of the throttle cable and transmitting on FRS1 causes the motor to stop with no way to restart it without unplugging the battery. I would assume the watchdog would kick in at some point but I have not seen it recover in the few minutes I waited.
Note – the watchdog is set to just a few seconds. One of the three failure modes (“hard off”) I’ve seen is that the TC screen to go blank and stay off permanently. I’m not sure if its a glitch in the power supply circuitry, or whether the rp2040 somehow locks up so hard that the watchdog can’t reset.
Paul and I have been testing a bunch of controllers and have found a couple of ways to improve the reliability of the controller with rewritten firmware. I’m planning on a proper write-up but in the meantime, I just released version 6.0 which uses FreeRTOS under the hood.
You can flash it with config.openppg.com but and its available there or from here Release [6.0] FreeRTOS, screen rewrite · openppg/eppg-controller · GitHub
We’ve been flying with it but let me know how it works for you.
Another quirk I noticed is the cruise control cutting off after 5-6 minutes. Is this intentional or a bug?
I charged the battery to 100% last weekend but did not get to fly, so I strapped it into the hitch carrier and set the cruise control to about 25% power just to drain the battery a bit. After 5-6 minutes, it would cut out. Did this 2 or 3 times.
When you say cut out what does that mean? Is that the hand controller resetting or the battery shutting off or it just the cruise turning off and the hand controller still being armed?
Here is a clip of it happening. Cruise control cuts off and the controller disarms. First time it cutout was about 1:20 in, I did it again and it ran for 45 minutes before doing the same thing.
@LarryT – that is exactly what I have seen in the field too (always on the ground), in addition to controlled bench testing. In your video, I can just see the TC screen go blank for a second or two as the motor stops spinning, and then TC reboots.
In the thread above, I call that one a “quick reset”: TC screen goes black, TC resets, no beeps from ESC.
And of course, when the TC resets, it comes back up disarmed.
Your time intervals sound like mine, too. I’ve had it take just 3 minutes, or sometimes as much as 70 minutes.
I will say that after I adjusted the firmware (main improvements are the clock frequency and better ESC telemetry parsing), and I have had no “random” resets, though I can still get them by trying hard with the VHF radio.
I haven’t tried Zach’s new firmware to see if it is also more robust.
