RCP continually crashing :-(

Report any thing that looks like a bug or problem in the system here

Moderators: JeffC, stieg

Post Reply
paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

RCP continually crashing :-(

Post by paulbutcher »

I have an RCP that I ran all last year with no problems. It also ran fine in the first race of this year, but suddenly it's started repeatedly crashing :-(

I would appreciate advice on how to diagnose the cause of the crashes. Here's a detailed description of what I'm seeing and what I've tried so far. I apologise that this is quite a long message, but I wanted to make sure that I included everything rather than miss out an important detail.

The problem first started during testing on Friday. At that time, I was running very out of date firmware on the RCP and software on the phone that I use as a dash. So on Saturday I upgraded both to the latest versions (2.9.0 and 1.4.2 respectively). Although the new software is a massive improvement (good work guys) it didn't solve the problem :-(

I have the RCP setup with the ShiftX lights that I use as *either* shift lights *or* error lights depending on circumstances - see this Lua script that I posted a while back for details:

https://www.autosportlabs.org/viewtopic.php?t=4416

When the problem started, what would happen is that I would be out on circuit, and suddenly I'd get an error showing up on the shift lights (i.e. just the red lights in the middle lit up). So that would suggest that some parameter has gone out of range (e.g. oil pressure has dropped to zero). At the same time, the Bluetooth connection between the RCP and the phone would fail (no updates on the phone). The only way to clear this was to power-cycle the RCP by switching off and back on at the master switch, at which point everything would work fine for another few minutes, but then I'd see the same again.

In the races on Sunday, I decided to stay out, even though I had a big red error light on the dash (scary, given that that supposedly means that the engine is about to lunch itself). What I then saw was that after a while the RCP would reboot (I can tell that it was rebooting because when it does it briefly lights up all the shift lights). Things would then work OK for another minute or so, after which it would again show the error lights and everything would stop again.

I also occasionally saw another failure mode, which was that I'd have the dash telling me that I was in the middle of a pitstop, even though I was tearing down the straight at well over 100mph.

Nothing shows up in the log which looks at all helpful. The log is recorded fine, until it suddenly stops. There's nothing untoward in the log that I can see (no battery voltage drop-off just beforehand, for example) it just suddenly stops. It *doesn't* start logging again after the RCP has rebooted though, which is odd (I have the Lua script setup to start logging as soon as GPS speed goes above 10mph).

We've disconnected all the sensors from the RCP apart from RPM, oil temp, and oil pressure, but still see the same behaviour (so the only things that are connected are those three plus power, 5v reference and the three outputs to the ShiftX lights). This made no difference.

Suggestions very welcome indeed.

Thanks!
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

On the plus side, even though I was racing without a dash or shift lights for most of the time, and deliberately ignoring the big red "your engine is about to destroy itself" warning light, I came P2 in race 1 and P3 in race 2 ;-)
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

stieg
Posts: 100
Joined: Fri Dec 20, 2013 1:37 am
Location: Madison, WI
Contact:

Post by stieg »

Hey Paul,

Thanks for reporting. So the first step would be to upload your config file. This way we can see if we can reproduce the issue locally.
I would appreciate advice on how to diagnose the cause of the crashes.
The best place to start is to plug the unit into a computer, turn it on and look at the system logs. They are on the same screen where you enter your Lua config in the App. You will see a button that says to enable logging. Do that and see if you can get the issue to happen again. Sometimes you get good log messages before a crash happens. You can also set it to the DEBUG level to get even more messages.
When the problem started, what would happen is that I would be out on circuit, and suddenly I'd get an error showing up on the shift lights (i.e. just the red lights in the middle lit up). So that would suggest that some parameter has gone out of range (e.g. oil pressure has dropped to zero). At the same time, the Bluetooth connection between the RCP and the phone would fail (no updates on the phone). The only way to clear this was to power-cycle the RCP by switching off and back on at the master switch, at which point everything would work fine for another few minutes, but then I'd see the same again.
Hrmm... are you by chance using lua script to start/stop your logging? If so mind commenting out those actions and controlling logging manually using the button on the unit. Last week I addressed an issue with similar behavior (https://github.com/autosportlabs/RaceCa ... issues/571) that will be available in the 2.9.1 release.

Does this issue happen while you are idling in the pits by chance? Or only while on track. Other tricks you can do to diagnose issues like this are to pull out pieces of your config and see what helps and what doesn't. In example try removing Lua script and see if it works. If so, then issue is related to something Lua is doing. Obviously this not ideal though... so lets see if we can get you fixed up before next weekend. Don't forget to upload your config.
Andrew Stiegmann (Stieg)
Principal Engineer
Autosport Labs Inc.

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

Thanks for the reply, Andrew - much appreciated.

I've attached my configuration to this post.

I am (as you will be able to see from the uploaded configuration) starting logging via Lua. I doubt that this is the problem as the problem first appeared on Friday when I was running exactly the same (very out of date) firmware and configuration as I had been all last year.

It's difficult for me to start logging by pressing the button because we currently have it mounted somewhere that I can't get to without removing bodywork. But I can disable that bit of the Lua script and see if that changes the behaviour.

I will enable system logs as you suggest and see what we see. Unfortunately we've never seen the problem while the car's idling in the pitlane or garage, which suggests that it's related to heat or vibration? Or actually, I guess that that might point to the call to startLogging as that only happens when the car is moving? But that raises the question of why everything worked perfectly until Friday?

It's clear that we're going to have to book a test day to diagnose this (as it only happens while the car is running). Not sure when we'll be able to do that exactly. The next race weekend is the weekend of 11th/12th June so we'll make sure that it's well before then!
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

stieg
Posts: 100
Joined: Fri Dec 20, 2013 1:37 am
Location: Madison, WI
Contact:

Post by stieg »

paulbutcher wrote: I've attached my configuration to this post.
I don't see it.
I am (as you will be able to see from the uploaded configuration) starting logging via Lua. I doubt that this is the problem as the problem first appeared on Friday when I was running exactly the same (very out of date) firmware and configuration as I had been all last year.

It's difficult for me to start logging by pressing the button because we currently have it mounted somewhere that I can't get to without removing bodywork. But I can disable that bit of the Lua script and see if that changes the behaviour.
Sounds good. The issue I mentioned was partially caused by a bad SD card as well. It may be worth it to re-format your SD card and try again.
I will enable system logs as you suggest and see what we see. Unfortunately we've never seen the problem while the car's idling in the pitlane or garage, which suggests that it's related to heat or vibration? Or actually, I guess that that might point to the call to startLogging as that only happens when the car is moving? But that raises the question of why everything worked perfectly until Friday?
Doubtful about heat or vibration. Likely a code issue based on how you described it. Lots of things can change between releases, and what may have previously not been a bug sometimes can become one due to new behaviors. Happens all the time in this industry.
It's clear that we're going to have to book a test day to diagnose this (as it only happens while the car is running). Not sure when we'll be able to do that exactly. The next race weekend is the weekend of 11th/12th June so we'll make sure that it's well before then!
Not necessarily. We do simulations internally by reading an IMU axis, setting a value like speed in Lua, and using that to trigger some of the changes that happen when the vehicle is on track. You could do something like this if the RCP is accessible (which it sounds like it isn't :( ). Why is your RCP hidden behind body panels?
Andrew Stiegmann (Stieg)
Principal Engineer
Autosport Labs Inc.

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

stieg wrote:
paulbutcher wrote: I've attached my configuration to this post.
I don't see it.
Damn! Sorry about that - it looks like the forum software throws away an attachment when you click the Preview button :-(

I'm traveling at the moment and won't have access to the file until Friday - I'll upload it then.
The issue I mentioned was partially caused by a bad SD card as well. It may be worth it to re-format your SD card and try again.
Will do. Thanks.

One additional bit of information that I should have given you is that I'd forgotten to insert the SD Card during the first session of testing when the problem first started happening. So it happens both with and without an SD Card present.
Doubtful about heat or vibration. Likely a code issue based on how you described it. Lots of things can change between releases, and what may have previously not been a bug sometimes can become one due to new behaviors. Happens all the time in this industry.
Yup, I write software for a living and am very aware of how this can happen (I even wrote a book on debugging software ;-). However note that the problem first appeared on Friday when I was running exactly the same firmware, Android software and configuration as I've been running since the middle of last year. I only upgraded to the most recent versions after the problem occurred on Friday.
Not necessarily. We do simulations internally by reading an IMU axis, setting a value like speed in Lua, and using that to trigger some of the changes that happen when the vehicle is on track. You could do something like this if the RCP is accessible (which it sounds like it isn't :( ). Why is your RCP hidden behind body panels?
Because in my single-seater, the available cockpit and dash space is *very* limited :-) It's easily accessible when the car's stationary (the body panels are attached with Dzus fasteners and removable within seconds) so we can run experiments when the car's in the garage very easily. I just can't get to it from the driver's seat while I'm in the car.

Anyway, I'll upload the configuration file on Friday.

Thanks for all your help.
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

paulbutcher wrote: Anyway, I'll upload the configuration file on Friday.
Attached!
Attachments
PaulB.rcp.zip
(1.95 KiB) Downloaded 503 times
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

stieg
Posts: 100
Joined: Fri Dec 20, 2013 1:37 am
Location: Madison, WI
Contact:

Post by stieg »

First off... that is a BEAUTIFUL looking Lua script. As a fellow programmer... thank you for doing it right and well. Now then... on to business...

Looking at your script I see nothing obvious wrong. Its dirt simple and I don't see how that is causing an issue. I'm still wary of the SD card logging issue however and do suspect it. One thing to try would be to simply comment out the logging of the data to the SD card. If that fixes it then we have isolated a root cause. I have a patch in 2.9.1 that addresses this very issue (along with a couple of other minor issues), but its untested. If you were so inclined and wanted to live on the bleeding edge, you could build a release of 2.9.1 from the repo and run with it. I have been using it locally for development... and its quite stable AFAICT. But of course YMMV. So to recap:

* Re-format the SD card and try that
* If you see the problem, just kill the logging altogether and see if that is what is causing the issue.
** If that fixes it, then I will tag and build a pre-release of 2.9.1 for you to test if you are so inclined.
** If that doesn't fix it, then I will load up your script locally and see if I can reproduce the same issue.

Does the issue happen at any particular speed, or just when your moving? Cheers.
Andrew Stiegmann (Stieg)
Principal Engineer
Autosport Labs Inc.

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

stieg wrote:First off... that is a BEAUTIFUL looking Lua script. As a fellow programmer... thank you for doing it right and well.
Thanks - one of the downsides of providing an embedded programming language for this kind of thing is that you're going to get a lot of scripts written by non-programmers. Pleased to be able to redress the balance a little, at least 😉
* Re-format the SD card and try that
* If you see the problem, just kill the logging altogether and see if that is what is causing the issue.
** If that fixes it, then I will tag and build a pre-release of 2.9.1 for you to test if you are so inclined.
** If that doesn't fix it, then I will load up your script locally and see if I can reproduce the same issue.
OK - will do. Thanks.
Does the issue happen at any particular speed, or just when your moving? Cheers.
It only occurred when I was out on circuit at racing speed, but I didn't notice any correlation between any particular speed and the problem occurring. Note that it's not deterministic: I think the max I managed was 6 laps before the problem occurred.

Given that I'm going to have to book a test day to try to reproduce it (one of the downsides of a single-seater is that I can't test on the public highway), is there any way to try to reproduce the problem without actually taking the car out on circuit?

Thanks,
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

brentp
Site Admin
Posts: 6282
Joined: Wed Jan 24, 2007 6:36 am

Post by brentp »

just a follow up, the v2.9.1 firmware has been released, if you haven't seen it yet. Give it a test and let us know how it works w/r/t SD card stability!
Brent Picasso
CEO and Founder, Autosport Labs
Facebook | Twitter

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

brentp wrote:just a follow up, the v2.9.1 firmware has been released, if you haven't seen it yet. Give it a test and let us know how it works w/r/t SD card stability!
Will do, Brent - FYI we're going to be testing the car on Friday 10th.

Thanks!
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

brentp
Site Admin
Posts: 6282
Joined: Wed Jan 24, 2007 6:36 am

Post by brentp »

Hi Paul,

Still seeing crashing problem with the latest firmware? (2.9.2)
Brent Picasso
CEO and Founder, Autosport Labs
Facebook | Twitter

paulbutcher
Posts: 42
Joined: Sat Dec 20, 2014 1:48 pm
Location: UK
Contact:

Post by paulbutcher »

brentp wrote:Hi Paul,

Still seeing crashing problem with the latest firmware? (2.9.2)
Sorry Brent - I should have reported back ages ago.

No, I'm not seeing the crashing problem any longer, but I can't be certain that the new firmware was the fix. We suspected an installation problem, so at the same time as updating, we also completely rewired the RCP's too.

I'm aware that this was bad practice from a debugging standpoint, but we needed the system to work and didn't have an opportunity to test before the next race weekend, so didn't feel that we could take the chance.

But anyway, the problem is no longer an issue.

Thanks for your help with this!
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con

brentp
Site Admin
Posts: 6282
Joined: Wed Jan 24, 2007 6:36 am

Post by brentp »

Great to know. Fyi, we're releasing new app and firmware shortly, watch for it!

Thanks again,
Brent Picasso
CEO and Founder, Autosport Labs
Facebook | Twitter

Post Reply