Can't recover from out of memory

lightningrod · Post by **lightningrod** » Thu May 30, 2019 3:23 pm

I have been steadily increasing my scripts complexity. I now have run out of memory, which I have been expecting.

However, I can no longer over write that script. It appears to write the new script successfully, but even a tiny script like setTickRate(1) returns an out of memory error. If I then read the configuration back, I see my old huge script is back.

This script has now survived a firmware upgrade and an attempt at a factory reset (because it clearly didn't work).

What now?
Here is log from attempted reset:
RaceCapture/Pro MK3 > resetConfig
Flashing Default Logger Config: PASS
Flashing Default Script: FAILED
Flashing Default Tracks: PASS
result="error";code=-4;cu: Got hangup signal

Here are the lua logs:

[lua] Initializing Lua state
[lua] memory usage: 17KB
[lua] Starting Lua Task
[wifi] Initializing...
[lua] Loading script. Length: 4986
[lua] Memory ceiling hit: 51391 > 51200
[lua] Startup script error: (not enough memory)
[lua] Failure: Failed to load script

Then on writing a new tiny script, I get this debug message:
lua: Failed to allocate memory for script buffer.

Post by **brentp** » Thu May 30, 2019 4:16 pm

Hi Rod,

Try to flash an older version of the firmware, which should trigger a factory reset - it's an alternative to the command line option.
https://wiki.autosportlabs.com/RaceCapt ... aceCapture

Is this happening with 2.16.0?

lightningrod · Post by **lightningrod** » Thu May 30, 2019 4:41 pm

The original problem happened with 2.15.1. Since then I upgraded to 2.16.0 and then back down to 2.15.1 again.

I also tried using the "resetConfig" command at the serial command line interface, which is where it looks like its trying to restore 3 partitions: Logger and Tracks are OK. But Script fails be be re-flashed.

Should I try a much older firmware? Is it worth trying 2.13?

Post by **brentp** » Thu May 30, 2019 4:48 pm

Thanks. So - does going between 2.15.x and 2.16.x not reset the lua script?

You can try going to an older firmware version as well. Starting with 2.15.x we reduced the amount of Lua memory available in order to enhance system stability. An older version of the firmware may let you get past that in order un-wedge things.

lightningrod · Post by **lightningrod** » Thu May 30, 2019 4:54 pm

Correct: changing from 2.15.1 to 2.16 did not clear the old script. Going to 2.16 again still did not clear it.
I'm just trying 2.13.0 and the script is still in the configuration when read from the rcp.

lightningrod · Post by **lightningrod** » Thu May 30, 2019 5:04 pm

I seem to have managed to clear the oversized script.
The downgrade to 2.13.0 didn't clear the script, but I was able to overwrite the script with this version.

Now upgraded to 2.16.0. The small script I loaded in 2.13 is still resident.

I'll go back to my debugging for now, but I hope that don't have to downgrade my firmware every time I run out of memory.

Thanks for the suggestions brent.

Post by **brentp** » Thu May 30, 2019 5:10 pm

Glad you got it figured out.

If you run into the problem again, can you try to do a factory reset using the command line using 2.16.0? Just to confirm.

Thanks,

lightningrod · Post by **lightningrod** » Fri May 31, 2019 2:31 am

So it happened again, although it doesn't happen very often.

I did try the factory reset under 2.16.0 but it didn't allow me to rewrite the stuck script.

However, this time during the firmware downgrade to 2.13 it lost connection around 40%. Now RCP3 does not boot.

It creates the serial connection. I can see that a /dev/ttyACM0 is created but it doesn't respond.

Solid green light power light, rapid flashing amber SDCard light.

Is there a recovery procedure from here?

Thanks.

Post by **brentp** » Fri May 31, 2019 6:23 pm

Hi Rod,

It looks like it failed the firmware update. When this happens, the system just starts up in bootloader mode so you can flash firmware again. Give it another try to see if it goes from 0-100%.

Note, you'll need to load a previously saved config under Setup to access the firmware update page.

Do you have an example script you can attach here that causes the problem to happen, so we can reproduce it?

Thanks,

lightningrod · Post by **lightningrod** » Sat Jun 01, 2019 12:58 am

Nice. I managed to find my way back into the firmware page. (It should not require a config file to access this). And it went smoothly back to 2.16

I should have thought to save the script for you. If it happens again. I will make a copy.

My recollection is that seemed to contain a mix of 2 different write/run attempts. One which had been run through minify and one that had not. I guess this could have been a user selection error where my Ctl-A didn't happen or it could have be copy paste error in the App (1.14.2 linux) or it could have been memory corruption on the rcp.

Happy to have my rcp back up. Disappointed it stayed home today while my car and I were at the track.

lightningrod · Post by **lightningrod** » Sat Jun 01, 2019 1:13 am

Here is the script that now refuses to be overwritten:

Code: Select all

_c="count"
lastRPM=0
rpmInc=false
function isWOT&#40;&#41;
if&#40;-getImu&#40;0&#41;&#41;>0.06 then
local ay=getImu&#40;1&#41;
if ay*ay<0.01 then return true end
end
local ret=false
if getGpsSpeed&#40;&#41;<5.0 then
local rpm=getChannel&#40;"RPM"&#41;
if rpm<lastRPM then
rpmInc=false
elseif rpm>lastRPM then
rpmInc=true
end
lastRPM=rpm
ret=rpmInc
if rpm<950 then ret=false end
end
return ret
end
acList=&#123;&#125;
acListN=0
acChansN=0
acChans=&#123;&#125;
function acAddRange&#40;name,min,max&#41;
local t=&#123;&#125;
t.chanDt=addChannel&#40;"dT_"..name,10,2,0,10,"ms/rpm"&#41;
t.name=name
t.chans=&#123;&#125;
t.minRPM=min
t.maxRPM=max
acResetTimer&#40;t&#41;
acList&#91;acListN&#93;=t
acListN=acListN+1
end
function acAddChannel&#40;name,logHz,prec,min,max,units,updateFn&#41;
acChans&#91;acChansN&#93;=&#123;&#125;
acChans&#91;acChansN&#93;.updateFn=updateFn
acChans&#91;acChansN&#93;.name=name
for i=0,acListN-1 do
local t=acList&#91;i&#93;
local tChan=&#123;&#125;
tChan.chan=addChannel&#40;name..t.name,logHz,prec,min,max,units&#41;
t.chans&#91;acChansN&#93;=tChan
end
acChansN=acChansN+1
end
function acResetTimer&#40;t&#41;
t.startT=0
t.startRPM=0
for i=0,acChansN-1 do
t.chans&#91;i&#93;.afSum=0
t.chans&#91;i&#93;.afCount=0
end
end
function acUpdateTimer&#40;t,rpm&#41;
if rpm>=t.minRPM then
if rpm<t.maxRPM then
if t.startRPM==0 then
println&#40;"start&#58;"..t.name&#41;
t.startRPM=rpm
t.startT=getUptime&#40;&#41;
else
println&#40;"update"&#41;
local dR=rpm-t.startRPM
if dR>0 then
local dT=getUptime&#40;&#41;-t.startT
local acc=dT/dR
setChannel&#40;t.chanDt,acc&#41;
end
end
for i=0,acChansN-1 do
acSampleAFR&#40;t,acChans&#91;i&#93;.updateFn&#40;&#41;&#41;
end
elseif t.startRPM~=0 then
println&#40;"finished&#58;"..t.name&#41;
acResetTimer&#40;t&#41;
end
end
end
function acSampleChan&#40;t,v&#41;
if v~=nil then
t.afCount=t.afCount+1
t.afSum=t.afSum+v
local av=t.afSum/t.afCount
println&#40;"afrac&#58;"..t.name..","..t.afSum..","..t.afCount..","..av&#41;
setChannel&#40;t.chanAF,av&#41;
end
end
function acOnTick&#40;&#41;
local wot=isWOT&#40;&#41;
local rpm=getChannel&#40;"RPM"&#41;
for i=0,acListN-1 do
local t=acList&#91;i&#93;
if wot then
acUpdateTimer&#40;t,rpm&#41;
else
if t.startRPM~=0 then acResetTimer&#40;t&#41;end;
end
end
end
afChan=-1
ispAFR=nil
ispHiBuf=nil
ispAFx=nil
function ispHlrHh&#40;h&#41;
if bit.band&#40;h,&#40;0xA2&#41;&#41;==&#40;0xA2&#41;then
ispHiBuf=h
ispNxtBHdlr=ispHlrHl
end
end
function ispHlrHl&#40;l&#41;
if bit.band&#40;l,&#40;0x80&#41;&#41;==&#40;0x80&#41;then
ispNxtBHdlr=ispHlrAFh
else
ispNxtBHdlr=ispHlrHh
end
end
function ispHlrAFh&#40;h&#41;
ispHiBuf=h
ispNxtBHdlr=ispHlrAFl
end
function ispHlrAFl&#40;l&#41;
ispNxtBHdlr=ispHlrLh
ispFunc=bit.band&#40;bit.rshift&#40;ispHiBuf,2&#41;,7&#41;
ispAFx=bit.band&#40;ispHiBuf,1&#41;*0x80+
bit.band&#40;l,0x7F&#41;
end
function ispHlrLh&#40;h&#41;
ispHiBuf=h
ispNxtBHdlr=ispHlrLl
end
function ispHlrLl&#40;l&#41;
local lambda=bit.band&#40;ispHiBuf,0x7F&#41;*0x80+bit.band&#40;l,0x7F&#41;
if ispFunc==0 then
ispAFR=&#40;lambda+500.0&#41;*ispAFx*0.0001
elseif ispFunc==1 then
ispAFR=lambda*0.1
else
println&#40;"afr&#58;func"..ispFunc.."mult&#58;"..ispAFx.."lambda&#58;"..lambda&#41;
end
ispNxtBHdlr=ispHlrHh
end
function afOnTick&#40;&#41;
local c=readCSer&#40;6,0&#41;
while c~=nil do
ispNxtBHdlr&#40;c&#41;
c=readCSer&#40;6,0&#41;
end
if ispAFR~=nil then
setChannel&#40;afChan,ispAFR&#41;
ispAFR=nil
end
end
function onTick&#40;&#41;
afOnTick&#40;&#41;
acOnTick&#40;&#41;
end
function getAF&#40;&#41;
return getChannel&#40;afChan&#41;
end
println&#40;"mem&#58;"..collectgarbage&#40;_c&#41;&#41;
acAddRange&#40;"3",2500,3500&#41;
println&#40;"mem&#58;"..collectgarbage&#40;_c&#41;&#41;
acAddRange&#40;"4",3500,4500&#41;
println&#40;"mem&#58;"..collectgarbage&#40;_c&#41;&#41;
acAddRange&#40;"5",4500,5500&#41;
println&#40;"mem&#58;"..collectgarbage&#40;_c&#41;&#41;
acAddRange&#40;"6",5500,6500&#41;
println&#40;"mem&#58;"..collectgarbage&#40;_c&#41;&#41;
acAddChannel&#40;"AF",10,1,7,25,"%O2",getAF&#41;
if initSer&#40;6,19200,8,0,1&#41;then println&#40;'serial initialized'&#41;else println&#40;'initSer failed&#58;port'..6&#41;end afChan=addChannel&#40;"AFR",10,1,8,23&#41;ispNxtBHdlr=ispHlrHh
setTickRate&#40;60&#41;

lightningrod · Post by **lightningrod** » Sat Jun 01, 2019 1:17 am

Here is the output from my attempt to do a factory reset (resetConfig). Note the "Flashing Default Script: FAILED" line.

Code: Select all

cu -l /dev/ttyACM1 
Connected.


=============================================
Welcome to RaceCapture/Pro MK3 version 2.16.0
=============================================

Available Commands&#58;

resetConfig   &#58; Resets All configuration Data to factory default Usage&#58; resetConfig 
testSD        &#58; Test Write to SD card. Usage&#58; testSD <lineWrites> <periodicFlush> <quietMode>
startTerminal &#58; Starts a debugging terminal session on the specified port. Usage&#58; startTerminal <port> <baud> &#91;echo 1|0&#93;
setLogLevel   &#58; Sets the log level Usage&#58; setLogLevel <level>
viewLog       &#58; Prints out logging messages to the terminal as they happen Usage&#58; viewLog 
setSerialLog  &#58; Enables/disables logging of  serial device for debug purposes Usage&#58; setSerialLog <port> <0|1>
flashConfig   &#58; Flashes the NVRAM with the current configuration of the LoggerConfig Usage&#58; flashConfig 
showTasks     &#58; Show status of running tasks Usage&#58; showTasks 
version       &#58; Gets the version numbers Usage&#58; version 
showStats     &#58; Info on system statistics. Usage&#58; showStats 
sysReset      &#58; Reset the system Usage&#58; sysReset &#91;bootloader 0|1&#93;
lua           &#58; Enter the lua script interpreter. Usage&#58; lua 
reloadScript  &#58; Reloads the current script. Usage&#58; reloadScript 
RaceCapture/Pro MK3 > resetConfig
Flashing Default Logger Config&#58; PASS
Flashing Default Script&#58; FAILED
Flashing Default Tracks&#58; PASS
result="error";code=-4;cu&#58; Got hangup signal

Disconnected.

lightningrod · Post by **lightningrod** » Sat Jun 01, 2019 1:23 am

I now have a couple of weeks before I'm on track again. So before I go up/down grading random firmware versions, I will leave it in this state if you like, so that I can help debug this (or you can help me). I'd rather get a longterm fix than to just keep "upgrading" firmware.

Post by **brentp** » Tue Jun 04, 2019 1:17 am

Noted, we will try to get it into a similar state and see what we can do.

Thanks,

psfp · Post by **psfp** » Tue Jun 04, 2019 10:43 am

I've been experiencing the same problem. Firmware 2.13.5 works for clearing the script data. Factory reset won't solve this issue when using 2.16.