Serial telemetry streaming via LUA or BT spoof. Fast enough?

[zechdz]

I posted in telemetry about wifi access on the RCP, and that I had purchased a chip. However that chip requires a lot of embedded (c-like) setup and I don't have the time to go through all that . So... instead I found a really convenient chip, ESP8266, which runs NodeMCU allowing for control through Lua scripts for WiFi connections.

I wanted to make a serial interface between the RCP and this chip using LUA on both sides, allowing for some sharing of code, and was wondering if streaming through the LUA UART interface would allow for high quality streaming as is done with the telemetry/bt modules.

My second thought was to make this wifi module spoof the current BT module you guys have, and just directly hardcode things like host/port into the NodeMCU code, to act like the RCP app would. If doing this is better than using Lua, do you guys have any documentation on the BT interface so I could spoof that serial port connection?

I took a quick look through the firmware code, but haven't been able to piece it all together. Keep getting lost between whats a settings communication and whats a streaming communication in regards to BT. And if I had more time I would probably try to add my own firmware module for NodeMCU streaming and what not, but right now I'm trying to find the quickest/easiest way to get wifi so I can concentrate on riding.

So in summary:

1) is the Lua library (module? plugin?) quick enough/good enough to stream serial to my wifi module (including access point information before streaming). (note: also figured this path would allow me to use the auxiliary streaming port (or telemetry connection in the box) for my wifi module and the BT module is in the firmware, so could have an android phone and direct wifi setup at the same time)

2) if not, is there docs on how to spoof the BT module, so that RCP firmware thinks its communicating via bluetooth but instead is using my module.



And when I get this stuff working I will open source my code so others can see/use/help work out bugs. The ESP8266 is very cheap and easy to use, so i'm sure other people might be interested.

[stieg]

The second approach sounds more realistic from my perspective. The BT device is treated as a simple NULL serial modem, so if your device is already setup to work the way you want and can handle a raw stream, then all you need to do is simply ignore the setup commands and listen for the JSON stream.

Check out https://github.com/autosportlabs/RaceCa ... luetooth.c to see how BT is setup (this is unreleased code... it is part of the next release and has undergone some cleanup).

Check out connectivityTask in https://github.com/autosportlabs/RaceCa ... vityTask.c to see how data is sent (hint, its just a RAW JSON stream after init).

Hope that helps.

[zechdz]

Thank you for the reply stieg. So I am on step 1 of this project which is to stream fake data from the wifi module. I managed to get that all setup, do auth, and then send meta and a stream of data.

However the data showing up on the race-capture website is not right. Is there an order I should send the data in? I'm sending the data in the same order as the meta was sent, but it doesn't work. I also tried alphabetical and that didn't work either...


Meta:

Code: Select all

do send_msg: {"s":{"meta":[{"ut":"G","nm":"AccelX","sr":25},{"ut":"G","nm":"AccelY","sr":25},{"ut":"G","nm":"AccelZ","sr":25},{"ut":"Volts","nm":"Battery","sr":1},{"ut":"Miles","nm":"Distance","sr":50},{"ut":"Count","nm":"LapCount","sr":1},{"ut":"Min","nm":"LapTime","sr":1},{"ut":"Degrees","nm":"Latitude","sr":50},{"ut":"Deg\/Sec","nm":"Longitude","sr":50},{"ut":"Count","nm":"Sector","sr":1},{"ut":"Min","nm":"SectorTime","sr":1},{"ut":"MPH","nm":"Speed","sr":50},{"ut":"","nm":"Time","sr":50},{"ut":"Deg\/Sec","nm":"Yaw","sr":25}]}}

Data:

Code: Select all

do send_msg: {"s":{"d":[2.5,2.5,-2.5,5.1,0,0,0,39.745716841307,139.07335743824,0,0,11.929658833858,0,50,30]}}

do send_msg: {"s":{"d":[2.5,2.5,-2.5,5.1,20,0,1,36.690151892458,144.0525280759,0,1,18.843932793869,1,50,30]}}
do send_msg:

[zechdz]

Here is some rough lua code following the logic of the RCP code found at:
https://github.com/autosportlabs/RaceCa ... nection.py

Code: Select all

function sendMeta()
    local msg = {s={meta={}}}
    local meta = {}

    local count = 0
    for sortedk, sortedv in pairs(channelmeta) do
        for k, v in pairs(sortedv) do
            print("Channel name: "..k)
            table.insert(meta, {
                nm= v.name,
                ut= v.units,
                sr= v.sampleRate,
    --            min= v.min,
    --            max= v.max
            })
            count=count+1
         end
     end

    msg.s.meta = meta
    sendMessage(cjson.encode(msg))
end

Code: Select all

function sendSample(count)
     local msg = {s={d={}}}
     local bitmasks = {}
     local channelData = getFakeChannelData(count)
     local bitmasks_needed = math.max(0, math.floor((tablelength(channelData) - 1) / 32) + 1)
     local channel_bit_position = 0
     local bitmask_index = 1
     local data = {}

    for x = 1, bitmasks_needed do
        table.insert(bitmasks, 0)
    end

    for sortedk, sortedv in pairs(channelmeta) do
        for k, v in pairs(sortedv) do
            if (channel_bit_position > 31) then
                bitmask_index = bitmask_index + 1
                channel_bit_position = 0
            end
    
            if (channelData[k] ~= nil) then
                bitmasks[bitmask_index] = bit.bor(bitmasks[bitmask_index],(bit.lshift(channel_bit_position,1)))
                table.insert(data, channelData[k])
            end
    
            channel_bit_position = channel_bit_position + 1
         end
     end

    for index, bitmask in pairs(bitmasks) do
        table.insert(data, bitmask)
    end

    msg.s.d = data
    sendMessage(cjson.encode(msg))
end

[/code]

[stieg]

You are very close. I see two missing bits.

First, did you send the auth message? You need to auth a device against RCL before you can send data, else it will drop you. Its the first thing we do. Unfortunately this is something that the telemetry channel does. You may need to fake it with lua. Look at https://github.com/autosportlabs/RaceCa ... s/sim900.c for writeAuthJson for an example.

Second, looking at the raw JSON you are missing the "tick" field. This is a counter that increments with every sample sent. It lives in the `s` (for sample) object next to meta. Example is below:

Your JSON code:

Code: Select all

{"s":{"meta":[{"ut":"G","nm":"AccelX","sr":25},

Our JSON code:

Code: Select all

{"s":{"t":0,"meta":[{"nm":...

See that `t` object. That is what you are missing.

For examples I would recommend looking at the JSON samples we use for testing. These are kept up to date as we use them for our unite tests:

https://github.com/autosportlabs/RaceCa ... onse1.json
https://github.com/autosportlabs/RaceCa ... onse2.json

[zechdz]

Yeah I did the auth part, just didnt include it in my post here. Then after auth i was ablento see live data on race-capture it just kept matching data to different meta. So my battery value would get distance values for example, even though I sent the meta and data in the same order.

I used some of the json samples, but I must've overlooked the tick key!

I'll give that a try.

[zechdz]

Very dumb miss

Code: Select all

bit.lshift(1,channel_bit_position)
                
vs

bit.lshift(channel_bit_position,1)
                

[stieg]

Oops.... silly parameter orders...

[zechdz]

Do you have any documents on the meaning of these values?

Code: Select all

{nm="Interval",ut="ms",min=0,max=0,prec=0,sr=1},
{nm="Utc",ut="ms",min=0,max=0,prec=0,sr=1},
{nm="LapCount",ut="",min=0,max=0,prec=0,sr=10},
{nm="LapTime",ut="Min",min=0.0,max=0.0,prec=4,sr=10},
{nm="Sector",ut="",min=0,max=0,prec=0,sr=10},
{nm="SectorTime",ut="Min",min=0.0,max=0.0,prec=4,sr=10},
{nm="CurrentLap",ut="",min=0,max=0,prec=0,sr=10}

1) What is Interval and UTC (I know what UTC is i just dont know what this data should mean, why would the RCP have access to a true UTC time... from gps?)
2) I'm not sure of the difference between CurrentLap and LapCount (is LapCount total lap count?)
3) I am also not sure how to use LapTime vs LapCount. (same as Sector vs SectorTime)

I tried this:

Code: Select all

LapCount=1,
LapTime=count/60,

but the lap time and lap count increased together with the value set in LapTime!



I also logged data of the weird lap count thing, but all the data disappeared in a day
https://www.race-capture.com/events/415 ... ing-logger

[stieg]

{nm="Interval",ut="ms",min=0,max=0,prec=0,sr=1},

nm - name (12 char max)
ut - unit (7 char max)
min - Minimum value
max - maximum value
prec - Precision (used for floating point)
sr - Sample Rate (samples/second)

1) What is Interval and UTC (I know what UTC is i just dont know what this data should mean, why would the RCP have access to a true UTC time... from gps?)

internval is uptime in milliseconds
utc is utc time in milliseconds. Its time is retrieved from GPS. GPS is effectively an atomic clock.

2) I'm not sure of the difference between CurrentLap and LapCount (is LapCount total lap count?)

CurrentLap is what lap you are on. LapCount is how many you have completed.

3) I am also not sure how to use LapTime vs LapCount. (same as Sector vs SectorTime)

LapTime is how long the last lap was. LapCount is how many laps you have completed.

I also logged data of the weird lap count thing, but all the data disappeared in a day Sad
https://www.race-capture.com/events/415 ... ing-logger

That is... strange. Did you complete a single lap?

[zechdz]

Thanks again for the quick response.

Nice to know about the max limits for name and units, whenever you guys get a chance document this stuff in github or api docs . I mostly say this because I noticed the announcement of podium, and I'm sure transitioning my code to podium is going to have similar hurdles, if the api changes.

Okay so the values I streamed was this below:
linearly increasing Time, Distance, LapTime, SectorTime

Code: Select all

              AccelX=2.50,
              AccelY=2.50,
              AccelZ=-2.50,
              Battery=5.1,
              Distance=20*count,
              LapCount=1,
              LapTime=count/60,
              Latitude=35 + (5*math.random()),
              Longitude=136 + (10*math.random()),
              Sector=0,
              SectorTime=count/60,
              Speed=20*math.random(),
              Time=count/60,
              Yaw=50,

From what you said about LapTime and CurrentLap I see what my problem is. I thought LapTime was the current lap time, but rather it is the last lap time. So I sent a new LapTime that increased at the same rate as Time (I wanted to simulate a long single lap for simplicity). But I never increased the LapCount. So it seems that live data thought i was increasing laps as I kept sending new LapTimes, but LapCount never increased so when you refresh the browser it sees that there was only ever LapCount=1 and only loads the first lap. This might be a bug on the server side, but its definitely a misuse of the API on my part.

I didn't know about CurrentLap at first because I based my data on what seems to be only a subset of the values I can use found at the link below. And since there were no API docs I just assumed LapTime had to be what CurrentLap is
https://github.com/autosportlabs/RaceCa ... pledata.py

[stieg]

Nice to know about the max limits for name and units, whenever you guys get a chance document this stuff in github or api docs Very Happy

Definitely. We still intend to cleanup the API at some point but yeah this has been a TODO for some time.

From what you said about LapTime and CurrentLap I see what my problem is. I thought LapTime was the current lap time, but rather it is the last lap time. So I sent a new LapTime that increased at the same rate as Time (I wanted to simulate a long single lap for simplicity). But I never increased the LapCount. So it seems that live data thought i was increasing laps as I kept sending new LapTimes, but LapCount never increased so when you refresh the browser it sees that there was only ever LapCount=1 and only loads the first lap. This might be a bug on the server side, but its definitely a misuse of the API on my part.

Yeah... the nomenclature is definitely confusing and could use some love.

RE the LapCount, RCL triggers a lap save when the LapCount changes. So to get laps to store, you need to complete the lap (ie increment LapCount). That will probably get you what you want .

Also... you are doing quite well considering 0 docs. Good work.

[zechdz]

I got it to mock at least Lat/Lon and some crappy lap/time info going around Motegi using some mock data files I created from a session I did with my old logger (Aim solo)

Code so far: https://github.com/zkdzegede/nodemcu-rcp

My bike has been in the shop, hence working on this project with only mock data. But I get the bike (with RCP) back this week, so then I'll be changing my mock UART interface to a real one and trying it with real Lat/Lon values.

The rest of the values will also hopefully be trivial passing of information uart->json. If not I'll have to re-do the firmware to free up some more RAM haha. The nodemcu is so constrained

[stieg]

Killer. Let us know how it goes. Also feedback on motorcycle data would be great. Would love to know what low hanging fruit can be addressed to make it more useful for ya.

[zechdz]

I still have a long road ahead to connect my analog sensors on the bike but i'll try to document it and let you guys know of anything that might help your product .

I have however run into another hurdle. I commented here: https://github.com/autosportlabs/RaceCa ... issues/165

So what i was able to do:

1) Connect ESO8266 TX -> RX (yes to itself), and *could* send/receive the same message.
2) Connect RCP TX -> RX (yes to itself), and *could not* receive its own data. So this might be my problem here...

The following two were done using RCP lua script and init(4, 9600, 8, 0, 1) and using the same settings on my ESP8266

3) Connect RCP TX -> ESP RX, and *could not* get any data at all
4) Connect RCP RX -> ESP TX, and *could* get data *but* it was always wrong. e.g. (AT became XZ). And many other things became QPGDM or something (maybe an error message... something about GodDaMn I presume)

Anyhoo, I'm stuck and cannot transmit any UART data between the two.