Do you want to ...
- use your RaceCapture device to find all the OBDII PIDs supported on a CAN bus?
- use your RaceCapture device to capture all or just certain-ID CAN messages on a CAN bus?
- know more about the inter-message timing on a CAN bus?
- sort captured CAN messages into groupings by CAN ID and/or PID?
- apply a collection of formulas to captured CAN messages and generate a csv file containing the computed values along with a summary showing the record number total, the min/max values and the max-min value for each field?
- select a collection of fields from a csv file and generate another csv file containing only those fields ... along with the record number total, the min/max values and max-min value for each field?
... then see the following free tools.
https://www.derman.com/Resources/Misc/C ... sTools.zip
(<5 MB)
Background
I am in the process of reverse engineering some info about my car and have produced some tools to help that process (see Analysis Process, below). I'm making this stuff available with the hope that some of it will help others.
For reference, here's the equipment I'm working with:
- RaceCapture/Pro MK3 running firmware 2.17.8
* CAN1 bus via standard OBDII CAN cable (CAN2 bus not on car's OBD)
* CAN2 bus via a custom CAN cable connection (no CAN2 bus data in examples)
- Macs running macOS 10.12.6
Free Tools
The link'd zip file includes the following:
- a slimLuaScripts.sh shell script that removes comments and most of the whitespace from a lua program
(the on-line minifier at https://mothereff.in/lua-minifier may reduce by an additional 7-10%)
- 2 lua scripts that run on RaceCapture units to log CAN bus messages:
* CAN PID Prober.lua (and the slim'd CAN PID Prober-.lua)
* CAN Msg Logger.lua (and the slim'd CAN Msg Logger-.lua)
* The "-.lua" versions of the scripts have been processed by the slimLuaScripts.sh utility
* note: the full/commented scripts are too large to run without all the comments and most of the whitespace being removed.
* The full versions of the scripts include extensive comments that serve as the documentation when running the slim'd versions
* The logger scripts provide a number of features:
+ CAN ID/PID filtering
+ ON/OFF logging control via front-panel button on applicable RC models
+ syncing SD card capture with logging activity on applicable RC models
+ bus/baud/termination selection (some only apply to some RC models)
+ tickRate control and various logging controls
+ features are enabled/disabled by setting variable values in the script
- a captureRClog.sh script that captures the log from a USB-connected RaceCapture unit into a file
* the script is capabile of doing high-rate/long-duration captures
* the terminal emulator I tried was not able to "keep up" during large captures
- a processCANmsgLogs.sh shell script to post-process the logs captured via the lua logger scripts
* creates 1 or 2 result files in either sorted/grouped or unsorted format:
+ one with all the CAN messages
+ if applicable, one with all the responses to PID queries
* can optionally sort messages into groups by their CAN ID or PID
* to each message, computes and adds
+ the time duration between messages
+ the time duration from first message logged
* adds a top-of-file field-header
- 2 shell scripts that apply a collection of user supplied formulas to compute values for CAN messages:
* calcCANvalues.sh
* calcPIDvalues.sh
Demo
For a quick overview, try these "demo" steps (10-15 minutes) ...
- in the Demo folder, open the CTlog.txt raw CAN message capture file and read the explanation I added to the top of the file (then close the file)
- run the command (adjusting the pathnames as required):
.../Scripts/processCANmsgLogs.sh .../Demo/CTlog.txt
* open each of the 2 resulting files to see what was generated:
+ CTlog-PIDsSorted.txt : contains the PID-query response messages, sorted and grouped by PID along with relative timing information
+ CTlog-CANidsSorted.txt : contains all the messages, sorted and grouped by CAN ID, along with relative timing information
* close the files
- run the command (adjusting the pathnames as required):
.../Scripts/processCANmsgLogs.sh -n .../Demo/CTlog.txt
* open each of the 2 resulting files to see what was generated:
+ CTlog-PIDs.txt : contains the unsorted/time-ordered PID-query response messages along with relative timing information
+ CTlog-CANids.txt : contains all the unsorted/time-ordered messages along with relative timing information
* close the files
- run the command (adjusting the pathnames as required):
.../Scripts/calcPIDvalues.sh -f .../Scripts/PIDforumlas.txt .../Demo/CTlog-PIDs.txt
* you could also use either the CTlog-PIDsSorted.txt or the CTlog.txt file, in which case it'll generate then use the CTlog-PIDs.txt file
* the PIDforumlas.txt file contains many of the standard PID formulas and a pair of "apply to all PIDs" example formulas ... you can add your own formulas
* open the resulting CTlog-PIDs-values.txt (CSV-format) file to see what was generated:
+ note the min/max info at the end of the file
+ if you're trying various formulas, this should help narrow in on the correct one
+ if you wanted to dynamically view this data in relation to other data, a "-values" file can be used/shown in RaceRender along with other data
- run the command (adjusting the pathnames as required):
.../Scripts/calcCANvalues.sh .../Demo/CTlog-CANids.txt
* you could also use either the CTlog-CANidsSorted.txt or the CTlog.txt file, in which case it'll generate then use the CTlog-CANids.txt file
* there are some example formulas defined in the script, 4 of which get applied (currently, I have no idea what CAN ID 61 is used for)
* open the resulting CTlog-CANids-values.txt (CSV-format) file to see what was generated:
+ note the min/max info at the end of the file
+ if you're trying various formulas, this should help narrow in on the correct one
+ if you wanted to dynamically view this data in relation to other data, a "-values" file can be used/shown in RaceRender along with other data
+ this approach may be ineffective when the CAN messages need to be decoded as multi-frame sequences.
- in the Demo folder, open the rcPartial.txt file and see that it's a partial RaceCapture log file that was captured on the SD card (then close the file)
- run the command (adjusting the pathnames as required):
.../Scripts/extractCSVdata.sh -fw 16 .../Demo/rcPartial.txt
* enter: 1,2,3,4,11,12 <return>
* open the resulting rcPartial-selected.txt (CSV-format) file to see what was generated:
+ note the min/max info at the end of the file
- run the command (adjusting the pathnames as required):
.../Scripts/slimLuaScripts.sh -h and read the help text
- run the command (adjusting the pathnames as required):
.../Scripts/processCANmsgLogs.sh -h and read the help text
- run the command (adjusting the pathnames as required):
.../Scripts/calcPIDvalues.sh -h and read the help text
- run the command (adjusting the pathnames as required):
.../Scripts/calcCANvalues.sh -h and read the help text
- open the file CAN PID Prober.lua and read the comments at the start of the file, including the area marked as CONFIGURATION.
- open the file CAN Msg Logger.lua and read the comments at the start of the file, including the area marked as CONFIGURATION.
- open the file captureRClog.sh and read the comments at the start of the file, including the area marked as CONFIGURATION.
That should give you an overview of the capabilities of these tools.
Running the Lua Scripts
- IMPORTANT: ensure you have a backup of your current configuration so you can restore it later
- the commented versions of the lua scripts are much too big to be loaded into the memory that's available
* use the "-.lua" versions for running on the RaceCapture unit (they were created using the slimLuaScripts.sh script)
- the amount of data that's available from the RaceCapture log via the app is minimal
* for small logging captures, the ON/OFF switching capability can help (on applicable units) by letting you capture short bursts, copy/paste, burst, copy/paste, etc.
* for large captures, connect the RaceCapture device, via USB, to a computer and use the captureRClog.sh script to capture the logging
- when running the CAN PID Prober.lua script, you should first disable the normal OBDII capabilities
Caveats
- While I've been using these tools for a while, I've also been constantly updating them. I think they're relatively bug free, but it is software so there are undoubtedly some bugs. If you find something and can give me a good explanation and/or test case, I can likely fix it fairly easily.
- in the CAN PID Prober.lua script, I have no way of testing the extended PID option and what's currently done is based upon what others have done/written about, but it could easily be wrong ... if you have better info, I'd like to hear
- the shell scripts have been developed/run only on macOS 10.12.6
* although I've tried to use fairly vanilla code, there may be issues when run on other operating systems/versions
* I've also included the Test folder so, using the files in the other folders, you could compare your results to the files in the Test folder (it's nowhere near a complete test, but it's something)
* the captureRClog.sh script is the most likely one to have portability issues ... other operating systems/versions may or may not support the character-special device files in the same way that macOS 10.12.6 does
Analysis Process
I'm using the following process to try to figure things out:
- use the CAN PID Prober.lua script to get all the PIDs supported by the vehicle
* capture the log onto a laptop
* post-process the captured data sorted into PID groups
* setup RaceCapture's OBDII channels with the PIDs you want
* compare values logged to SD card with known or "seems reasonable" values
* adjust for differences between logged and known/reasonable values
- for non-PID information, use the CAN Msg Logger.lua script to (optionally filter and) capture messages
* capture the log onto a laptop
* post-process the captured data sorted into CAN ID groups
* use calcPIDvalues.sh to run trial data calculations on post-processed data
* compare and correlate captured/calculated data to other data captured during the same test run: e.g., video data of console, etc. and other GPS/etc. data captured via the normal RaceCapture data-logging
- in some cases, it's useful to generate a RaceRender video to view the stream of calculated data values (using the generated CSV file) alongside other data
* use RaceRender's graphing capability, where applicable
* this can help identify trends and possible correlations
- as valid info is determined, program RaceCapture to collect the info I want when on-track (and/or use a lua script to capture and create/update virtual channels)
CAN Bus Analysis Tools (free)
CAN Bus Analysis Tools (free)
Last edited by aUzer on Mon Nov 28, 2022 12:03 am, edited 3 times in total.