I think I found a new problem. I think GEMS wants lat/long in dms, not decimal. Unfortunately I don't know how it represents dms in text, so I'm not sure how to handle the conversion.
edit - I assumed it was dms because in the GEMS DA program it was presenting lat/long in the track view as dms. I found this post though that might indicate it's expecting radians. Still not sure on either, I'll try doing conversion at some point and seeing what DA things. Here's the post I found:
http://www.gems.co.uk/forum/viewtopic.php?f=6&t=607
Data Analysis Software - Motec, Gems, AEM
Moderators: JeffC, rdoherty, stieg, brentp
How did you change things in Gems? I goofed around with it but didn't figure it out (of course it was after 3am and i was brain dead tooneoraptor wrote:I noticed the same problem, and I solve it by changing the unit of latitude and longitude channels in Gems.
As for the link you gave, I wanted give both the racing line and the track layout. That's why they suggest me to export the data in kml format and import it in Google maps or equivalent
).I just play a bit with your script and AEM Data Analysis and here are my observations :
* if I don't remove manually the line with the wrong characters, the conversation is completed successfully, but I cannot import it. Maybe it is safer to remove the incorrect lines instead of replacing the bad characters with 0 (Here is a sample file with error. Search
* I needed to change the end of line (script line 209) from "\r\n" to "\n" to make it works (maybe because I have european encoding)
* In the header, I have some data (probably link with the previous point). I will make some other tries tomorow
* Otherwise, it works fine and fast. Congrats
In order to use the latitude/longitude channels, you just need to set their unit:
* open a channel list view
* right click on lattitude/longitude and click Channel properties
* enter degree in base unit. From now, it will work for the various tools (Distance calculation and track layout)
In order to have the track map :
* Step 1: valid distance channel :
* Go to Data > Distance Channel setup and configure one of the 3 options (distance direct / from speed / from GPS)
* Step 2 : build a track map :
* Add a track widget
* click on the wrench > Edit
* Instead of Yaw/G, use GPS and specify your latitude/longitude channels
If you need more details, just ask
It is good to have some good tools (and the math channels are also very powerfull in AEMData. Definetely a nice piece of software !!
* if I don't remove manually the line with the wrong characters, the conversation is completed successfully, but I cannot import it. Maybe it is safer to remove the incorrect lines instead of replacing the bad characters with 0 (Here is a sample file with error. Search
* I needed to change the end of line (script line 209) from "\r\n" to "\n" to make it works (maybe because I have european encoding)
* In the header, I have some data (probably link with the previous point). I will make some other tries tomorow
* Otherwise, it works fine and fast. Congrats
In order to use the latitude/longitude channels, you just need to set their unit:
* open a channel list view
* right click on lattitude/longitude and click Channel properties
* enter degree in base unit. From now, it will work for the various tools (Distance calculation and track layout)
In order to have the track map :
* Step 1: valid distance channel :
* Go to Data > Distance Channel setup and configure one of the 3 options (distance direct / from speed / from GPS)
* Step 2 : build a track map :
* Add a track widget
* click on the wrench > Edit
* Instead of Yaw/G, use GPS and specify your latitude/longitude channels
If you need more details, just ask
It is good to have some good tools (and the math channels are also very powerfull in AEMData. Definetely a nice piece of software !!
-
andylaurence
- Posts: 43
- Joined: Tue Apr 30, 2013 10:35 am
I've been using XoomCentre for analysis. It's a great bit of software and free. It baulks if I log different channels at different rates, so I log all channels at the same rate and I've written a plugin to import the data directly from the SD card. Happy to share - just PM an email address and I'll send it over. Plugins are written in Javascript and you can use them to create maths channels, draw graphs, output data and do all sorts.
Sprinting an ADR Sport 2
www.endurancelay.co.uk
www.endurancelay.co.uk
- Ugh, I removed the code that was handling your non-numerical erroneous data. I was previously taking the last valid data point and overwriting any data that was characters. When I re-factored the code to handle midnight time rollover I consolidated a couple functions and one of the removed ones had the non-number handling section. doh. I'll re-add it soon.neoraptor wrote:I just play a bit with your script and AEM Data Analysis and here are my observations :
* if I don't remove manually the line with the wrong characters, the conversation is completed successfully, but I cannot import it. Maybe it is safer to remove the incorrect lines instead of replacing the bad characters with 0 (Here is a sample file with error. Search
* I needed to change the end of line (script line 209) from "\r\n" to "\n" to make it works (maybe because I have european encoding)
* In the header, I have some data (probably link with the previous point). I will make some other tries tomorow
* Otherwise, it works fine and fast. Congrats
- The \r isn't really needed anywhere... I think I put it in there so the output would open in notepad on my windows box just fine haha. So I'll just remove that.
- You'll have to elaborate on what you mean here. I don't understand.
So I tried this. I didn't know where to find the degrees measurement, never thought to look under Angle (which now seems obvious). However after modifying the units to degrees I still can't get the track map to auto calculate properly. Instead of a map of the route taken in my log it shows a long straight line, and hovering over the line the coordinates aren't even close to matching what's in the lat/long data. I know my lat/long are right as I can somewhat see a map if I do an x/y plot, though this is backwards due to how lat/long translate to x/y. Any ideas? I thought perhaps it was that you were using the AEM DA and I was using GEMS DA, but I tried AEM DA and it does the exact same thing. This won't be a huge deal for pre-programmed tracks I don't think since we'll have the KML (I'll be testing this next weekend at the Summit Point for time trials), but for open tracks or tracks that aren't pre-mapped this would still be useful so I'd really like to know how you're getting it working.neoraptor wrote:In order to use the latitude/longitude channels, you just need to set their unit:
* open a channel list view
* right click on lattitude/longitude and click Channel properties
* enter degree in base unit. From now, it will work for the various tools (Distance calculation and track layout)
In order to have the track map :
* Step 1: valid distance channel :
* Go to Data > Distance Channel setup and configure one of the 3 options (distance direct / from speed / from GPS)
* Step 2 : build a track map :
* Add a track widget
* click on the wrench > Edit
* Instead of Yaw/G, use GPS and specify your latitude/longitude channels
Maybe we can exchange data? I can't seem to attach my data log due to size (even compressed it's larger than the 256KB limit) but if you can pm me your email address I'll mail it to you, and you can respond with yours if you don't mind.
Btw thanks for mentioning AEM DA... didn't realize the GEMS DA didn't have the Math channels enabled and AEM DA did. I'll be switching to AEM DA
So I was wrong earlier when I said I'd removed the code that should handle non-numerical data points. It's actually still there. I'm not sure how I got confused in my own code but it's still removing it when it's nicely contained between commas.
After seeing the code is still present I went and tried to use your 3-line log data example showing the erroneous data points. You have multiple issues in that data. Not only do you have non-numerical entries in the second data line for latitude and longitude, you also have extra data beyond the fields (according to the headers you only have 11 fields). Line one has five extra fields, two has three, and line three has five again. I have no idea why this might be but my script drops that extra data. I don't think that's a problem though, right?I wonder though if this is indicative a problem with your rc unit. Maybe a bugged firmware or something else.
So since my code does appear to handle the non-numerical data points I removed the \r per your suggestion and re-ran it. I then tried to import into Dlog99 and it failed. Re-adding the \r though and your data loaded just fine. This must be why I added the \r, so that Dlog99 was reading carriage returns properly (I have a vague recollection of this, but I was programming with half a brain so I can't recall exactly).
So try keeping the \r and running my script on your data and see what happens. If you still though would rather I remove the offending lines from the output I'll do that. Also, I'll pm you my email address so you can send me your datalog so I can try things on my end.
After seeing the code is still present I went and tried to use your 3-line log data example showing the erroneous data points. You have multiple issues in that data. Not only do you have non-numerical entries in the second data line for latitude and longitude, you also have extra data beyond the fields (according to the headers you only have 11 fields). Line one has five extra fields, two has three, and line three has five again. I have no idea why this might be but my script drops that extra data. I don't think that's a problem though, right?I wonder though if this is indicative a problem with your rc unit. Maybe a bugged firmware or something else.
So since my code does appear to handle the non-numerical data points I removed the \r per your suggestion and re-ran it. I then tried to import into Dlog99 and it failed. Re-adding the \r though and your data loaded just fine. This must be why I added the \r, so that Dlog99 was reading carriage returns properly (I have a vague recollection of this, but I was programming with half a brain so I can't recall exactly).
So try keeping the \r and running my script on your data and see what happens. If you still though would rather I remove the offending lines from the output I'll do that. Also, I'll pm you my email address so you can send me your datalog so I can try things on my end.
I figured out why my track wasn't displaying properly in AEM DA. After entering the units for my lat/long like you said via the channel properties I was getting a big straight line which really puzzled me when the data looked fine. After digging around the Track Editor (going into Track View Configuration by clicking the wrench for the track map and clicking the Edit button) I noticed there's an interesting setting. It had "Entire Log" with two boxes, first with 00.000 and second with 25:12.206. More interesting was in the drop down were various options that didn't quite make sense to me at first but there was one named "Fixed Range". This made me think that the track view was being calculated based on a defined time range of data in the log.
Sure enough I was right. I realized that the beginning of my data has the gps blank, so when my conversion script ran it was zero'ing out the data until the real gps lat/long began showing up. This was screwing up the Track Editor's calculations! It basically thought I was starting at coordinates 0,0, then suddenly jumping way over to my actual coordinates, making the range of values that had to fit on the track screen HUGE, and of course reducing my map to a straight line. If I changed the dropdown from "Entire Log" to "Fixed Range" and set the start time to something like 3 seconds (a time after which I had real lat/long values), and the end time to the end of my log (I used seconds, I think it might take other formats too like HH:MM:SS but not sure) and hit the "Calculate" button at the bottom of the Track Editor window my track popped up! Attached is what I get now.
So how to handle this? I decided I might as well throw out data that doesn't have any GPS coordinates to make it simple. Since GPS data is pretty frequent you aren't losing much data, less than 1 second worth if you are using 1hz resolution. I updated my script to v1.2 to add code to remove the data lacking gps coord in the beginning. This fixed things nicely I've attached that version.
Sure enough I was right. I realized that the beginning of my data has the gps blank, so when my conversion script ran it was zero'ing out the data until the real gps lat/long began showing up. This was screwing up the Track Editor's calculations! It basically thought I was starting at coordinates 0,0, then suddenly jumping way over to my actual coordinates, making the range of values that had to fit on the track screen HUGE, and of course reducing my map to a straight line. If I changed the dropdown from "Entire Log" to "Fixed Range" and set the start time to something like 3 seconds (a time after which I had real lat/long values), and the end time to the end of my log (I used seconds, I think it might take other formats too like HH:MM:SS but not sure) and hit the "Calculate" button at the bottom of the Track Editor window my track popped up! Attached is what I get now.
So how to handle this? I decided I might as well throw out data that doesn't have any GPS coordinates to make it simple. Since GPS data is pretty frequent you aren't losing much data, less than 1 second worth if you are using 1hz resolution. I updated my script to v1.2 to add code to remove the data lacking gps coord in the beginning. This fixed things nicely
I've attached that version.- Attachments
-
- rccsv2gems_v1.2.zip
- (3.04 KiB) Downloaded 1044 times
-
- track.JPG (20.97 KiB) Viewed 187622 times
I think you can safely delete the line that contain "0" value for GPS data. But don't delete the lines where there is no GPS data as this might be wanted (ex: shock travel acquired @ 100Hz)
I already flash my unit to the latest firmware. I will try with an other SD card to solve my wrong character problem (it is really a sporadic fault as it only appears on a few logs).
Maybe we can check if each line has the right number of parameter according to the headers and remove it if it is not good. That way, we will have a more robust import.
Concerning the \r\n, I solve the problem :
I noticed a small error in your script : when you clean the header line, you suppress the \r (chain length is one to big).
In your script (line 68), this solves the problem :
I can then let \r\n and my output is now ok to be imported.
I send you some of my correct and incorrect logs so we can make the script more robust by removing invalid lines
I already flash my unit to the latest firmware. I will try with an other SD card to solve my wrong character problem (it is really a sporadic fault as it only appears on a few logs).
Maybe we can check if each line has the right number of parameter according to the headers and remove it if it is not good. That way, we will have a more robust import.
Concerning the \r\n, I solve the problem :
I noticed a small error in your script : when you clean the header line, you suppress the \r (chain length is one to big).
In your script (line 68), this solves the problem :
Code: Select all
for(my $i = 0; $i <= $#{$array[0]}[color=red][b]-1[/b][/color] ; $i++){I send you some of my correct and incorrect logs so we can make the script more robust by removing invalid lines
Neoraptor reminded me that he and I had worked quite a bit on the script offline and went through a number of revisions that I never shared. Here's the latest v1.6. I hopefully captured the major changes in the changelog (shown in the comments at the beginning of the script) but I think I left some stuff out as I was getting ready for a race and got a bit tired of working on the script so I'll have to peruse our emails between each other to make sure. Regardless here are the logged changes since 1.2:
## Changes: 1.6 - Added time interpolation. Found that Dlog99 was dropping data
## when data points had the same time values, which they would
## as this script was setting empty time values to the last known
## good time value. To prevent the dropping I added a function
## that interpolates time for data points between logged time
## and sets the interpolated time values in the output.
## 1.5 - Removed second tag_junk_lines function call from main as it
## was redundant. Added options for new parameters mingps and
## disable-gps-cleanup. "--mingps=X" sets the minimum
## number of gps satellites required for gps data to be valid,
## defaults to 4. "--disable-gps-cleanup" will disable cleaning up
## by the data (defaults to false) and switch pre-cleanup to setting
## all gps values prior to the first good set to the same as the
## first good set. Added lots of output to the
## various data processing functions to indicate their status.
## Enabled the gps lat/long conversion from degrees to radians
## to make AEM/GEMS DA setup more seamless since they expect
## radians by default.
## 1.4 - Fixed an issue with how we fill in the blanks. We were filling
## in the blanks with last known good values, including "good"
## values from lines tagged for removal. Changed to ignore those
## lines so values from them don't corrupt the rest of the data.
## 1.3 - Added comments. Changed junk character handling from setting
## values to prior "good" values to instead just delete the
## offending lines. Also changed ctrl M stripping to be all
## elements as random ^M were screwing up other text matches.
## Moved the ^M stripping to earlier in the pipeline too.
Hmm re-reading my changelog notes makes me cringe I am not a developer...
## Changes: 1.6 - Added time interpolation. Found that Dlog99 was dropping data
## when data points had the same time values, which they would
## as this script was setting empty time values to the last known
## good time value. To prevent the dropping I added a function
## that interpolates time for data points between logged time
## and sets the interpolated time values in the output.
## 1.5 - Removed second tag_junk_lines function call from main as it
## was redundant. Added options for new parameters mingps and
## disable-gps-cleanup. "--mingps=X" sets the minimum
## number of gps satellites required for gps data to be valid,
## defaults to 4. "--disable-gps-cleanup" will disable cleaning up
## by the data (defaults to false) and switch pre-cleanup to setting
## all gps values prior to the first good set to the same as the
## first good set. Added lots of output to the
## various data processing functions to indicate their status.
## Enabled the gps lat/long conversion from degrees to radians
## to make AEM/GEMS DA setup more seamless since they expect
## radians by default.
## 1.4 - Fixed an issue with how we fill in the blanks. We were filling
## in the blanks with last known good values, including "good"
## values from lines tagged for removal. Changed to ignore those
## lines so values from them don't corrupt the rest of the data.
## 1.3 - Added comments. Changed junk character handling from setting
## values to prior "good" values to instead just delete the
## offending lines. Also changed ctrl M stripping to be all
## elements as random ^M were screwing up other text matches.
## Moved the ^M stripping to earlier in the pipeline too.
Hmm re-reading my changelog notes makes me cringe
I am not a developer...- Attachments
-
- rccsv2gems_v1.6.zip
- rccsv2gems v1.6
- (5.61 KiB) Downloaded 1059 times
Up on github here:
https://github.com/autosportlabs/RCP2GEMS
jbf11 - let me know your github account and I'll make you a maintainer. Thanks for this! Thinking we should make it a 'save as' in Race Analyzer.
https://github.com/autosportlabs/RCP2GEMS
jbf11 - let me know your github account and I'll make you a maintainer. Thanks for this! Thinking we should make it a 'save as' in Race Analyzer.