openlog firmware, puzzle

Other Miscellaneous
1

I’m using the openlog to record GPS output at 5hz, it does this well, but I’d like to increase the frequency to 10hz. The standard openlog firmware’s write speed is too slow, I get dropped points.

Here’s the output from the analysing software.

Observed time distribution (in ms):

100 11494

200 57

so in roughly 11,500 data points there’s 57 misses.

So I thought changing to the openlog_light firmware would allow me to use 10hz without any missed points.

However this was not the case, it actually made things a lot worse

Observed time distribution (in ms):

100 14434

200 5245

Now there’s 5,245 misses in about 20,000 data points.

Here’s the end notes from the arduino compiler

this is the default openlog firmware

Sketch uses 27,072 bytes (83%) of program storage space. Maximum is 32,256 bytes.

Global variables use 1,273 bytes (62%) of dynamic memory, leaving 775 bytes for local variables. Maximum is 2,048 bytes.

And here’s the openlog_light

Sketch uses 16,166 bytes (50%) of program storage space. Maximum is 32,256 bytes.

Global variables use 1,557 bytes (76%) of dynamic memory, leaving 491 bytes for local variables. Maximum is 2,048 bytes.

Low memory available, stability problems may occur.

So it seems the firmware is smaller but global variables are much bigger.

Am I doing something wrong here?

Is there anyway to reduce the global variables? (whatever they are?)

2

Hi decrepit,

Are your tests for dropped data points with your GPS running at 5Hz or 10Hz? I am not exactly sure why you would be running into more dropped data but it might be something with your receive buffer overflowing. Also, what baud rate are you running on the OpenLog/GPS and are you feeding the data directly into the OpenLog from the GPS or are you passing it through a microcontroller? Does changing that help or hurt your dropped data? Have you tried using the minimal firmware if it can work for your setup?

As for the global variables, that increase is mostly due to the receive buffer increasing in size with the light and minimal versions of the firmware. The light firmware sets the receive buffer to 850 bytes and the minimal sets it to 1024 (standard is set to 512).

I’ll try and speak with the engineer who wrote the firmware to see if they have any insights into what may be causing the increased data drops using the light firmware.

3

Hi TS-Mark, Thanks for your reply

WOOPS!!! That must be my oldtimers kicking in, running at 10hz as the 100ms shows.

The gps is connected directly to the openlog. Yes I also assumed buffer overflow, I thought the spare memory shown at the end of the compilation was the buffer, and that was causing the problem because it’s smaller for the light and even smaller again for the minimal.

I’m running at 115200, one of the reasons I haven’t tried the minimal, because I’m not sure I’m smart enough to set the board rate by code. But I will give it a try

4

I’ve just checked the instructions for using openlog_minimal, and they are over my head. I could try it but I suspect the baud rate will be wrong.

Is there a single step guide about how to program the baud rate to 115200, I couldn’t find one.

5

I don’t understand code, but I had a look inside minimal and noticed this

SdFat sd;

long setting_uart_speed = 115200;

Looks like default baud is 115200, so I’ll give it a try tomorrow and see what happens, fingers crossed.

6

God news and bad news, openlog-minimal works at 115200 without any coding. Bad news is that it’s the worst for missing points.

openlogger Minimal

Opening files ‘/media/decrep/OPENLOG/minimal test2.ubx’ …

File: minimal test2.ubx

Observed time distribution (in ms):

100 7526

200 24609

So I’m going to try something I have very little idea about,

Seeing that the smaller amount of “local varibles” left, according the Arduino IDE, gives the most missing points. I’m going to see if I can decrease the allotted buffy size of 1024 to the left over

Openlog-Minimal compilation end note

Sketch uses 12,906 bytes (40%) of program storage space. Maximum is 32,256 bytes.

Global variables use 1,705 bytes (83%) of dynamic memory, leaving 343 bytes for local variables.

343 bytes.

If my somewhat hazy theory is correct that should give 1024bytes left over, if so will that improve things??? Watch this space, you may soon find out.

7

OK good so far. (I think)

Sketch uses 12,906 bytes (40%) of program storage space. Maximum is 32,256 bytes.

Global variables use 1,024 bytes (50%) of dynamic memory, leaving 1,024 bytes for local variables. Maximum is 2,048 bytes.

That’s not quite what i expected, but now I have to do a bit of soldering and leave the device running for an hour to test the effect.

8

Well that blows my theory, but it is a small improvement.

100 8617

102 1

200 18831

we’ve gone from about 3:1 missing to 2:1 missing.

I’m now at a loss, but I suspect I’m either doing something wrong or there’s a bug

9

just in case it’s relevant, it’s a SanDisk 8Gb HC1 card formatted by sdformatter to fat32

10

Just a totally random experiment this time, just changing the variables to see what happens, in case it helps.

So this time I did two things at once, turned both buffers on at 5012bytes each, another minor improvement.

File: rx+tx 512.ubx

Observed time distribution (in ms):

100 9233

200 12237

4:3 missing

11

Hi again,

It looks like you are on the right track just playing around with variables here to try and reduce the lost data. The SD card your using should have no issues writing at high speeds so it’s definitely something going on with the OpenLog/firmware. One quick idea (even though it kind of goes against what you are trying to do here) is to run the OpenLog and your GPS module at a lower baud rate. Try dropping it to 57600bps or 38400 and see if that helps anything. At the very least, it will increase the time it takes for the receive buffer to fill up.

The engineer who wrote the firmware is out of the office today so I will try and talk to them tomorrow or later in the week if they are out longer and will get back to you with any suggestions they have.

12

Thanks Mark, I’m open to any suggestions, I’ll give the next baudrate down a try. It’s all data gathering, something may come out of it at the end.

13

OK, so that’s interesting, here’s what I’ve got. I’ve gone back to the standard “openlog” firmware

Open log 57600 baud rate

Observed time distribution (in ms):

100 2589

200 11147

Openlog 38400 Baud

Observed time distribution (in ms):

100 24048

200 2584

300 17

Openlog 19200 baud

Observed time distribution (in ms):

100 26778

200 560

Openlog 9600 baud

Observed time distribution (in ms):

100 21779

200 609

300 207

400 1

So best baud rate is 19200, a bit of a surprise, when I started this I was assured 115200 was the way to go.

I’ll see what happens tomorrow, I guess I should try openlog-light at 19200.

14

A sanity check, saving to the computer with u-center at 19200 just to make sure there’s no GPS problem, all this seems a bit random.

Opening files ‘/media/decrep/TSB USB DRV/GPS/gps/COM3_191002_034243.ubx’ …

File: COM3_191002_034243.ubx

Observed time distribution (in ms):

100 31998

0 time errors (0 above 5.0 knots), about 0 missing points

15

So back to openlog-light at 19200

Observed time distribution (in ms):

100 43337

200 867

that’s about 50:1 good to missed, marginally better than openlog at 48:1 good to missed. It’s getting close, can I do better?

16

Hi again decrepit,

I have a few other suggestions that might help get better results but you might be pushing the limits of the ATMega328 processor on the OpenLog. One suggestion is to try modifying the OpenLog-Light firmware and removing serial prints and then play around with increasing your receive buffer size. Most of the serial prints are not necessary, especially in your application so removing those can free up some memory for a larger receive buffer. From there, you’ll want to play around with baud rates/buffer size to see what works best. 19200 might be the best baud rate here, though.

The other suggestion would be to try a larger capacity SD card or one that can write even faster than your HC1. A larger capacity will help increase the write speed so you can empty that receive buffer more quickly so it does not overflow and you lose data.

The engineer who wrote the firmware is out for the week unfortunately so I can’t pick his brain but I’ll send them an e-mail and see if they can get back to me with any other suggestions for continuous logging to the OpenLog at high speeds.

17

Thanks Mark,

That sounds well worth a try, I’m busy tomorrow, I’ll get back with any results. I have a bigger faster card in a logomatic v3 version, but I thought the openlogger had a problem with newer faster cards, so haven’t tried it yet.

I also noticed while I was playing around the firmware was set to class h, maybe I have to change that to match the sandisk extreme uhs-1 U3

18

Well bigger faster card doesn’t make much difference.

Openlog-light Sandisk Extreme 32GB HC1 V30 U3 A1.

old linux formatted? with plenty of files on.

Observed time distribution (in ms):

100 24098

200 8357

Ratio less than 3:1

So reformatted with SdFormater;

sandisk extreme formated with sd formatter

Observed time distribution (in ms):

100 36055

196 1

200 1408

that’s better 25:1 but we’ve had better with the old card.

Mark, when you say remove serial prints, do you mean commenting out something like this. trouble is, I’ve no real idea what’s going on here, I’m just guessing where to start and where to stop with the “//”

//Handle errors by printing the error type and blinking LEDs in certain way
//The function will never exit - it loops forever inside blink_error
void systemError(byte error_type)
{
  NewSerial.print(F("Error "));
  switch(error_type)
  {
  case ERROR_CARD_INIT:
    NewSerial.print(F("card.init")); 
    blink_error(ERROR_SD_INIT);
    break;
  case ERROR_VOLUME_INIT:
    NewSerial.print(F("volume.init")); 
    blink_error(ERROR_SD_INIT);
    break;
  case ERROR_ROOT_INIT:
    NewSerial.print(F("root.init")); 
    blink_error(ERROR_SD_INIT);
    break;
  case ERROR_FILE_OPEN:
    NewSerial.print(F("file.open")); 
    blink_error(ERROR_SD_INIT);
    break;
  }
}

void setup(void)
{
  pinMode(statled1, OUTPUT);
19

// comments out a single line, beginning with the //

In this case, just comment out the lines beginning with NewSerial.print

// NewSerial.print(F(“card.init”));

You can look at the serial datastream to see which prints get called the most often and start by just commenting them out as they would be the ones affecting processor load the most.

/mike

20

Thanks Mike I’ll give that a try