Looking for some advice on how to decrease the time taken to read the sd card file list.
The situation i have run into is that when i have 50, 60, 70 or more log file on the SD card and use the ‘Root_file_exists’ function from the 'rootdir.c library to find the next available file name (eg similar to existing logomatic code) the routine takes considerable time; greater than 3-400 ms (for 30-40 file) and 1.2-2 sec (for 70+) files.
Now this seems to be independent of the file size; so I’m thinking that it has some relationship to the size of the way the file allocation table is read… but i could be way off as well.
If is a matter of determining what the next file name should be, why not store the name (or the next sequential number) in EEPROM and do away with the scanning of the FAT altogether?
Had thought of both those options; I do actually hold the next sequence number in memory for that occasion when i need to create the next file.
Here is code:
void createFile(char cFileName[], char cFileContent[], char cOpen){
//Usage: createFile(char cFileName[], char cFileContent[], char cOpen);
//Inputs: char cFileName[] - Filename to be created
// char cFileContent[] - array of content to be added to file
// char cOpen - '0' to close file after creation
//Outputs: None
//Description: creates the a file based on content passed and close if flag not set
#ifdef DEBUG
sendDataToRadio("## createFile",1);
#endif
signed int stringSize;
#ifdef DEBUG
sendDataToRadio("check if file exists: ",0);
sendDataToRadio(cFileName,0);
#endif
if(!root_file_exists(cFileName)){ // If the file doesn't exist, then create it!
#ifdef DEBUG
sendDataToRadio(" File Doesn't Exist",1);
#endif
fd = root_open_new(cFileName);
if(fd == NULL){ // need to do something if file cant be created
#ifdef DEBUG
sendDataToRadio(" Error Creating File",1);
#endif
while(1){
flashStatLights(50);
}
}
}
else{
#ifdef DEBUG
sendDataToRadio(" File Exists",1);
#endif
fd = root_open(cFileName);
if(fd == NULL){ // need to do something if file cant be created
#ifdef DEBUG
sendDataToRadio(" Error Opening File",1);
#endif
while(1){
flashStatLights(50);
}
}
}
// store content
stringSize = strlen(cFileContent);
if (stringSize >0){
fat16_write_file(fd, (unsigned char*)cFileContent, stringSize);
sd_raw_sync();
#ifdef DEBUG
sendDataToRadio("Content Written to File",1);
#endif
}
if(cOpen=='0'){
#ifdef DEBUG
sendDataToRadio("File Closed",1);
#endif
fat16_close_file(fd);
}
}
Problem is that the ‘Root_file-exists’ function still takes the same amount of time to do its check and the more files the longer this time is.
Whilst i thought of it i’m just not that keen on not doing the file exists check; that would seem rather poor programming on my behalf.
I will preface my comments by saying that I haven’t used this particular library, so the comments may not be relevant to your case. I’m surprised at how long it takes to scan the directory if it has more than 60/70 files, so you may need other strategies to keep the performance at an acceptable level. If the file system has a flush call, maybe you might want to keep the file open and flushing after every write.
FAT file systems have a certain number of files that are in the root directory. The root is usually represented as a contiguous block of memory or disk space. These entries are quickly and easily accessed. Creating folders (and expanding the files in a folder) is done by the FAT system by creating a file entry (hidden from you) and then storing the directory information in the file. As the number of entries exceed what can be stored in a sector, a new (possibly non-contiguous) sector is created and used. There is overhead in creating and walking over these additional entries, and this might account for your delay after 60/70 files. Maybe 64 is your magic number for the number of files before you starting seeing significant performance issues.
If you can get a more detailed error result from the file open function, I’d consider refactoring the code to get rid of the file exists call. My reasoning is that you can try to open the file. If it fails because the file doesn’t exist, then create it. This will eliminate one directory scan (one scan to see if the file exists and then a second to actually open the file).
riden:
FAT file systems have a certain number of files that are in the root directory. The root is usually represented as a contiguous block of memory or disk space. These entries are quickly and easily accessed. Creating folders (and expanding the files in a folder) is done by the FAT system by creating a file entry (hidden from you) and then storing the directory information in the file. As the number of entries exceed what can be stored in a sector, a new (possibly non-contiguous) sector is created and used. There is overhead in creating and walking over these additional entries, and this might account for your delay after 60/70 files. Maybe 64 is your magic number for the number of files before you starting seeing significant performance issues.
Hey, i think your onto something there… have been able to some some time doing some testing around this. There is a definite change in the time the program takes when there is less than 63 files more than 64 more files. Will do some more testing and update this shortly.