Is there a way to speed up file creation with FATFileSystem?

Mbed OS Best practice
1

Hi everyone,
Recently i have some trouble with FATFileSystem inside a data logging project. Basically i am logging data to a microSD card (SPI interface), the sensor has a HW FIFO of 32 slots, which means the MCU needs to continuously readout the sensor’s FIFO within 64 millisecond intervals to avoid overflows.

The whole project works as expected with only one exception:
I am logging each minute into another file and it seems that with FAT only the first 3 files are generated rapidly enough. The more files there are on the SD card, the longer it takes to create the next one…
…and i get the impression as if file generation with FATFileSystem would be of blocking nature…thus i can not find a way to record continuously and store the data splitted up in several files.
I also find if file creation gets slow, it is not enough to just erase the files. I need to format the SD card to get fast creation times again…but that is only for the first 3 files again. Also tried to put each file in a separate directory. In which case the opening time for each file does not grow as rapidly, but the creation time of the folders does and those 2 add up just as rapidly.

Can anyone give me some advice how could i create a new file in a non-blocking way?
Would perhaps creating those files in a separate thread make any difference?
Or is FATFileSystem just not fast enough for such purpose? (If so which filesystem should i take?)

2

the problem is the FAT structure itself, it does not like many small files. Also littleFS(2) is very slow in this case, as there are more and more disk reads neccessary to find the next entry in a directory.
Appending to an existing file should perform much better.

3

If appending to existing files doesn’t prove to be fast enough you can try to use the plain SDBlockDevice (the erase and program methods, as shown in this example application) rather than a whole file system. However, this approach will require to build either an another Mbed or PC program to readout the data from the SD card later on and save it in files on another SD card or on a PC’s disk.

4

Thanks for both of you for the tips!
Made some benchmarking: seems if i generate empty files in a for loop first and then reopen them in append mode then it might work out. This way all the files are created really fast (within 40 microseconds all the 255 files), and then get reopened fast enough for the first 200 (within 32 milliseconds though reopening time grows again as the index increases). …however the 201th file takes all of a sudden 60+ milliseconds to reopen, which definitely would cause a FIFO overflow.

Will do some further testing, hope i will get constant results and FAT will not surprise me with yet another trick. In that case i can live with less than 200 files per directory.

I already tried the SDBlockDevice example without FAT before and that was blazing fast, i just thought adding FAT FS would be a straight forward decision to make the life of endusers easier. Did not expect such a huge difference. Anyway if FAT has some more aces under its sleeves, i will do what Zoltan proposed.

5

Made some extensive testing. Turns out FAT filesystem is a really bad choice as no matter how one structures the directory tree, the more files there are on the SD card the longer it takes to create a new file. Even using append mode works only until you have about 40-50 files on the card, afterwards you just can not create or reopen a file fast enough to avoid sensor FIFO overflow.

Here is a simplified code to compare different methods:

#include "mbed.h"
#include "FATFileSystem.h"
#include "SDBlockDevice.h"
#include "errno.h"

#define MODE 2 // 1: ofstream   2: std::FILE    3: mbed::FILE    4: plain POSIX ops without any filesystem (blob)
#define SPI_FREQ 12000000

UnbufferedSerial pc(USBTX, USBRX, 115200);
FileHandle *mbed::mbed_override_console(int fd) { return &pc; }

Timer T;

SDBlockDevice sd(MBED_CONF_SD_SPI_MOSI, MBED_CONF_SD_SPI_MISO, MBED_CONF_SD_SPI_CLK, MBED_CONF_SD_SPI_CS, 1000000);

FATFileSystem filesystem("fs");

#if MODE == 1
#include <cstdio>
#include <fstream>
#include <sstream>
ofstream ofile;

#elif MODE == 2
std::FILE *ofile;

#elif MODE == 3
mbed::File ofile;

#endif

uint8_t buf[512];

char fileName_prefix[5] = {0};
char fileName_suffix[26] = {0};
char fileName[32] = {0};

int main() {

  printf("Starting\r\n");
  fflush(stdout);

#if MODE < 4
  T.start();
  filesystem.mount(&sd);
  T.stop();
  printf("Mounting took: %llu\r\n", T.elapsed_time().count());
  fflush(stdout);
  T.reset();
#endif

#if MODE == 4
  if (0 != sd.init()) {
    printf("Init failed \n");
    return -1;
  }

  printf("sd size: %llu\n", sd.size());
  printf("sd read size: %llu\n", sd.get_read_size());
  printf("sd program size: %llu\n", sd.get_program_size());
  printf("sd erase size: %llu\n", sd.get_erase_size());
  fflush(stdout);
#endif

  sd.frequency(SPI_FREQ);

  // Create the dummy data
  for (int i = 0; i < sizeof(buf); i++) {
    buf[i] = (uint8_t)i;
  }

  printf("Dummy buffer created\r\n");
  fflush(stdout);

  for (int n = 0; n < 10; n++) {

#if MODE == 1 || MODE == 2
    strcpy(fileName_prefix, "/fs/");

#elif MODE == 3
    strcpy(fileName_prefix, "");
#endif

#if MODE < 4
    sprintf(fileName_suffix, "test_ofstream%d.txt", n);
    strcpy(fileName, fileName_prefix);
    strcat(fileName, fileName_suffix);
    T.start();

#if MODE == 1
    ofile.open(fileName, std::ofstream::app);

#elif MODE == 2
    ofile = fopen(fileName, "a");

#elif MODE == 3
    ofile.open(&filesystem, fileName, O_WRONLY | O_APPEND);

#endif

    T.stop();
    printf("Opening %s file took %llu\r\n", fileName, T.elapsed_time().count());
    fflush(stdout);
    T.reset();

#endif

    T.start();

    for (int o = 0; o < 16 * 512; o++) { // Write 4 megabytes of data
#if MODE == 1
      ofile.write((char *)&buf, sizeof(buf)); // 136KB/sec@12MHz

#elif MODE == 2
      fwrite(&buf, 1, sizeof(buf), ofile); // 78KB/sec@12MHz

#elif MODE == 3
      ofile.write(&buf, sizeof(buf)); // 151KB/sec@12MHz

#elif MODE == 4
      // erase in advance is needed
      if (0 != sd.erase((n * sizeof(buf) + o * sizeof(buf)), 512)) {
        printf("Error Erasing block \n");
        fflush(stdout);
      }
      // write data
      sd.program(&buf, (n * sizeof(buf) + o * sizeof(buf)), 512);   // 190KB/sec @12MHz

#endif

    }

#if MODE == 1
    ofile.flush();

#elif MODE == 2
    fflush(ofile);

#elif MODE == 3
    ofile.sync();

#endif

    T.stop();

    printf("Writing data took: %llu\r\n", T.elapsed_time().count());
    fflush(stdout);
    T.reset();
    T.start();

#if MODE == 1
    ofile.close();

#elif MODE == 2
    fclose(ofile);

#elif MODE == 3
    ofile.close();

#endif

    T.stop();
    printf("Closing took: %llu\r\n\n", T.elapsed_time().count());
    fflush(stdout);
    T.reset();
  }
  return 0;
}

In terms of file creation time there are only slight differences between using ofstream/std::FILE/mbed::File, however the data transfer rate using std::FILE is only about 50% of any other method. With ofstream i might be missing something as only the 1st cycle has good transfer rate, consecutive write cycles are ca. 10 times slower. Do not know why.

By the way i found the example located at the bottom of the documentation of File API is wrong as it uses std::FILE instead of mbed::File thus makes the documentation misleading.

So if anyone needs to log data to an SD card without a sensor FIFO overflow on a single core MCU will need to use the SD card like a blob storage without FATFS or LittleFS. In which case one shall store the starting addresses&length of each recorded data stream in a separated memory range of the SD card that would allow identication&retrieving valid data in a later moment. In other words one would need to come up with a very basic custom filesystem.