Constraints on SD card access: no free file descriptors

[eliosmoreira]

Platform

ESP32

IDE / Tooling

PlatformIO

What happened?

Hi, I am using Arduino-audio-tools with this library but in different RTOS tasks.
I made a custom class networkManager which simplifies the user experience in the main setup/loop functions for this library. I made a callback function _jsProcessorCallback so whenever the user needs to process a javascript file variables, he can just attach his own templateProcessor.

The error pops when an audio file is already being played (not in the same directory) and the AWS has a request for its index: simple html, css and js files (Using minified versions from 450 lines of code in total).
In the user template processor callback, the variable "%audio_files%" triggers a process of accessing SD and retrieving a list of files in a directory. (~25 files in total).
I cannot find a clear answer but i believe this has something to do with AWS internal queuing. If i use AWS method for accessing SD directly, it probably takes too long to close the files so i run out of file descriptors and so i cannot get the list.

User callback:

String jsProcessor(const String &var) {
    if (var.equals("audio_files")){
        String path = "/music/background";
        File dir = SD.open(path);
        if (!dir || !dir.isDirectory()) {
            Serial.println("Failed to open directory");
            dir.close();
            return String();
        }

        String fileList;
        File file = dir.openNextFile();

        while (file) {
            if (!file.isDirectory()) {
                LOG.debug(file.name());
                fileList += "\"" + String(file.name()) + "\",";
            }
            file.close();
            file = dir.openNextFile();
        }
        dir.close();
        LOG.debug(fileList.c_str());
        return fileList;
    }
}

The buggy way (works if audio player is not being used):

// ---- part of onNotFound callback ----
// Send processed JS
if(_jsProcessorCallback && mime == "text/javascript") {
    request->send(SD, filePath, String(), false,
            [this](const String& var) -> String { return this->_jsProcessorCallback(var); });
    return;
}

// Send non-processed
request->send(SD, filePath, mime);
return;

But if i get the file, store the content into a String and send it, the error disappears (always works):

// ---- part of onNotFound callback ----
// Get file content
String content;
File file = SD.open(filePath);

if(file && !file.isDirectory()){
    size_t size = file.size();
    char* buffer = (char*)malloc(size + 1);
    file.readBytes(buffer, size);
    buffer[size] = '\0';
    content = String(buffer);
    free(buffer);
}
file.close();

// Send processed JS
if(_jsProcessorCallback && mime == "text/javascript") {
    request->send(200, mime, content,
        [this](const String& var) -> String {return this->_jsProcessorCallback(var);});
    return;
}

// Send non-processed
request->send(200, mime, content);
return;

Stack Trace

E (9795) vfs_fat: open: no free file descriptors
[  9819][E][vfs_api.cpp:301] VFSFileImpl(): fopen(/sd/music/background/B25.wav) failed

Minimal Reproductible Example (MRE)

Cannot get an MRE at the moment

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[eliosmoreira]

In the meantime, incrementing file descriptors in SD.begin() mitigates the error but still, why so many files opened at the same time? What are the underlying processes that make this happen and is there any way of reducing its use?
And, maybe this should be a wiki topic? 🤔

[mathieucarbou]

Moving that to a support discussion since this is not related to ESPAsyncWebServer but more about how FS works on ESP32.

There are some issues in browsing the FS: slow and takes resources.

You can also see this discussion: mathieucarbou/ESPAsyncWebServer#165 (reply in thread)

I would suggest to refactor your app, use the lest possible disk access (eventually refactor your UI to only serve 1 big file and/or use WebSocket like ESP-DASH). Flash access is slow and should be avoided as much as possible, and concurrent access even more.

[me-no-dev]

what do you see in the network tab of the browser debug console? How many javascripts are requested? In your handler up there you print directory list, which consumes some open file descriptors. Can you provide a full minimal compilable example and maybe screenshot of the bowser network requests when a fail occurs?

[eliosmoreira]

I have done some debugging, and at the error point i have opened:

• Audiotools audio files source directory
• Audiotools audio file being played
• AWS javascript file which is about to be template-processed:
  • in the callback: directory of the audio files
  • in the callback: one audio file per loop

This makes 5 file descriptors which is the max default. All should work flawlessly. So, i believe Audio-tools or AWS could possibly have one more opened file/directory somewhere. This makes sense with my findings above: if i bring the file to a String and then close the file, the callback works as expected (less one file descriptor being used).

Now that this makes sense, we can close this topic. I am sorry for the confusion and thank you @mathieucarbou @me-no-dev ! Have a great day and keep the good work 🙏