[AOSP] add file descriptor support to file_data_loader

[davidlin54]

Summary:
For Google's ODP, the model will execute in a process that does not have access to the file system. Instead, the caller must open the file then pass ownership of the FD to the model executor.

1. Added support for reading files by FD passed by the prefix "fd:///" (copied from tensorflow https://cs.android.com/android/platform/superproject/main/+/main:external/federated-compute/fcp/tensorflow/file_descriptor_filesystem.cc;l=41;bpv=1;bpt=1) into file_data_loader.
2. Added test cases by opening file first and passing the FD to the data loader

Test Plan:

1. run C++ tests
2. use script with following instruction:

$ cd aosp
$ source build/envsetup.sh
$ lunch aosp_arm64-trunk_staging-userdebug
$ m executor_runner
$ FILENAME="<model_path>"
$ exec {FD}<${FILENAME}     # open file for read, assign descriptor
$ echo "Opened ${FILENAME} for read using descriptor ${FD}"
$ out/host/linux-x86/bin/executor_runner --model_path fd:///${FD} --is_fd_uri=true
$ exec {FD}<&-    # close file

result:

Opened out/host/linux-x86/bin/add.pte for read using descriptor 10
I 00:00:00.000256 executorch:executor_runner.cpp:94] Model file fd:///10 is loaded.
I 00:00:00.000279 executorch:executor_runner.cpp:103] Using method forward
I 00:00:00.000282 executorch:executor_runner.cpp:150] Setting up planned buffer 0, size 48.
I 00:00:00.000294 executorch:executor_runner.cpp:173] Method loaded.
I 00:00:00.000310 executorch:executor_runner.cpp:183] Inputs prepared.
I 00:00:00.000336 executorch:executor_runner.cpp:192] Model executed successfully.
I 00:00:00.000345 executorch:executor_runner.cpp:196] 1 outputs: 
Output 0: tensor(sizes=[1], [2.])

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 83f3ff9 with merge base 97a4600 ():

NEW FAILURE - The following job has failed:

• pull / test-binary-size-linux-gcc / linux-job (gh)
  RuntimeError: Command docker exec -t 6caca31670f698c518f9b27fbd70f78324f3d58c32be9060b8a8caa88cf17f94 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kirklandsign]

Should be mention that the fd ownership will be passed?

[davidlin54]

does passing the FD ownership mean that ET is responsible for closing the file afterwards? or is ownership of the FD independent of ownership of the file opened?

edit: read the destructor for FileDataLoader, makes sense now!

[kirklandsign]

Probably not necessary to put this file to repo. Maybe just mention it in commit message?

[kirklandsign]

Overall LGTM! Let's try to merge to main, not aosp-test?

I will rebase aosp-test later

[kirklandsign]

Do you think it's possible to factor the common code between here and FileDataLoader::from?

[davidlin54]

@kirklandsign moved centralized logic to a private method

[dbort]

This shouldn't have been part of FileDataLoader; we should create a new DataLoader with this functionality. This also caused the test-binary-size-linux-gcc job to start failing: https://github.com/pytorch/executorch/actions/runs/11674336719/job/32509252770