Using WebDataset on TPU + Google Cloud Storage

[zw615]

Hi, there!

I wonder if it is possible to use WDS data stored on GCS Bucket when training on TPU. I know the recommended format is TFDS when using GCS Bucket + TPU. But I already have a dataset in WDS format. And I have noticed the parser_wds.py in bits_and_tpu branch, so I guess it is possible to do so.

What do I need to do to read WDS files on GCS Bucket? Like replacing every filepath with gcsfs.GCSFileSystem().open(filepath)?

Thanks a lot!

[rwightman]

@zeyuwang615 yeah, in theory it should but I ran into issues with the gs support, gsutil pipes locking up / erroring out frequently, etc when I was trying to get it working in the past so I never had it working properly. I've found tf.data (TFDS) works really well on gs:// and have had webdataset working very nicely with s3, go figure.

[zw615]

Got it. In this case, it seems the problem really lies in the GCP support for wds, but not the data format itself. I see what I can do. Thanks Ross!