A `torch.utils.data.Dataset` demo

[jinyu121]

Hi~

Look at this~ https://gist.github.com/jinyu121/02a505bb2547006ceb3fdbba3bee556c

Works fine (certainly) to me

[longcw]

For anyone who wants to use this dataloader, you can move imdb.parse to collate_fn and convert numpy array to tensor. Then the data processing part will run on multiple processes, and the data will be shared on the memeroy by using tensor.
Another solution is to move _im_processor to __get_item__.

[ztyxd]

@jinyu121 this web https://gist.github.com/jinyu121/02a505bb2547006ceb3fdbba3bee556c may loss
efficacy. Can u give me another web please? I really want to konw how to use the torch.utils.data.Dataset instead of multiprocessing

[hixiaye]

For anyone who wants to use this dataloader, you can move imdb.parse to collate_fn and convert numpy array to tensor. Then the data processing part will run on multiple processes, and the data will be shared on the memeroy by using tensor.
Another solution is to move _im_processor to __get_item__.

Hi, I don't understand what the move imdb.parsetocollate_fn meaning. And where I should convert numpy array to tensor? Could you give me some advices? thx

[hixiaye]

Hi~

Look at this~ https://gist.github.com/jinyu121/02a505bb2547006ceb3fdbba3bee556c

Works fine (certainly) to me

Hi, I don't understand what the move imdb.parsetocollate_fn meaning. And where I should convert numpy array to tensor? Could you give me some advices? thx