Thoughts on padding images of different sizes for VisionTransformer?

[priyamtejaswin]

Hello,

I have a problem where I am trying to preserve the image aspect ratio (as best as possible).

The shorter side of each image is resampled to 384 (so I can use vit_base_patch32_384), and the longer side has a upper limit of 640.

Consider a batch with two images,

batch = [
    img_0,  # (384 x 384)
    img_1,  # (384 x 640)
]

I know can pad img_0 to match the dimensions of img_1.

Questions

1. Is there some way for the VisionTransformer to ignore the padded pixels?

2. Ignoring all padding might be impossible at times, since the patches have a fixed size. But could ViT ignore patches which only contain padding?

[samils7]

Hi, I had the same questions for a while and here are my thoughts.

We know where we are padding in every image. So, how many padded patches will be created and their positions.

HF transformers have bool_masked_pos parameter in the forward func of embeddings. I think we can use this masking on padded patches.

if bool_masked_pos is not None:
    seq_length = embeddings.shape[1]
    mask_tokens = self.mask_token.expand(batch_size, seq_length, -1)
    # replace the masked visual tokens by mask_tokens
    mask = bool_masked_pos.unsqueeze(-1).type_as(mask_tokens)
    embeddings = embeddings * (1.0 - mask) + mask_tokens * mask

[rwightman]

While you could use padding/masking here, as you say, it's not likley it'd line up exactly on patch sizes. Esp for the 384x384 and larger for the few that have it, squash crop (distorting the aspect) actually works quite well and often better.