I have a lots of (image, text) pairs and would like to learn a feature extractor (or maybe a FeatureExtractorFunction) for a combined embedding.

It seems that when given a list of these multimodal pairs FeatureExtract doesn’t seem to do anything smart here in terms of trying to learn a joint semantic embedding:

FeatureExtract@Table[{RandomImage[1,{10,10},ColorSpace->”RGB"], ResourceFunction["RandomString"][20]}, 3000];

Instead of concatenating the BERT features of the text with the Resnet features of the image, I wanted to add an attention layer with transformers but this is easier for me in pytorch.

Perhaps I’m missing the obvious thing here...

  • $\begingroup$ What would be the point of adding an attention and fine-tune BERT+Resnet? To reduce diminesionality of concatenated embeddings? $\endgroup$ – swish Sep 25 at 6:20
  • $\begingroup$ @swish to train an embedding with better latent features $\endgroup$ – M.R. Sep 25 at 18:57
  • $\begingroup$ Why attention though? Just add one linear layer after concatenated embeddings and train all weights, including BERT's and Resnet's. $\endgroup$ – swish Sep 25 at 22:56
  • 1
    $\begingroup$ Checkout the papers on dynamic fusion $\endgroup$ – M.R. Sep 25 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.