# How to get features for captioned images?

I have a lots of (image, text) pairs and would like to learn a feature extractor (or maybe a FeatureExtractorFunction) for a combined embedding.

It seems that when given a list of these multimodal pairs FeatureExtract doesn’t seem to do anything smart here in terms of trying to learn a joint semantic embedding:

FeatureExtract@Table[{RandomImage[1,{10,10},ColorSpace->”RGB"], ResourceFunction["RandomString"][20]}, 3000];


Instead of concatenating the BERT features of the text with the Resnet features of the image, I wanted to add an attention layer with transformers but this is easier for me in pytorch.

Perhaps I’m missing the obvious thing here...

• What would be the point of adding an attention and fine-tune BERT+Resnet? To reduce diminesionality of concatenated embeddings? – swish Sep 25 at 6:20
• @swish to train an embedding with better latent features – M.R. Sep 25 at 18:57
• Why attention though? Just add one linear layer after concatenated embeddings and train all weights, including BERT's and Resnet's. – swish Sep 25 at 22:56
• Checkout the papers on dynamic fusion – M.R. Sep 25 at 23:47