4
$\begingroup$

I have a lots of (image, text) pairs and would like to learn a feature extractor (or maybe a FeatureExtractorFunction) for a combined embedding.

It seems that when given a list of these multimodal pairs FeatureExtract doesn’t seem to do anything smart here in terms of trying to learn a joint semantic embedding:

FeatureExtract@Table[{RandomImage[1,{10,10},ColorSpace->”RGB"], ResourceFunction["RandomString"][20]}, 3000];

Instead of concatenating the BERT features of the text with the Resnet features of the image, I wanted to add an attention layer with transformers but this is easier for me in pytorch.

Perhaps I’m missing the obvious thing here...

Improve this question
$\endgroup$
4
  • $\begingroup$ What would be the point of adding an attention and fine-tune BERT+Resnet? To reduce diminesionality of concatenated embeddings? $\endgroup$
    – swish
    Commented Sep 25, 2019 at 6:20
  • $\begingroup$ @swish to train an embedding with better latent features $\endgroup$
    – M.R.
    Commented Sep 25, 2019 at 18:57
  • $\begingroup$ Why attention though? Just add one linear layer after concatenated embeddings and train all weights, including BERT's and Resnet's. $\endgroup$
    – swish
    Commented Sep 25, 2019 at 22:56
  • 1
    $\begingroup$ Checkout the papers on dynamic fusion $\endgroup$
    – M.R.
    Commented Sep 25, 2019 at 23:47

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.