Plotting large scale image embeddings

Question

I'm looking for help making publication-quality graphics for visualizing 2D embeddings of image datasets, so that they look like this:

These two-dimensional embeddings of thumbnails (sometimes with magnifying zoom-in's - as shown above in red) are commonly seen plots in ML or computer vision papers, for instance:

• Learning visual similarity for product design with convolutional neural networks
• Multi-Modal Embedding for Main Product Detection in Fashion
• Learning Type-Aware Embeddings for Fashion Compatibility

This is typically done to show that a neural network learns a function that maps input images to a latent space such that the l2-norm in that space approximated the semantic distance in the input space.

After learning some latent embedding space and using DimensionReduce (with the TSNE method) you have one (x,y) point for each image, these are then otherwise masaged into a nice disk. I think this is how it's done:

1. Images are conformed (maybe aspect ratio preserving crops?)
2. The (x,y) points are snapped to some chosen disc-shaped grid (based on step 1)
3. Overlapping photos are somehow dealt with (randomly perhaps or keeping ones with similar entropy)
4. Larger versions of the images are shown for selected areas in rectangular magnifying-glass callouts

Here's another nice example:

Example data and code

To get you started, here's the typical input to this problem - a few thousand images and their embedding coordinates:

xy = CloudGet @ CloudObject[
   "https://www.wolframcloud.com/objects/user-6795023f-6362-45bf-9097-de07f37f89a1/embedding-example-xy"];
thumbs = CloudGet @ CloudObject[
   "https://www.wolframcloud.com/objects/user-6795023f-6362-45bf-9097-de07f37f89a1/embedding-example-thumbs"];
Length /@ {xy, thumbs}
MapThread[Labeled, {thumbs[[;; 5]], xy[[;; 5]]}]

AbsoluteTiming[
 data = MapThread[
   Inset[#1, #2, Automatic, Scaled[.02]] &, {thumbs, xy}];
 g = Graphics[data, ImageSize -> Full]]

So this doesn't look good yet, and if you zoom in you can see unwanted overlaps:

It requires the 4 steps above, e.g. to conform the images, massage the blob into a centered grid shape with minimal margins, and magnified with callouts and exported to PDF.

Possible solutions:

• Using WordCloud to pack them?
• Using ExternalFunction with some python libraries?

Answer 1

The first step in imitating the desired pictures is to plot everything on a grid. This can be done by rounding the positions to integer locations:

data = MapThread[Inset[#1, #2, Automatic, Scaled[.02]] &, {thumbs, 5 Round[xy]}];
g = Graphics[data, ImageSize -> 900]

Now you can pick out a subset of the images using something like:

sel = Select[data[[All]], 0 < #[[2, 1]] < 25 && 10 < #[[2, 2]] < 35 &];
Graphics[sel /. {0.02 -> 0.4}, ImageSize -> 500]