2
$\begingroup$

I'm looking for help making publication-quality graphics for visualizing 2D embeddings of image datasets, so that they look like this:

enter image description here

These two-dimensional embeddings of thumbnails (sometimes with magnifying zoom-in's - as shown above in red) are commonly seen plots in ML or computer vision papers, for instance:

This is typically done to show that a neural network learns a function that maps input images to a latent space such that the l2-norm in that space approximated the semantic distance in the input space.

After learning some latent embedding space and using DimensionReduce (with the TSNE method) you have one (x,y) point for each image, these are then otherwise masaged into a nice disk. I think this is how it's done:

  1. Images are conformed (maybe aspect ratio preserving crops?)
  2. The (x,y) points are snapped to some chosen disc-shaped grid (based on step 1)
  3. Overlapping photos are somehow dealt with (randomly perhaps or keeping ones with similar entropy)
  4. Larger versions of the images are shown for selected areas in rectangular magnifying-glass callouts

Here's another nice example:

enter image description here

Example data and code

To get you started, here's the typical input to this problem - a few thousand images and their embedding coordinates:

xy = CloudGet @ CloudObject[
   "https://www.wolframcloud.com/objects/user-6795023f-6362-45bf-9097-de07f37f89a1/embedding-example-xy"];
thumbs = CloudGet @ CloudObject[
   "https://www.wolframcloud.com/objects/user-6795023f-6362-45bf-9097-de07f37f89a1/embedding-example-thumbs"];
Length /@ {xy, thumbs}
MapThread[Labeled, {thumbs[[;; 5]], xy[[;; 5]]}]

enter image description here

AbsoluteTiming[
 data = MapThread[
   Inset[#1, #2, Automatic, Scaled[.02]] &, {thumbs, xy}];
 g = Graphics[data, ImageSize -> Full]]

enter image description here

So this doesn't look good yet, and if you zoom in you can see unwanted overlaps:

enter image description here

It requires the 4 steps above, e.g. to conform the images, massage the blob into a centered grid shape with minimal margins, and magnified with callouts and exported to PDF.

Possible solutions:

  • Using WordCloud to pack them?
  • Using ExternalFunction with some python libraries?
$\endgroup$
  • $\begingroup$ In your desired picture, it looks like all the little pictures have exactly the same dimensions, and so are easy to pack regularly (that's why 25 pictures fit in every red box). In your example data set, the pictures have many different sizes and so cannot be packed regularly. $\endgroup$ – bill s May 30 at 18:26
  • $\begingroup$ Right, would be nice to handle the general case, but please feel free to center crop/pad them if you want to simplify it. $\endgroup$ – M.R. May 30 at 18:57
  • $\begingroup$ @bills I think you are right, I updated the question to reflect this $\endgroup$ – M.R. May 30 at 20:49
1
$\begingroup$

The first step in imitating the desired pictures is to plot everything on a grid. This can be done by rounding the positions to integer locations:

data = MapThread[Inset[#1, #2, Automatic, Scaled[.02]] &, {thumbs, 5 Round[xy]}];
g = Graphics[data, ImageSize -> 900]

enter image description here

Now you can pick out a subset of the images using something like:

sel = Select[data[[All]], 0 < #[[2, 1]] < 25 && 10 < #[[2, 2]] < 35 &];
Graphics[sel /. {0.02 -> 0.4}, ImageSize -> 500]

enter image description here

$\endgroup$
  • $\begingroup$ You picked 5 by hand? It might be better if somehow the points were slightly rearranged and compressed into a disc though... I think the overlaying callout magnifications are the tricky part $\endgroup$ – M.R. May 30 at 22:53
  • $\begingroup$ I just picked it so that it seemed to look OK. You can apply any function you wish instead of the 5*Round. $\endgroup$ – bill s May 31 at 0:30
  • $\begingroup$ if you can put more work into your solution, compressing the space and adding the overlays I’ll accept $\endgroup$ – M.R. Jun 3 at 2:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.