I have a large dataset with rows (100k+) for products and columns for features.

Now I want to create a similarity matrix with NormalizedSquaredEuclideanDistance. The desired output would be a symmetric matrix with products as columns and rows and the similarity measures as entries.

For[p = 1, p <= Length[dataset[[All,1]]], p++, 
  For[n = 1, n <= Length[dataset[[All,1]]], n++,
    SimMat[[p, n]] = 
        dataset[[n, 2 ;; Length[dataset[[n]]]]], 
        dataset[[p, 2 ;; Length[dataset[[p]]]]]]]

There are some problems:

  1. NormalizedSquaredEuclideanDistance does not work with how I called the rows.

  2. Using two For-loops for such a big dataset seems not very efficient,

  • $\begingroup$ I am finding it hard to understand this question. Can you give us a small example of input along with the output you expect from this example? By small, I mean an input matrix of dimension, say, 4 x 4. $\endgroup$ – m_goldberg May 1 '16 at 12:51
  • $\begingroup$ Is your dataset / product-feature matrix sparse? For example, if you have 100 feature-columns, for a given product $p$ do all feature-columns have associating values with $p$? $\endgroup$ – Anton Antonov May 1 '16 at 17:09
  • $\begingroup$ It is sparse, as many columns are dummies for categorical features. $\endgroup$ – Zappageck May 3 '16 at 5:21
  • 5
    $\begingroup$ Possible duplicate: (21861) $\endgroup$ – Mr.Wizard May 6 '16 at 2:19

If you have Mathematica 10.3 or above you can use DistanceMatrix:

DistanceMatrix[dataset2, DistanceFunction -> NormalizedSquaredEuclideanDistance]

I'm assuming the same data as defined by kglr, you have not given us an example. If you don't have Mathematica 10.3 there's still HierarchicalClustering`DistanceMatrix which is used in the same way.

| improve this answer | |
  • $\begingroup$ I up-voted this answer because of the use of DistanceMatrix, but DistanceMatrix can be much slower than Outer for a list of sparse vectors. $\endgroup$ – Anton Antonov May 1 '16 at 21:36
  • $\begingroup$ @AntonAntonov. Thanks. Do you know why that is the case? $\endgroup$ – RunnyKine May 1 '16 at 21:39
  • $\begingroup$ Sorry, no. I just did some benchmarking with Outer and DistanceMatrix. $\endgroup$ – Anton Antonov May 1 '16 at 22:59
  • $\begingroup$ @AntonAntonov but DistanceMatrix is orders of magnitude faster than Outer for normal matrices. $\endgroup$ – Sjoerd C. de Vries May 3 '16 at 6:25
  • $\begingroup$ @SjoerdC.deVries May be it is a good idea to have an answer with benchmarking. $\endgroup$ – Anton Antonov May 3 '16 at 10:02
dataset2 = RandomReal[1, {5, 7}]; (* this stands for dataset[[All,2;;]] in your case*)

dataset2 // MatrixForm

Mathematica graphics

output = Outer[NormalizedSquaredEuclideanDistance, dataset2, dataset2, 1];
output // MatrixForm

Mathematica graphics

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.