# Good way to create a similarity / distance matrix for a large dataset [duplicate]

I have a large dataset with rows (100k+) for products and columns for features.

Now I want to create a similarity matrix with NormalizedSquaredEuclideanDistance. The desired output would be a symmetric matrix with products as columns and rows and the similarity measures as entries.

For[p = 1, p <= Length[dataset[[All,1]]], p++,
For[n = 1, n <= Length[dataset[[All,1]]], n++,
SimMat[[p, n]] =
NormalizedSquaredEuclideanDistance[
dataset[[n, 2 ;; Length[dataset[[n]]]]],
dataset[[p, 2 ;; Length[dataset[[p]]]]]]]

There are some problems:

1. NormalizedSquaredEuclideanDistance does not work with how I called the rows.

2. Using two For-loops for such a big dataset seems not very efficient,

• I am finding it hard to understand this question. Can you give us a small example of input along with the output you expect from this example? By small, I mean an input matrix of dimension, say, 4 x 4. May 1, 2016 at 12:51
• Is your dataset / product-feature matrix sparse? For example, if you have 100 feature-columns, for a given product $p$ do all feature-columns have associating values with $p$? May 1, 2016 at 17:09
• It is sparse, as many columns are dummies for categorical features. May 3, 2016 at 5:21
• Possible duplicate: (21861) May 6, 2016 at 2:19

If you have Mathematica 10.3 or above you can use DistanceMatrix:

DistanceMatrix[dataset2, DistanceFunction -> NormalizedSquaredEuclideanDistance]

I'm assuming the same data as defined by kglr, you have not given us an example. If you don't have Mathematica 10.3 there's still HierarchicalClustering`DistanceMatrix which is used in the same way.

• I up-voted this answer because of the use of DistanceMatrix, but DistanceMatrix can be much slower than Outer for a list of sparse vectors. May 1, 2016 at 21:36
• @AntonAntonov. Thanks. Do you know why that is the case? May 1, 2016 at 21:39
• Sorry, no. I just did some benchmarking with Outer and DistanceMatrix. May 1, 2016 at 22:59
• @AntonAntonov but DistanceMatrix is orders of magnitude faster than Outer for normal matrices. May 3, 2016 at 6:25
• @SjoerdC.deVries May be it is a good idea to have an answer with benchmarking. May 3, 2016 at 10:02
dataset2 = RandomReal[1, {5, 7}]; (* this stands for dataset[[All,2;;]] in your case*)

dataset2 // MatrixForm

output = Outer[NormalizedSquaredEuclideanDistance, dataset2, dataset2, 1];
output // MatrixForm