# Good way to create a similarity / distance matrix for a large dataset [duplicate]

I have a large dataset with rows (100k+) for products and columns for features.

Now I want to create a similarity matrix with NormalizedSquaredEuclideanDistance. The desired output would be a symmetric matrix with products as columns and rows and the similarity measures as entries.

For[p = 1, p <= Length[dataset[[All,1]]], p++,
For[n = 1, n <= Length[dataset[[All,1]]], n++,
SimMat[[p, n]] =
NormalizedSquaredEuclideanDistance[
dataset[[n, 2 ;; Length[dataset[[n]]]]],
dataset[[p, 2 ;; Length[dataset[[p]]]]]]]


There are some problems:

1. NormalizedSquaredEuclideanDistance does not work with how I called the rows.

2. Using two For-loops for such a big dataset seems not very efficient,

• I am finding it hard to understand this question. Can you give us a small example of input along with the output you expect from this example? By small, I mean an input matrix of dimension, say, 4 x 4. May 1, 2016 at 12:51
• Is your dataset / product-feature matrix sparse? For example, if you have 100 feature-columns, for a given product $p$ do all feature-columns have associating values with $p$? May 1, 2016 at 17:09
• It is sparse, as many columns are dummies for categorical features. May 3, 2016 at 5:21
• Possible duplicate: (21861) May 6, 2016 at 2:19

If you have Mathematica 10.3 or above you can use DistanceMatrix:

DistanceMatrix[dataset2, DistanceFunction -> NormalizedSquaredEuclideanDistance]


I'm assuming the same data as defined by kglr, you have not given us an example. If you don't have Mathematica 10.3 there's still HierarchicalClusteringDistanceMatrix which is used in the same way.

• I up-voted this answer because of the use of DistanceMatrix, but DistanceMatrix can be much slower than Outer for a list of sparse vectors. May 1, 2016 at 21:36
• @AntonAntonov. Thanks. Do you know why that is the case? May 1, 2016 at 21:39
• Sorry, no. I just did some benchmarking with Outer and DistanceMatrix. May 1, 2016 at 22:59
• @AntonAntonov but DistanceMatrix is orders of magnitude faster than Outer for normal matrices. May 3, 2016 at 6:25
• @SjoerdC.deVries May be it is a good idea to have an answer with benchmarking. May 3, 2016 at 10:02
dataset2 = RandomReal[1, {5, 7}]; (* this stands for dataset[[All,2;;]] in your case*)

dataset2 // MatrixForm output = Outer[NormalizedSquaredEuclideanDistance, dataset2, dataset2, 1];
output // MatrixForm
` 