# Good way to create a similarity / distance matrix for a large dataset [duplicate]

I have a large dataset with rows (100k+) for products and columns for features.

Now I want to create a similarity matrix with NormalizedSquaredEuclideanDistance. The desired output would be a symmetric matrix with products as columns and rows and the similarity measures as entries.

For[p = 1, p <= Length[dataset[[All,1]]], p++,
For[n = 1, n <= Length[dataset[[All,1]]], n++,
SimMat[[p, n]] =
NormalizedSquaredEuclideanDistance[
dataset[[n, 2 ;; Length[dataset[[n]]]]],
dataset[[p, 2 ;; Length[dataset[[p]]]]]]]


There are some problems:

1. NormalizedSquaredEuclideanDistance does not work with how I called the rows.

2. Using two For-loops for such a big dataset seems not very efficient,

• I am finding it hard to understand this question. Can you give us a small example of input along with the output you expect from this example? By small, I mean an input matrix of dimension, say, 4 x 4. – m_goldberg May 1 '16 at 12:51
• Is your dataset / product-feature matrix sparse? For example, if you have 100 feature-columns, for a given product $p$ do all feature-columns have associating values with $p$? – Anton Antonov May 1 '16 at 17:09
• It is sparse, as many columns are dummies for categorical features. – Zappageck May 3 '16 at 5:21
• Possible duplicate: (21861) – Mr.Wizard May 6 '16 at 2:19

If you have Mathematica 10.3 or above you can use DistanceMatrix:

DistanceMatrix[dataset2, DistanceFunction -> NormalizedSquaredEuclideanDistance]


I'm assuming the same data as defined by kglr, you have not given us an example. If you don't have Mathematica 10.3 there's still HierarchicalClusteringDistanceMatrix which is used in the same way.

• I up-voted this answer because of the use of DistanceMatrix, but DistanceMatrix can be much slower than Outer for a list of sparse vectors. – Anton Antonov May 1 '16 at 21:36
• @AntonAntonov. Thanks. Do you know why that is the case? – RunnyKine May 1 '16 at 21:39
• Sorry, no. I just did some benchmarking with Outer and DistanceMatrix. – Anton Antonov May 1 '16 at 22:59
• @AntonAntonov but DistanceMatrix is orders of magnitude faster than Outer for normal matrices. – Sjoerd C. de Vries May 3 '16 at 6:25
• @SjoerdC.deVries May be it is a good idea to have an answer with benchmarking. – Anton Antonov May 3 '16 at 10:02
dataset2 = RandomReal[1, {5, 7}]; (* this stands for dataset[[All,2;;]] in your case*)

dataset2 // MatrixForm


output = Outer[NormalizedSquaredEuclideanDistance, dataset2, dataset2, 1];
output // MatrixForm
`