6
$\begingroup$

I have a large dataset with rows (100k+) for products and columns for features.

Now I want to create a similarity matrix with NormalizedSquaredEuclideanDistance. The desired output would be a symmetric matrix with products as columns and rows and the similarity measures as entries. 

For[p = 1, p <= Length[dataset[[All,1]]], p++, 
  For[n = 1, n <= Length[dataset[[All,1]]], n++,
    SimMat[[p, n]] = 
      NormalizedSquaredEuclideanDistance[
        dataset[[n, 2 ;; Length[dataset[[n]]]]], 
        dataset[[p, 2 ;; Length[dataset[[p]]]]]]]

There are some problems:

  1. NormalizedSquaredEuclideanDistance does not work with how I called the rows.

  2. Using two For-loops for such a big dataset seems not very efficient,

Improve this question
$\endgroup$
4
  • $\begingroup$ I am finding it hard to understand this question. Can you give us a small example of input along with the output you expect from this example? By small, I mean an input matrix of dimension, say, 4 x 4. $\endgroup$
    – m_goldberg
    May 1, 2016 at 12:51
  • $\begingroup$ Is your dataset / product-feature matrix sparse? For example, if you have 100 feature-columns, for a given product $p$ do all feature-columns have associating values with $p$? $\endgroup$
    – Anton Antonov
    May 1, 2016 at 17:09
  • $\begingroup$ It is sparse, as many columns are dummies for categorical features. $\endgroup$
    – Zappageck
    May 3, 2016 at 5:21
  • 5
    $\begingroup$ Possible duplicate: (21861) $\endgroup$
    – Mr.Wizard
    May 6, 2016 at 2:19

2 Answers 2

Reset to default
9
$\begingroup$

If you have Mathematica 10.3 or above you can use DistanceMatrix:

DistanceMatrix[dataset2, DistanceFunction -> NormalizedSquaredEuclideanDistance]

I'm assuming the same data as defined by kglr, you have not given us an example. If you don't have Mathematica 10.3 there's still HierarchicalClustering`DistanceMatrix which is used in the same way.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I up-voted this answer because of the use of DistanceMatrix, but DistanceMatrix can be much slower than Outer for a list of sparse vectors. $\endgroup$
    – Anton Antonov
    May 1, 2016 at 21:36
  • $\begingroup$ @AntonAntonov. Thanks. Do you know why that is the case? $\endgroup$
    – RunnyKine
    May 1, 2016 at 21:39
  • $\begingroup$ Sorry, no. I just did some benchmarking with Outer and DistanceMatrix. $\endgroup$
    – Anton Antonov
    May 1, 2016 at 22:59
  • $\begingroup$ @AntonAntonov but DistanceMatrix is orders of magnitude faster than Outer for normal matrices. $\endgroup$
    – Sjoerd C. de Vries
    May 3, 2016 at 6:25
  • $\begingroup$ @SjoerdC.deVries May be it is a good idea to have an answer with benchmarking. $\endgroup$
    – Anton Antonov
    May 3, 2016 at 10:02
5
$\begingroup$ 
dataset2 = RandomReal[1, {5, 7}]; (* this stands for dataset[[All,2;;]] in your case*)

dataset2 // MatrixForm

Mathematica graphics

output = Outer[NormalizedSquaredEuclideanDistance, dataset2, dataset2, 1];
output // MatrixForm

Mathematica graphics

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.