# Cosine similarity calculation

I am interested in calculating the cosine distance between each pair of the element of a sparse matrix. I am using the built-in function DistanceMatrix with the option CosineDistance.

My data is a sparse matrix sp with dimension ~{30000,6} and number of non zero vectors in sp is ~3000. I calculate the distance matrix in the following manner:

sp = SparseArray[
Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} ->
RandomReal, {3000}]]

DistanceMatrix[sp,
DistanceFunction -> CosineDistance]; // AbsoluteTiming


The calculation does not converge, any suggestion why the calculation does not converge.

I have a list of sparse matrices, and for each one of them, I want to calculate the cosine distance. I calculate the distance matrix with @Henrik solution. When I run the code outside Module (or Table as in the following example), I get the distance matrix with right dimensions, but when I run it inside Module, I get wrong dimensions of the distance matrix.

SeedRandom;
sp = matrices[];
ilist = Flatten[
SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
B = SparseArray[{}, {1, 1} Length[sp], 0.];
B[[ilist, ilist]] =
SparseArray@
DistanceMatrix[sp[[ilist]],
DistanceFunction -> CosineDistance]; // AbsoluteTiming // First

Table[
sp = matrices[[i]];
ilist = Flatten[
SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
BB = SparseArray[{}, {1, 1} Length[sp], 0.];
BB[[ilist, ilist]] =
SparseArray@
DistanceMatrix[sp[[ilist]],
DistanceFunction -> CosineDistance], {i, 1}]

• Please provide all necessary code to reproduce your problem, especially if you're asking why a specific example does not behave as expected. Otherwise, it is very hard or even impossible to help you Jul 8, 2018 at 9:41
• Please see an example, I added a sparse matrix. Jul 8, 2018 at 10:14
• Try this: Table[sp = matrices[[i]]; ilist = Flatten[ SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]]; BB = SparseArray[{}, {1, 1} Length[sp], 0.]; BB[[ilist, ilist]] = SparseArray@ DistanceMatrix[sp[[ilist]], DistanceFunction -> CosineDistance]; BB, {i, 1} ]. The return value should be BB. But the return value of BB[[ilist, ilist]] = ... is only the submatrix. Jul 8, 2018 at 15:08

You can try the following. It computes the distance matrix of merely the nonzero rows.

SeedRandom;
sp = SparseArray[ Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} -> RandomReal, {3000}]];
ilist = Flatten[SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
A = SparseArray[{}, {1, 1} Length[sp], 0.];
A[[ilist, ilist]] = SparseArray@DistanceMatrix[
sp[[ilist]],
DistanceFunction -> CosineDistance
]; // AbsoluteTiming // First


0.384662

• Thank you @Henrik Jul 8, 2018 at 10:32
• You're welcome! Jul 8, 2018 at 10:32
• I would need the code in order to say anything about it. Jul 8, 2018 at 14:36
• I edited the question Jul 8, 2018 at 15:00