# Cosine similarity calculation

I am interested in calculating the cosine distance between each pair of the element of a sparse matrix. I am using the built-in function DistanceMatrix with the option CosineDistance.

My data is a sparse matrix sp with dimension ~{30000,6} and number of non zero vectors in sp is ~3000. I calculate the distance matrix in the following manner:

sp = SparseArray[
Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} ->
RandomReal[1], {3000}]]

DistanceMatrix[sp,
DistanceFunction -> CosineDistance]; // AbsoluteTiming


The calculation does not converge, any suggestion why the calculation does not converge.

I have a list of sparse matrices, and for each one of them, I want to calculate the cosine distance. I calculate the distance matrix with @Henrik solution. When I run the code outside Module (or Table as in the following example), I get the distance matrix with right dimensions, but when I run it inside Module, I get wrong dimensions of the distance matrix.

SeedRandom[123];
sp = matrices[[1]];
ilist = Flatten[
SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
B = SparseArray[{}, {1, 1} Length[sp], 0.];
B[[ilist, ilist]] =
SparseArray@
DistanceMatrix[sp[[ilist]],
DistanceFunction -> CosineDistance]; // AbsoluteTiming // First

Table[
sp = matrices[[i]];
ilist = Flatten[
SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
BB = SparseArray[{}, {1, 1} Length[sp], 0.];
BB[[ilist, ilist]] =
SparseArray@
DistanceMatrix[sp[[ilist]],
DistanceFunction -> CosineDistance], {i, 1}]

• Please provide all necessary code to reproduce your problem, especially if you're asking why a specific example does not behave as expected. Otherwise, it is very hard or even impossible to help you – Lukas Lang Jul 8 '18 at 9:41
• Please see an example, I added a sparse matrix. – Kiril Danilchenko Jul 8 '18 at 10:14
• Try this: Table[sp = matrices[[i]]; ilist = Flatten[ SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]]; BB = SparseArray[{}, {1, 1} Length[sp], 0.]; BB[[ilist, ilist]] = SparseArray@ DistanceMatrix[sp[[ilist]], DistanceFunction -> CosineDistance]; BB, {i, 1} ]. The return value should be BB. But the return value of BB[[ilist, ilist]] = ... is only the submatrix. – Henrik Schumacher Jul 8 '18 at 15:08

You can try the following. It computes the distance matrix of merely the nonzero rows.

SeedRandom[123];
sp = SparseArray[ Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} -> RandomReal[1], {3000}]];
ilist = Flatten[SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
A = SparseArray[{}, {1, 1} Length[sp], 0.];
A[[ilist, ilist]] = SparseArray@DistanceMatrix[
sp[[ilist]],
DistanceFunction -> CosineDistance
]; // AbsoluteTiming // First


0.384662

• Thank you @Henrik – Kiril Danilchenko Jul 8 '18 at 10:32
• You're welcome! – Henrik Schumacher Jul 8 '18 at 10:32
• I would need the code in order to say anything about it. – Henrik Schumacher Jul 8 '18 at 14:36
• I edited the question – Kiril Danilchenko Jul 8 '18 at 15:00