3
$\begingroup$

I am interested in calculating the cosine distance between each pair of the element of a sparse matrix. I am using the built-in function DistanceMatrix with the option CosineDistance.

My data is a sparse matrix sp with dimension ~{30000,6} and number of non zero vectors in sp is ~3000. I calculate the distance matrix in the following manner:

sp = SparseArray[
  Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} -> 
    RandomReal[1], {3000}]]

 DistanceMatrix[sp, 
    DistanceFunction -> CosineDistance]; // AbsoluteTiming

The calculation does not converge, any suggestion why the calculation does not converge.

I have a list of sparse matrices, and for each one of them, I want to calculate the cosine distance. I calculate the distance matrix with @Henrik solution. When I run the code outside Module (or Table as in the following example), I get the distance matrix with right dimensions, but when I run it inside Module, I get wrong dimensions of the distance matrix.

SeedRandom[123];
sp = matrices[[1]];
ilist = Flatten[
   SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
B = SparseArray[{}, {1, 1} Length[sp], 0.];
B[[ilist, ilist]] = 
    SparseArray@
     DistanceMatrix[sp[[ilist]], 
      DistanceFunction -> CosineDistance]; // AbsoluteTiming // First

Table[
 sp = matrices[[i]];
 ilist = Flatten[
   SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
 BB = SparseArray[{}, {1, 1} Length[sp], 0.];
 BB[[ilist, ilist]] = 
  SparseArray@
   DistanceMatrix[sp[[ilist]], 
    DistanceFunction -> CosineDistance], {i, 1}]
$\endgroup$
  • 2
    $\begingroup$ Please provide all necessary code to reproduce your problem, especially if you're asking why a specific example does not behave as expected. Otherwise, it is very hard or even impossible to help you $\endgroup$ – Lukas Lang Jul 8 '18 at 9:41
  • $\begingroup$ Please see an example, I added a sparse matrix. $\endgroup$ – Kiril Danilchenko Jul 8 '18 at 10:14
  • 1
    $\begingroup$ Try this: Table[sp = matrices[[i]]; ilist = Flatten[ SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]]; BB = SparseArray[{}, {1, 1} Length[sp], 0.]; BB[[ilist, ilist]] = SparseArray@ DistanceMatrix[sp[[ilist]], DistanceFunction -> CosineDistance]; BB, {i, 1} ]. The return value should be BB. But the return value of BB[[ilist, ilist]] = ... is only the submatrix. $\endgroup$ – Henrik Schumacher Jul 8 '18 at 15:08
3
$\begingroup$

You can try the following. It computes the distance matrix of merely the nonzero rows.

SeedRandom[123];
sp = SparseArray[ Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} -> RandomReal[1], {3000}]];
ilist = Flatten[SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
A = SparseArray[{}, {1, 1} Length[sp], 0.];
A[[ilist, ilist]] = SparseArray@DistanceMatrix[
 sp[[ilist]], 
 DistanceFunction -> CosineDistance
 ]; // AbsoluteTiming // First

0.384662

$\endgroup$
  • $\begingroup$ Thank you @Henrik $\endgroup$ – Kiril Danilchenko Jul 8 '18 at 10:32
  • $\begingroup$ You're welcome! $\endgroup$ – Henrik Schumacher Jul 8 '18 at 10:32
  • 1
    $\begingroup$ I would need the code in order to say anything about it. $\endgroup$ – Henrik Schumacher Jul 8 '18 at 14:36
  • $\begingroup$ I edited the question $\endgroup$ – Kiril Danilchenko Jul 8 '18 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.