4
$\begingroup$

I am interested in calculating the cosine distance between each pair of the element of a sparse matrix. I am using the built-in function DistanceMatrix with the option CosineDistance.

My data is a sparse matrix sp with dimension ~{30000,6} and number of non zero vectors in sp is ~3000. I calculate the distance matrix in the following manner:

sp = SparseArray[
  Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} -> 
    RandomReal[1], {3000}]]

 DistanceMatrix[sp, 
    DistanceFunction -> CosineDistance]; // AbsoluteTiming

The calculation does not converge, any suggestion why the calculation does not converge.

I have a list of sparse matrices, and for each one of them, I want to calculate the cosine distance. I calculate the distance matrix with @Henrik solution. When I run the code outside Module (or Table as in the following example), I get the distance matrix with right dimensions, but when I run it inside Module, I get wrong dimensions of the distance matrix. 

SeedRandom[123];
sp = matrices[[1]];
ilist = Flatten[
   SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
B = SparseArray[{}, {1, 1} Length[sp], 0.];
B[[ilist, ilist]] = 
    SparseArray@
     DistanceMatrix[sp[[ilist]], 
      DistanceFunction -> CosineDistance]; // AbsoluteTiming // First

Table[
 sp = matrices[[i]];
 ilist = Flatten[
   SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
 BB = SparseArray[{}, {1, 1} Length[sp], 0.];
 BB[[ilist, ilist]] = 
  SparseArray@
   DistanceMatrix[sp[[ilist]], 
    DistanceFunction -> CosineDistance], {i, 1}]
Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Please provide all necessary code to reproduce your problem, especially if you're asking why a specific example does not behave as expected. Otherwise, it is very hard or even impossible to help you $\endgroup$
    – Lukas Lang
    Jul 8, 2018 at 9:41
  • $\begingroup$ Please see an example, I added a sparse matrix. $\endgroup$
    – Kiril Danilchenko
    Jul 8, 2018 at 10:14
  • 1
    $\begingroup$ Try this: Table[sp = matrices[[i]]; ilist = Flatten[ SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]]; BB = SparseArray[{}, {1, 1} Length[sp], 0.]; BB[[ilist, ilist]] = SparseArray@ DistanceMatrix[sp[[ilist]], DistanceFunction -> CosineDistance]; BB, {i, 1} ]. The return value should be BB. But the return value of BB[[ilist, ilist]] = ... is only the submatrix. $\endgroup$
    – Henrik Schumacher
    Jul 8, 2018 at 15:08

1 Answer 1

Reset to default
3
$\begingroup$

You can try the following. It computes the distance matrix of merely the nonzero rows.

SeedRandom[123];
sp = SparseArray[ Table[{RandomInteger[{1, 30000}], RandomInteger[{1, 6}]} -> RandomReal[1], {3000}]];
ilist = Flatten[SparseArray[Unitize[Total[Abs[sp], {2}]]]["NonzeroPositions"]];
A = SparseArray[{}, {1, 1} Length[sp], 0.];
A[[ilist, ilist]] = SparseArray@DistanceMatrix[
 sp[[ilist]], 
 DistanceFunction -> CosineDistance
 ]; // AbsoluteTiming // First

0.384662

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you @Henrik $\endgroup$
    – Kiril Danilchenko
    Jul 8, 2018 at 10:32
  • $\begingroup$ You're welcome! $\endgroup$
    – Henrik Schumacher
    Jul 8, 2018 at 10:32
  • 1
    $\begingroup$ I would need the code in order to say anything about it. $\endgroup$
    – Henrik Schumacher
    Jul 8, 2018 at 14:36
  • $\begingroup$ I edited the question $\endgroup$
    – Kiril Danilchenko
    Jul 8, 2018 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.