0
$\begingroup$

I want to factorize a matrix $a_{ij}$ as $$ a_{ij} = \sum_k u_{ik} v_{jk} $$ using variational method for some reason. That is, I want to minimize the cost function $$ F = \sum_{ij} b_{ij}^2 $$ with respect to $u_{ik}$ and $v_{jk}$ where $$ b_{ij} = \sum_k u_{ik} v_{jk} - a_{ij} $$ I use gradient descent to find the minimum, therefore I need to calculate the derivative of $F$ with respect to $u_{ik}$ and $v_{jk}$. These can be calculated very easily with hand for example $$ \frac{\partial F}{\partial u_{ik}} = 2 \sum_j b_{ij} v_{jk} $$ However, I want to see how Mathematica solves this quite typical problem so that I don't need to use hand to calculate a much more complex version of the problem.

The code I wrote is

KD[x_, y_] := KroneckerDelta[x, y]
u /: D[u[a_, b_], u[c_, d_], NonConstants -> {u}] := KD[a, c] KD[b, d]
v /: D[v[a_, b_], v[c_, d_], NonConstants -> {v}] := KD[a, c] KD[b, d]
b[i_, j_] := Sum[u[i, k] v[j, k], {k, Infinity}] - a[i, j]
F = Sum[b[i, j]^2, {i, Infinity}, {j, Infinity}];
$Assumptions = {i, j, k, i1, k1} \[Element] Integers && i > 0 && j > 0 && k > 0 && i1 > 0 && k1 > 0;
D[F, u[i1, k1], NonConstants -> {u}]

after a few minutes I got the result $$ \sum _i^{\infty } \sum _j^{\infty } 2 \left( \begin{array}{ll} \{ & \begin{array}{ll} v[j,\text{k1}] & i==\text{i1} \\ 0 & \text{True} \\ \end{array} \\ \end{array} \right) \left(-a[i,j]+\sum _k^{\infty } u[i,k] v[j,k]\right) $$

Is there an elegant way to evaluate the simple derivative $\frac{\partial F}{\partial u_{ik}}$ with Mathematica?

|improve this question
$\endgroup$
  • 1
    $\begingroup$ One such factorization would have the u matrix be the identity and the v matrix be the one being factored. I mention this to point out that you really need to state what it is you specifically you want to achieve in your factorization. Without that information there's not much that can be said. $\endgroup$ – Daniel Lichtblau Jun 26 '14 at 15:22
  • $\begingroup$ I want to factorize the matrix $a$ as the product of two other matrix $u$ and $v$. If I put orthonormal and rank constrain on $u$ and $v$, then it is singular value decomposition. Anyway, my question is not how to factorize the matrix but how to calculate the simple derivative. Hope this clarified my question. $\endgroup$ – entron Jun 26 '14 at 16:36
  • $\begingroup$ The result you cite is not wrong, it's just not in the form you want. What is the practical use case where this result doesn't agree with what you want to achieve? In particular, why do you need infinity as the upper limit if your goal is an actual matrix factorization (since matrices have a finite rank)? I'm asking mainly to understand if this is worth the trouble... $\endgroup$ – Jens Jun 26 '14 at 20:40
  • $\begingroup$ @Jens, I will factorize $a$ numerically with python. The actual cost function is more complex, but the principle should be the same. The result generated by Mathematica is not reduced to the simplest form(the delta function hasn't been sumed away). To use the analytical form of the gradient calculated by Mathematica in external program, I want the derivatives are of their simplest form so that I don't waste any resource. Regarding the upper limit, I use infinity because the finial form of the derivatives don't depend on the upper limit and infinity makes the syntax a bit simpler. $\endgroup$ – entron Jun 26 '14 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.