Suppose we need to construct an unconventional loss function for my neural network: vector MSE but taking into account the "similarity" of variables.

My network takes data vectors ($m=40$ D) and returns result vectors ($n=20$ D). It works on a batch of $ K=2 \cdot 10 ^ 4$ examples. The proportion of certain components in the mixture is modeled. It is important that the mistake of overestimating one component in exchange for underestimating a component similar to it is valued as less harmful than changing the proportion of very different components.

Let MeanSquaredLossLayer[] (but constructed ab initio) be the starting point. Suppose the components in the resulting vector are ordered so that the most similar ones are adjacent. Actually, you only need to count the differences not only between the corresponding variables in the modeled vector and target $[\Delta f_i]_{i=1}^n=[f_i^m -f_i^t]_{i=1}^n$, and between all the variables $[ \Delta f_{ij} ]_{i,j=1}^n=[ f_i^m -f_j^t]_{i,j=1}^n$ and then multiply the table by the array of coefficients slightly differing from the identity matrix, e.g. $$\Bigg(\begin{smallmatrix} 1&0.2&0&0&0\\ 0.2&1&0.2&0&0\\ 0&0.2&1&0.2&0\\ 0&0&0.2&1&0.2\\ 0&0&0&0.2&1\\ \end{smallmatrix}\Bigg)$$ but $n \times n$ or slightly more complicated.

Then all you need to do is square it elementwise and add up all over the table. Such a result is a measure of the modeling error in a given case. Averaging over the entire batch gives the average error value.

The question is how to embed an array of numbers into the network. This is to be a fixed table, arbitrarily set in advance. The task seems simple, but in no way can I construct any modifiedLossLayer[] containing such an array. On the other hand, its introduction through an additional port NetPort["t"] makes it impossible to use such an error function in the learning process with a validation set (although learning itself is possible via a separate learning network as in the documentation: lossNet in ref/LossFunction).

If any of you knew the way and would like to share, I would be grateful. If this is a trivial problem, I am sorry, but seriously I can not find a solution even though I am not doing bad with NN.

  • $\begingroup$ In fact, I found a completely different way to achieve a similar effect when learning the network. It is sufficient to blur the boundaries in target vectors, e.g. by convolution with small gaussian. The question about the possibility of placing permanent arrays in the network, however, remains valid for other purposes. $\endgroup$ – Druid Mar 13 '20 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.