7
$\begingroup$

I am developing a weighted KNN algorithm. In a step, I need to do the sum of weights of each class. For example:

weightsPerclass = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

Here the first row contains the weights and the second row contains the class labels.

I need to do the sum for each class and I return

{{1, 2, 3, 5}, {2.6, .2, .6, 0}}

We can use a loop but I need a more compact solution. Any suggestion?

$\endgroup$
5
$\begingroup$

Many ways to do this.

w = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};
class = Union[w[[2]]];
Map[Total, Cases[Transpose[w], {x_, y_} /; y == # :> x] & /@ class];
{class, %}

Mathematica graphics

$\endgroup$
  • 1
    $\begingroup$ FYI: There is a performance problem with this approach. The entire list must be rescanned for every element in class which will cause considerable slow-down as the length of that list increases. This is the problem I was referring to in (25591). $\endgroup$ – Mr.Wizard Oct 9 '14 at 12:50
9
$\begingroup$

Additional methods using WeightedData, EmpiricalDistribution, GatherBy and SparseArray:

{weights,classes} = weightsPerclass = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

WeightedData

wd = WeightedData[classes, weights];

The property "EmpiricalPDF" almost gives what we need

wd["EmpiricalPDF"]
(* {{1,2,3,5},{0.8125,0.0625,0.125,0.}} *)

except that it has to be de-normalized by Total[weights]:

{First@# , Total[weights] Last@#} &@wd["EmpiricalPDF"]
(* {{1,2,3,5},{2.6,0.2,0.4,0.}} *)

Defining a function you can get the result in one step:

wdF = With[{p = WeightedData[#2, #1]["EmpiricalPDF"], w = Total @ #}, {1, w} p] &;

wdF[weights, classes]
(* {{1,2,3,5},{2.6,0.2,0.4,0.}} *)

EmpiricalDistribution

ed = EmpiricalDistribution[Rule @@ weightsPerclass];
ed["Domain"]
(* {1,2,3,5} *)

Total[weights] ed["Weights"]
(* {2.6,0.2,0.4,0.} *)

{%%, %}
(* {{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0.}} *)

You can also define a function that returns the desired output in one step:

ClearAll[wedF]
wedF = With[{d = EmpiricalDistribution[# -> #2], w = Total@#}, {d["Domain"], w d["Weights"]}] &;

wedF[weights, classes]
(* {{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0.}} *)

GatherBy

ClearAll[gthrBy];
gthrBy = Transpose[{Last@First@#, Total[First /@ #]} & /@ GatherBy[Transpose@#, Last]] &;

gthrBy@weightsPerclass
(* {{1,2,3,5},{2.6,0.2,0.4,0}} *)

SparseArray

System`SetSystemOptions["SparseArrayOptions" -> {"TreatRepeatedEntries" ->Total}];
sa = SparseArray[classes -> weights];
System`SetSystemOptions["SparseArrayOptions" -> {"TreatRepeatedEntries" -> First}];
{Flatten[sa["NonzeroPositions"]], sa["NonzeroValues"]}
(* {{1,2,3,5},{2.6,0.2,0.4,0}} *)

See also: Optimizing 2D binning code and Fast 2D binning algorithm

GroupBy - Version 10

Through[{Keys,Values}[GroupBy[Transpose@weightsPerclass,Last -> First, Total]]]
 (* {{1,2,3,5}, {2.6,0.2,0.4,0}} *)
$\endgroup$
  • $\begingroup$ You're a madman. $\endgroup$ – kale Oct 9 '14 at 13:38
  • $\begingroup$ @kguler, thanks $\endgroup$ – BetterEnglish Oct 9 '14 at 14:51
  • $\begingroup$ @kguler Why the empirical PDF does not give directly the best resuls. Why we have to multiply the results by the Total[weights]. Could you explain this point please? $\endgroup$ – BetterEnglish Oct 9 '14 at 15:42
  • 1
    $\begingroup$ @Developer2000, PDF values are normalized so that they add up to 1 over the domain; i.e. pdf[i] is totalWeight [i] / totalWeight[j in Domain]. To get the raw weights we need to multiply pdf values by the total weight of all values in the domain. $\endgroup$ – kglr Oct 9 '14 at 16:12
5
$\begingroup$

If you have Association functionality there is a rather nice approach using Merge:

w = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

Thread[#2 -> # & @@ w] ~Merge~ Total
<|1 -> 2.6, 2 -> 0.2, 3 -> 0.4, 5 -> 0|>

The output is itself an association. It may be desirable to keep the format. If not:

List @@@ Normal[%]\[Transpose]
{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}

If you cannot use associations it is better is to use GatherBy:

{#[[1, 2]], Tr @ #[[All, 1]]} & /@ GatherBy[w\[Transpose], Last]\[Transpose]
{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}

Unlike Nasser's code this does not rescan the list for every unique element in the second part which would result in considerable slow-down as that number increases.

$\endgroup$
5
$\begingroup$

Using Reap and Sow:

Transpose[
 Last@Reap[
   MapThread[Sow, weightsPerclass], _, {#1, Total@#2} &]]
{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}
$\endgroup$
3
$\begingroup$

Late to the party, but here's a way using PositionIndex (v10):

With[
 {pi = PositionIndex[weightsPerclass[[2]]]},
 {Keys@pi, Plus @@ (weightsPerclass[[1, #]]) & /@ Values@pi}
 ]

{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}

$\endgroup$
2
$\begingroup$

Step for step:

data = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

mult = data[[1]] * data[[2]]

{1, 0.4, 0.9, 0.4, 0.3, 0.3, 0.9, 0}

union = Union @ data[[2]]

{1, 2, 3, 5}

pos = Flatten /@ Map[Position[data[[2]], #] &, union]

{{1, 4, 6, 7}, {2}, {3, 5}, {8}}

total = Total /@ Map[mult[[pos[[#]]]] &, Range @ Length @ pos]

{2.6, 0.4, 1.2, 0}

Join[{union, total}]

{{1, 2, 3, 5}, {2.6, 0.4, 1.2, 0}}

$\endgroup$
  • $\begingroup$ Please see my comment under Nasser's answer. Mapping Position (or the equivalent) should be avoided whenever possible if performance is a concern. $\endgroup$ – Mr.Wizard Oct 9 '14 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.