7
$\begingroup$

I am developing a weighted KNN algorithm. In a step, I need to do the sum of weights of each class. For example:

weightsPerclass = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

Here the first row contains the weights and the second row contains the class labels.

I need to do the sum for each class and I return

{{1, 2, 3, 5}, {2.6, .2, .6, 0}}

We can use a loop but I need a more compact solution. Any suggestion?

Improve this question
$\endgroup$

6 Answers 6

Reset to default
5
$\begingroup$

Many ways to do this. 

w = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};
class = Union[w[[2]]];
Map[Total, Cases[Transpose[w], {x_, y_} /; y == # :> x] & /@ class];
{class, %}

Mathematica graphics

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ FYI: There is a performance problem with this approach. The entire list must be rescanned for every element in class which will cause considerable slow-down as the length of that list increases. This is the problem I was referring to in (25591). $\endgroup$
    – Mr.Wizard
    Commented Oct 9, 2014 at 12:50
9
$\begingroup$

Additional methods using WeightedData, EmpiricalDistribution, GatherBy and SparseArray:

{weights,classes} = weightsPerclass = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

WeightedData

wd = WeightedData[classes, weights];

The property "EmpiricalPDF" almost gives what we need

wd["EmpiricalPDF"]
(* {{1,2,3,5},{0.8125,0.0625,0.125,0.}} *)

except that it has to be de-normalized by Total[weights]:

{First@# , Total[weights] Last@#} &@wd["EmpiricalPDF"]
(* {{1,2,3,5},{2.6,0.2,0.4,0.}} *)

Defining a function you can get the result in one step:

wdF = With[{p = WeightedData[#2, #1]["EmpiricalPDF"], w = Total @ #}, {1, w} p] &;

wdF[weights, classes]
(* {{1,2,3,5},{2.6,0.2,0.4,0.}} *)

EmpiricalDistribution

ed = EmpiricalDistribution[Rule @@ weightsPerclass];
ed["Domain"]
(* {1,2,3,5} *)

Total[weights] ed["Weights"]
(* {2.6,0.2,0.4,0.} *)

{%%, %}
(* {{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0.}} *)

You can also define a function that returns the desired output in one step:

ClearAll[wedF]
wedF = With[{d = EmpiricalDistribution[# -> #2], w = Total@#}, {d["Domain"], w d["Weights"]}] &;

wedF[weights, classes]
(* {{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0.}} *)

GatherBy

ClearAll[gthrBy];
gthrBy = Transpose[{Last@First@#, Total[First /@ #]} & /@ GatherBy[Transpose@#, Last]] &;

gthrBy@weightsPerclass
(* {{1,2,3,5},{2.6,0.2,0.4,0}} *)

SparseArray

System`SetSystemOptions["SparseArrayOptions" -> {"TreatRepeatedEntries" ->Total}];
sa = SparseArray[classes -> weights];
System`SetSystemOptions["SparseArrayOptions" -> {"TreatRepeatedEntries" -> First}];
{Flatten[sa["NonzeroPositions"]], sa["NonzeroValues"]}
(* {{1,2,3,5},{2.6,0.2,0.4,0}} *)

See also: Optimizing 2D binning code and Fast 2D binning algorithm

GroupBy - Version 10

Through[{Keys,Values}[GroupBy[Transpose@weightsPerclass,Last -> First, Total]]]
 (* {{1,2,3,5}, {2.6,0.2,0.4,0}} *)
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ You're a madman. $\endgroup$
    – kale
    Commented Oct 9, 2014 at 13:38
  • $\begingroup$ @kguler, thanks $\endgroup$
    – BetterEnglish
    Commented Oct 9, 2014 at 14:51
  • $\begingroup$ @kguler Why the empirical PDF does not give directly the best resuls. Why we have to multiply the results by the Total[weights]. Could you explain this point please? $\endgroup$
    – BetterEnglish
    Commented Oct 9, 2014 at 15:42
  • 1
    $\begingroup$ @Developer2000, PDF values are normalized so that they add up to 1 over the domain; i.e. pdf[i] is totalWeight [i] / totalWeight[j in Domain]. To get the raw weights we need to multiply pdf values by the total weight of all values in the domain. $\endgroup$
    – kglr
    Commented Oct 9, 2014 at 16:12
5
$\begingroup$

If you have Association functionality there is a rather nice approach using Merge:

w = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

Thread[#2 -> # & @@ w] ~Merge~ Total

<|1 -> 2.6, 2 -> 0.2, 3 -> 0.4, 5 -> 0|>

The output is itself an association. It may be desirable to keep the format. If not:

List @@@ Normal[%]\[Transpose]

{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}

If you cannot use associations it is better is to use GatherBy:

{#[[1, 2]], Tr @ #[[All, 1]]} & /@ GatherBy[w\[Transpose], Last]\[Transpose]

{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}

Unlike Nasser's code this does not rescan the list for every unique element in the second part which would result in considerable slow-down as that number increases.

Improve this answer
$\endgroup$
5
$\begingroup$

Using Reap and Sow:

Transpose[
 Last@Reap[
   MapThread[Sow, weightsPerclass], _, {#1, Total@#2} &]]

{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}
Improve this answer
$\endgroup$
3
$\begingroup$

Late to the party, but here's a way using PositionIndex (v10):

With[
 {pi = PositionIndex[weightsPerclass[[2]]]},
 {Keys@pi, Plus @@ (weightsPerclass[[1, #]]) & /@ Values@pi}
 ]

{{1, 2, 3, 5}, {2.6, 0.2, 0.4, 0}}

Improve this answer
$\endgroup$
2
$\begingroup$

Step for step:

data = {{1, 0.2, .3, .4, .1, .3, .9, 0}, {1, 2, 3, 1, 3, 1, 1, 5}};

mult = data[[1]] * data[[2]]

{1, 0.4, 0.9, 0.4, 0.3, 0.3, 0.9, 0}

union = Union @ data[[2]]

{1, 2, 3, 5}

pos = Flatten /@ Map[Position[data[[2]], #] &, union]

{{1, 4, 6, 7}, {2}, {3, 5}, {8}}

total = Total /@ Map[mult[[pos[[#]]]] &, Range @ Length @ pos]

{2.6, 0.4, 1.2, 0}

Join[{union, total}]

{{1, 2, 3, 5}, {2.6, 0.4, 1.2, 0}}

Improve this answer
$\endgroup$
1
  • $\begingroup$ Please see my comment under Nasser's answer. Mapping Position (or the equivalent) should be avoided whenever possible if performance is a concern. $\endgroup$
    – Mr.Wizard
    Commented Oct 9, 2014 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.