2
$\begingroup$

My question is in two parts. The first is about picking discrete elements inside a list of Associations to match a fixed percentage of a final sub-list; the second is about overcoming the rounding problem the first part creates.

I have data in a List of Associations, a portion of which looks like this

{<|"Sector" -> "A", "ID" -> 1101, "Year" -> 1990, "AnnReturn" -> 0.0946|>, <|"Sector" -> "A", "ID" -> 1102,  "Year" -> 1990, "AnnReturn" -> 0.0901|>, <|"Sector" -> "A", "ID" -> 1103, "Year" -> 1990, "AnnReturn" -> 0.00876|>, <|"Sector" -> "A", "ID" -> 1104, "Year" -> 1990, "AnnReturn" -> 0.0242|>, <|"Sector" -> "A", "ID" -> 1105, "Year" -> 1990, "AnnReturn" -> 0.0783|>, <|"Sector" -> "A", "ID" -> 1106, "Year" -> 1990, "AnnReturn" -> 0.0433|>, <|"Sector" -> "A", "ID" -> 1107, "Year" -> 1990, "AnnReturn" -> 0.0253|>, <|"Sector" -> "A", "ID" -> 1108, "Year" -> 1990, "AnnReturn" -> 0.1144|>, <|"Sector" -> "A", "ID" -> 1109, "Year" -> 1990, "AnnReturn" -> 0.1036|>, <|"Sector" -> "A", "ID" -> 111, "Year" -> 1990, "AnnReturn" -> 0.1216|>, <<145845>>}

The first Key = Sector has four possibilities, A, B, C and D

I need to compose a sub-list OF UNIQUE VALUES which has a fixed (but uneven) percentage of each sector. Here is an example where the proportions correspond to "A","B","C" and "D"

{0.13, 0.19, 0.28, 0.4}

Note the decimals must add to unity.

So, to be completely tedious, a sub-list of, say, 1033 elements would have

Round[1033 {0.13, 0.19, 0.28, 0.4}] 

to get this

(*{134, 196, 289, 413}*)

...which adds to 1032. What I need should be the total sought less the first three proportions, producing the fourth. I am sure there is a better way than this, but this works

Join[Round[1033 #], {1033 - Plus @@ Round[1033 #]}] &[{0.13, 0.19, 
  0.28}]

@MarcoB suggested this

grupBySec = GroupBy[netLoA, #"Sector" &];
RandomChoice[{0.13, 0.19, 0.28, 0.4} -> grupBySec]

but it only randomizes the sector. It looks like the Documentation for weights but those examples do not use Associations

$\endgroup$
4
  • $\begingroup$ Group your list of associations by the value corresponding to the sector key, then sample from the list of lists with RandomChoice, using {0.13, 0.19, 0.28, 0.4} as weights? $\endgroup$
    – MarcoB
    Sep 1, 2022 at 0:55
  • $\begingroup$ @MarcoB - my FAULT!!! The sample is without replacement. I edited the above (BOLD). Since you got 1033 picks out of 1000 there are duplicates. Am presently combing through your code to find a solution $\endgroup$
    – Rogo
    Sep 1, 2022 at 18:56
  • $\begingroup$ That's actually an easy fix: use RandomSample instead of RandomChoice. I've updated my answer to reflect that and added a check for duplicates. $\endgroup$
    – MarcoB
    Sep 1, 2022 at 20:34
  • $\begingroup$ Great!! I really appreciate it. I know SE frowns on superfluous commentary but it is hard to restrain myself... $\endgroup$
    – Rogo
    Sep 2, 2022 at 1:22

1 Answer 1

2
$\begingroup$

Let me first generate some fake data fakeList that has the same structure as your list:

n = 10000;
fakeList = MapThread[
   <|"Sector" -> #1, "ID" -> #2, "Year" -> #3, "AnnReturn" -> #4|> &,
   {
     RandomChoice[{"A", "B", "C", "D"}, n],
     Range[1101, 1101 + n - 1],
     RandomInteger[{1990, 2000}, n],
     RandomReal[{0, 0.2}, n]
   }
];

Let's now assign your weights to a variable for readability:

weights = {0.13, 0.19, 0.28, 0.4};

With those in hand, let's reorganize the list, grouping it by "Sector". We also sort it to make sure that the groups are listed in alphabetical order of sector, so the relative weights are correctly assigned to each sector later on:

organizedList = SortBy[#Sector &@*First]@GatherBy[fakeList, #Sector &]

Let's now generate the number of samples to pick for each group, based on the number of desired samples overall. I use your example of 1033 samples:

With[{samples = 1033},
 counts = {##, samples - Total[{##}]} & @@ Round[samples weights[[;; 3]]]
]

(* Out: {134, 196, 289, 414} *)

Now let's pick a random selection without replacement of samples according to the output above from each grouped list. We do that using RandomSample, which never samples any element more than once from its input.

picks =
  MapThread[
    RandomSample[#1, #2] &,
    {organizedList, counts}
  ] // Flatten

(* Out: 
{<|"Sector" -> "A", "ID" -> 1952, "Year" -> 1994, "AnnReturn" -> 0.163849|>, 
 <|"Sector" -> "A", "ID" -> 1822, "Year" -> 1998, "AnnReturn" -> 0.0230599|>, 
 <|"Sector" -> "A", "ID" -> 1670, "Year" -> 1992, "AnnReturn" -> 0.0859481|>, 
 
 <<1027>>,

 <|"Sector" -> "D", "ID" -> 1651, "Year" -> 1999, "AnnReturn" -> 0.102282|>, 
 <|"Sector" -> "D", "ID" -> 2074, "Year" -> 1996, "AnnReturn" -> 0.192641|>, 
 <|"Sector" -> "D", "ID" -> 1539, "Year" -> 1990, "AnnReturn" -> 0.00170126|>}
*)

Finally, let's check that we did indeed obtain the right number of samples from each "Sector" in the output:

CountsBy[#Sector &][picks]

(* Out: <|"A" -> 134, "B" -> 196, "C" -> 289, "D" -> 414|> *)

... and that there are no duplicates in the output:

DeleteDuplicates[picks] == picks
(* True *)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.