How to organically merge nested associations?

Question

Suppose I want to construct an association of associations, such as a list of people with attributes:

peopleFacts=<| alice-> <|age->29,shoeSize->7|>, bob-> <|age->27,sex->male|> |>

However, I want to grow and update this organically by adding facts as I learn them.

peopleFacts[["steve","hairColor"]] = "red";
peopleFacts[["bob","age"]] = "22";
peopleFacts[["steve","major"]] = "physics";

It's possible to accomplish this awkwardly by either (a) filling the database with blank entries or (b) laboriously checking at each level of association to see if an entry is blank before filling it in (except the last level, where AssociateTo helps you). But I think there must be a more elegant way. Here is what I've tried.

This method breaks because it tosses out the second key:

 In[]:= peopleFacts[["steve","hairColor"]] = "red";
        peopleFacts

Out[]:= <|steve -> red, alice-> <|age->29,shoeSize->7|>, bob-> <|age->27,sex->male|> |>

This method drops existing data:

 In[]:= peopleFacts

Out[]:= <| alice-> <|age->29,shoeSize->7|>, bob-> <|age->27,sex->male|> |>

 In[]:= AssociateTo[peopleFacts, alice-> <|"sport"->"baseball"|>;
        peopleFacts

Out[]:= <| alice-> <|sport->baseball|>, bob-> <|age->27,sex->male|> |>

This method just doesn't evaluate:

 In[]:= AssociateTo[peopleFacts[["chris"]], "favoriteFood" -> "sushi"]

Out[]:= AssociateTo[peopleFacts[["chris"]], "favoriteFood" -> "sushi"]

EDIT: Here is a way-too-awkward method adapted from this answer by SuTron.

 In[]:= peopleFacts

Out[]:= <| alice-> <|age->29,shoeSize->7|>, bob-> <|age->27,sex->male|> |>

 In[]:= Module[{temp = peopleFacts["alice"]},
          AssociateTo[temp, "sport"->"baseball"];
          AssociateTo[peopleFacts, "alice" -> temp];
        ];
        peopleFacts

Out[]:= <| alice-> <|age->29,shoeSize->7,sport->baseball|>, bob-> <|age->27,sex->male|> |>

It's not hard to imagine defining a custom update function like

  NestedAssociateTo[peopleFacts,{"steve","haircolor","red"}]

that would handle this all for you, but I'd much rather have a nice native Mathematica solution that is optimized, and that I don't have to maintain or worry about.

This question feels familiar but I can recall neither an answer nor the original I may be thinking of. (For the sake of my own organization alone this is vaguely related: (88442)) — Mr.Wizard, Commented Dec 20, 2015 at 2:03
Ahh, I think my question is essentially a special case of (82915), except that additional work is needed to generalize the answer to that question for arbitrary levels of nesting. — Jess Riedel, Commented Dec 20, 2015 at 14:52
Seeing Kuba's solution I am reminded that these are at least distantly related: (83507), (86578) — Mr.Wizard, Commented Dec 20, 2015 at 16:36
@JessRiedel - can you edit to make your symbols vs strings consistent? You have, both bob and "bob". — alancalvitti, Commented Aug 31, 2017 at 21:12

Kuba · Accepted Answer · 2016-08-12 13:31:36Z

Initial data:

peopleFacts = <|
    alice -> <|age -> 29, shoeSize -> 7|>, 
    bob -> <|age -> 27, sex -> male,  hair -> <|Color -> RGBColor[1, 0, 0]|>
    |>
|>;

Here is a version of RecurAssocMerge reduced to a single definition.

MergeNested = If[MatchQ[#, {__Association}], Merge[#, #0], Last[#]] &

MergeNested @ {peopleFacts, <|bob -> <|hair -> <|length -> 120|>|>|>}

 <|
   alice -> <|
     age -> 29, 
     shoeSize -> 7|>, 
   bob -> <|
     age -> 27, 
     sex -> male,  
     hair -> <|Color -> RGBColor[1, 0, 0], length -> 120|>
   |>
 |>

Special case of 2-level deep association

Merge[{
   peopleFacts,
   <|bob -> <|hairColor -> 1|>|>
 },
 Association
]

"Tidy" approach to write NestedMerge:

RecurAssocMerge[a : {__Association}] := Merge[a, RecurAssocMerge];
RecurAssocMerge[a_] := Last[a];

adding key to deep level association:

RecurAssocMerge[
  {peopleFacts, <|bob -> <|hair -> <|length -> 120|>|>|>}
 ]

 <|alice -> <|age -> 29, shoeSize -> 7|>, 
   bob -> <|age -> 27, sex -> male, hair -> <|
        Color -> RGBColor[1, 0, 0], length -> 120 |>
   |>
 |>

entirely new tree

RecurAssocMerge[
   {peopleFacts, <|kuba -> <|hair -> <|length -> 120|>|>|>}
 ]

 <|
    alice -> <|age -> 29, shoeSize -> 7|>, 
    bob -> <|age -> 27, sex -> male, hair -> <|Color -> RGBColor[1, 0, 0]|>
    |>, 
    kuba -> <|hair -> <|length -> 120|>|>
|>

Section added by Jess Riedel:

Specialize to single new entry

RecurAssocMerge defined above is a general method for merging nested Associations. We can define an abbreviation for the special case when we are adding only a single new entry.

RecurAssocMerge[ini_Association, path_List, value_] := RecurAssocMerge[{
   ini, Fold[<|#2 -> #|> &, value, Reverse@path]
}]

Then we can just do

RecurAssocMerge[peopleFacts, {bob, hair, length}, 120]

 <|alice -> <|age -> 29, shoeSize -> 7|>, 
   bob -> <|age -> 27, sex -> male, hair -> <|
            Color -> RGBColor[1, 0, 0], length -> 120 |>
     |>
   |>

Notes

If you want to modify peopleFacts the peopleFacts = Merge... is needed of course.

how to modify your code if it is not known per se on what level of Association to Merge? (I mean, if bob is nested deeper?) — garej, Commented Dec 20, 2015 at 11:11
Merge is a good solution. Would be good if it had HoldFirst as the structure could be come big. — Edmund, Commented Dec 20, 2015 at 12:00
@Kuba, can you please modify your answer to show the general case where the Associations are nested to deeper levels? I don't understand your comment to @garej about using GroupBy. — Jess Riedel, Commented Dec 20, 2015 at 15:20
@JessRiedel Accepted after corrections. There were some undefined functions in the last section. But I've changed it anyway to something more compact. Also, there were string keys in the last example while everything is done on symbols, so I dropped them to be consistent. I'd call the function MergeNestedAssociation but that is not important. And finally, thanks for the edit! :) — Kuba, Commented Dec 20, 2015 at 19:14
v.useful, succinct solution of what would seem to be a common idiom (perhaps justifying a system implementation to accommodate the OP's initial tendency: peopleFacts[["bob","hair","Length"]] = "red") Related to 2249 and implements the advocated use of Part to "Creatively Set" values in List`s/Association`s (or with operator form peopleFacts @= CreativeSet[{"bob", "hair", "Length"}, "red"] where CreativeSet[pos_,value_] := Function[a,RecurAssocMerge[a, pos,value]]) — Ronald Monson, Commented Jan 20, 2016 at 7:52

Edmund · Accepted Answer · 2015-12-20 14:04:09Z

Update

Created an upsert function to update/insert new keys and values into a nested association structure. It automatically inserts nested associations where they do not exists and does not need to be assigned back to the original association. It updates existing keys when they are found.

ClearAll[upsert]
Attributes[upsert] = {HoldFirst};
upsert[dat_?AssociationQ, key_, value__] :=
 If[First@Dimensions@{value} == 1,
  dat[key] = value,
  (
   If[KeyExistsQ[dat, key] == False, dat[key] = <||>];
   upsert[dat[key], First@{value}, Sequence @@ Rest@{value}]
  )
  ]

Can use upsert with as many nested levels as needed.

peopleFacts = <|"alice" -> <|"age" -> 29, "shoeSize" -> 7|>, 
   "bob" -> <|"age" -> 27, "sex" -> "male"|>|>;

Insert "steve" and association "haircolor" key/value.

upsert[peopleFacts, "steve", "haircolor", "Red"];
peopleFacts

(* <|"alice" -> <|"age" -> 29, "shoeSize" -> 7|>, 
 "bob" -> <|"age" -> 27, "sex" -> "male"|>, 
 "steve" -> <|"haircolor" -> "Red"|>|> *)

Insert "tim", association "music" key/value, and nested association "rock" key/value.

upsert[peopleFacts, "tim", "music", "rock", "jimmy"];
peopleFacts

(* <|"alice" -> <|"age" -> 29, "shoeSize" -> 7|>, 
 "bob" -> <|"age" -> 27, "sex" -> "male"|>, 
 "steve" -> <|"haircolor" -> "Red"|>, 
 "tim" -> <|"music" -> <|"rock" -> "jimmy"|>|>|> *)

Update "alice" "age".

upsert[peopleFacts, "alice", "age", 25];
peopleFacts

(* <|"alice" -> <|"age" -> 25, "shoeSize" -> 7|>, 
 "bob" -> <|"age" -> 27, "sex" -> "male"|>, 
 "steve" -> <|"haircolor" -> "Red"|>, 
 "tim" -> <|"music" -> <|"rock" -> "lenny"|>|>|> *)

Original Post

Each time there is a new key that has an association as its value you must initialise it as an association. Then you can use the feature of Association that creates a key when a value is assigned to a non-existing key.

peopleFacts = <|"alice" -> <|"age" -> 29, "shoeSize" -> 7|>, "bob" -> <|"age" -> 27, "sex" -> "male"|>|>;

peopleFacts["steve"] = <||>;
peopleFacts
(* <|alice -> <|age -> 29, shoeSize -> 7|>, 
 bob -> <|age -> 27, sex -> male|>, steve -> <||>|> *)

peopleFacts["steve"]["hairColor"] = "Red";
peopleFacts
(* <|alice -> <|age -> 29, shoeSize -> 7|>, 
 bob -> <|age -> 27, sex -> male|>, steve -> <|hairColor -> Red|>|> *)

peopleFacts["bob"]["age"] = 22;
peopleFacts
(* <|alice -> <|age -> 29, shoeSize -> 7|>, 
 bob -> <|age -> 22, sex -> male|>, steve -> <|hairColor -> Red|>|> *)

peopleFacts["steve"]["major"] = "Physics";
peopleFacts
(* <|alice -> <|age -> 29, shoeSize -> 7|>, 
 bob -> <|age -> 22, sex -> male|>, 
 steve -> <|hairColor -> "Red", major -> "Physics"|>|> *)

Hope this helps.

I'm afraid this doesn't help much. Using your method, I have to check whether I'm writing over existing data, right? If I already have a bunch of facts about steve, then initializing peopleFacts[steve] = <||> erases them all. — Jess Riedel, Commented Dec 20, 2015 at 2:29
When you say "Each time there is a new key that has an association as its value you must initialise it as an association", are you just unaware of a more elegant method, or do you have some reason to be sure none exists? — Jess Riedel, Commented Dec 20, 2015 at 2:31
I know, but that becomes very cumbersome if you're adding elements to a database that's several layers nested. — Jess Riedel, Commented Dec 20, 2015 at 2:40
HoldFirst passes the first parameter by reference. This prevents a copy of it begin created (would eat up memory for large items). It also allows the association passed in to be directly updated (no need to assign back to the original association). — Edmund, Commented Dec 20, 2015 at 14:15

alancalvitti · Accepted Answer · 2017-08-04 17:14:04Z

3

Try this, we're using it on a daily basis

Nest[Merge,f,n]

To your starting data, slightly modified (strings vs symbols):

peopleFacts = <|"alice" -> <|"age" -> 29, "shoeSize" -> 7|>, 
   "bob" -> <|"age" -> 27, "sex" -> "male"|>|>;

And new facts:

newFacts = <|
  "steve" -> <|"hairColor" -> "red", "major" -> "physics"|>,
  "bob" -> <|"age" -> 22|>|>

Update semantics for replacing existing values: (1) add newFacts last, (2) apply Last eg:

{peopleFacts, newFacts} // Nest[Merge, Last, 2]

<|"alice" -> <|"age" -> 29, "shoeSize" -> 7|>, "bob" -> <|"age" -> 22, "sex" -> "male"|>, "steve" -> <|"hairColor" -> "red", "major" -> "physics"|>|>

Keep in mind Merge does not yet have full complement of options like JoinAcross eg inner/outer/left/right and will not impute missing keys, so often KeyIntersection is required.

Also will admit for some types of ragged hierarchies, there's no current easy way to merge all relevant branches automatically (an All level spec). In other words, knowledge of the schema is required.

edited Aug 4, 2017 at 17:14

answered Aug 3, 2017 at 21:41

alancalvitti

15.2k3 gold badges27 silver badges93 bronze badges

$\begingroup$ Let us continue this discussion in chat. $\endgroup$
– Kuba
Commented Aug 4, 2017 at 19:14
$\begingroup$ @Kuba, chat is frozen - just added a recursive approach based on your answer. Let me know if you find it useful. This is the 3rd example of query recursion that I'd like to present @ WTC. $\endgroup$
– alancalvitti
Commented Aug 31, 2017 at 21:26
$\begingroup$ Can't focus too much on this topic atm. Does it do anything more than my MergeNested? Or is there any reason to prefer one over another? Performance? Flexibility? Because readability goes for MergeNested I guess. $\endgroup$
– Kuba
Commented Sep 1, 2017 at 11:56
$\begingroup$ I think the way you defined MergeNested it only works on 2 levels - if so the comparison should be made to the general recursive version. $\endgroup$
– alancalvitti
Commented Sep 1, 2017 at 16:27
$\begingroup$ I'm not sure I understand. The example is already 3 level deep: <|bob -> <|hair -> <|length -> 120|>|>|>. Could you elaborate? $\endgroup$
– Kuba
Commented Sep 1, 2017 at 18:12

| Show 3 more comments

alancalvitti · Accepted Answer · 2017-08-31 21:35:24Z

2

This recursive Query is inspired by Kuba's answer:

nestedMerge[f_] := 
  Query[Merge[Identity] /* 
    Query[All, {MatchQ[{__Association}], Identity} /* 
      Replace[{{True, data_} :> 
         nestedMerge[f][data], {False, data_} :> Query[f][data]} ]]];

Test on peopleFacts - note appended data is rewritten as nested Association vs OP:

{peopleFacts, <|steve -> <|hairColor -> red|>|>, <|
    bob -> <|age -> 22|>|>, <|steve -> <|major -> physics|>|>} // 
  nestedMerge[Last] // Dataset

edited Aug 31, 2017 at 21:35

answered Aug 31, 2017 at 21:21

alancalvitti

15.2k3 gold badges27 silver badges93 bronze badges

Add a comment |

Stack Exchange Network

How to organically merge nested associations?

4 Answers 4

Specialize to single new entry

Notes

Update

Your Answer

Not the answer you're looking for? Browse other questions tagged
associations
gathering
or ask your own question.

Linked

Hot Network Questions

How to organically merge nested associations?

4 Answers 4

Specialize to single new entry

Notes

Update

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged associationsgathering or ask your own question.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
associations
gathering
or ask your own question.