Efficient manipulation of Associations passed to functions, how-to? [duplicate]

Question

I have some data wrapped into a MyData[data_Association] "structure".

My Association contains some big arrays and I do not want to copy them each time I modify it.

By example, if I want to add a field, does the following code

foo[MyData[data_Association]] :=
  Block[{datacpy},
   datacpy = data;
   datacpy["extraField"] = 1;
   Return[MyData[datacpy]];
   ];

data=<|"bigArray"->{{}}|>; (* {{}} not empty in real situation! *)

foo[MyData[data]]

MyData[<|"bigArray" -> {{}}, "extraField" -> 1|>]

copy the "bigArray" field even if it is not modified? (or does it use some binding/reference mechanism).

Answer 1

I think you're asking whether the foo operation will store two independent copies of the big array. That is, given:

MemoryInUse[]
ds = MyData @ Association["BigArray"->RandomReal[1,10^8]];
MemoryInUse[]

42312736

842355744

Will your foo operation basically double the memory in use? Here is your foo function (rewritten to eliminate the superfluous Return that should never be used at the end of a Module or Block):

foo[MyData[data_Association]] := Block[{datacpy = data},
    datacpy["extraField"] = 1;
    MyData[datacpy]
]

Then:

MemoryInUse[]
foo[ds];
MemoryInUse[]

845119768

845122344

I think the above shows that the answer is no.

Answer 2

Here's a method you could use if you really want to directly modify your association yourself. To do that, you need to store the association in a variable and use Hold attributes to prevent it from evaluating. In your case, it seems like it's easiest to just make MyData HoldFirst:

SetAttributes[MyData, HoldFirst]

You can now define your function as:

foo[MyData[data_Symbol?AssociationQ]] := (
  data["extraField"] = 1;
  MyData[data]
);

The AssociationQ pattern test will ensure that the symbol data refers to an association. You can use the function like so:

actualData = <|"bigArray" -> {{}}|>;
foo[MyData[actualData]]
actualData

Out[15]= MyData[actualData]

Out[16]= <|"bigArray" -> {{}}, "extraField" -> 1|>

Note that MyData[actualData] does not show the association because of the HoldFirst attribute. You could define custom formatting rules to make it show a preview of the data. You could also try to turn MyData into a sort of constructor that will automatically create a new symbol to hold an association for you. For example:

ClearAll[MyData]
SetAttributes[MyData, HoldFirst]
MyData[stuff : Except[_Symbol]] := With[{
    evaluatedStuff = stuff
  },
   Module[{storeVar = evaluatedStuff},
     MyData[storeVar]
   ] /; AssociationQ[evaluatedStuff]
 ];
MyData[AssociationThread[Range[10], RandomReal[1, 10]]]
First[%]

MyData[storeVar$6198]

<|1 -> 0.236334, 2 -> 0.354161, 3 -> 0.314371, 4 -> 0.738186, 5 -> 0.916299, 6 -> 0.0289776, 7 -> 0.831803, 8 -> 0.533609, 9 -> 0.316124, 10 -> 0.211526|>

All of this is roughly how Association itself actually works. Note the attributes on Association:

In[29]:= Attributes[Association]

{HoldAllComplete, Protected}

This is what is meant when people say that Association is a constructor and a container. You can tell that some sort of construction step takes place when you define an association:

Unevaluated[<||>] === <||>

False

Answer 3

It all depends on how you want to do the modification. In general, Mathematica is actually very efficient about handling pass-by-value calls. If you look at memory consumption "bigArray" will not be copied:

struct[] :=
  struct[<||>];
struct[a_]@"Add"[field_ , val_] :=
  struct[Append[a, field -> val]];
struct[a_]@"Modify"[field_ , fn_] :=
  struct[ReplacePart[a, field -> fn@a[field]]];

myStruct = struct[];

Block[{arr = RandomReal[{}, {500, 500}]},
 (* done to prevent the memory getting stuck to Out *)
 myStruct = myStruct@"Add"["bigArray", arr];
 {arr // ByteCount, memPrev = MemoryInUse[]}
 ]

{2000152, 974574520}

Block[{},
 (* done to prevent the memory getting stuck to Out *)
 myStruct = myStruct@"Add"["empty", None];
 MemoryInUse[] - memPrev
 ]

3472

And in fact "bigArray" is really stored as an object with an internal ID as can be seen in the many low-level functions that depend on explicit expression identity.

Of course, you could do this with a direct mutable OOP approach, too, but I won't get into that here.