3
$\begingroup$

This question already has an answer here:

I have some data wrapped into a MyData[data_Association] "structure".

My Association contains some big arrays and I do not want to copy them each time I modify it.

By example, if I want to add a field, does the following code

foo[MyData[data_Association]] :=
  Block[{datacpy},
   datacpy = data;
   datacpy["extraField"] = 1;
   Return[MyData[datacpy]];
   ];

data=<|"bigArray"->{{}}|>; (* {{}} not empty in real situation! *)

foo[MyData[data]]

MyData[<|"bigArray" -> {{}}, "extraField" -> 1|>]

copy the "bigArray" field even if it is not modified? (or does it use some binding/reference mechanism).

$\endgroup$

marked as duplicate by Leonid Shifrin, Community May 10 at 12:45

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Maybe store the associations not in an array but in a dictionary? (like dict[key]=key->value) and store dictionary globally? $\endgroup$ – Kagaratsch May 7 at 19:54
  • $\begingroup$ I find the "wrapping" trick SomeSymbol[data...] very useful and I wanted to keep using this simple approach. But thanks for the suggestion. $\endgroup$ – Picaud Vincent May 7 at 19:57
  • $\begingroup$ I feel this is a dupe, but feel free to object. $\endgroup$ – Leonid Shifrin May 10 at 12:18
4
$\begingroup$

I think you're asking whether the foo operation will store two independent copies of the big array. That is, given:

MemoryInUse[]
ds = MyData @ Association["BigArray"->RandomReal[1,10^8]];
MemoryInUse[]

42312736

842355744

Will your foo operation basically double the memory in use? Here is your foo function (rewritten to eliminate the superfluous Return that should never be used at the end of a Module or Block):

foo[MyData[data_Association]] := Block[{datacpy = data},
    datacpy["extraField"] = 1;
    MyData[datacpy]
]

Then:

MemoryInUse[]
foo[ds];
MemoryInUse[]

845119768

845122344

I think the above shows that the answer is no.

$\endgroup$
  • $\begingroup$ I feel really uncomfortable not knowing about Return[] usage -> I just found this mathematica.stackexchange.com/questions/58059/… Hopefully for my mood the post starts with "Return is surprisingly under-documented" $\endgroup$ – Picaud Vincent May 7 at 20:26
  • 1
    $\begingroup$ Return is almost useless. It isn't really under-documented, but if you expect it to be like "return" in other languages, you'll find nothing to support such usage. $\endgroup$ – John Doty May 7 at 20:40
  • $\begingroup$ @JohnDoty thanks, this is indeed the trap I fell into. $\endgroup$ – Picaud Vincent May 7 at 21:00
3
$\begingroup$

Here's a method you could use if you really want to directly modify your association yourself. To do that, you need to store the association in a variable and use Hold attributes to prevent it from evaluating. In your case, it seems like it's easiest to just make MyData HoldFirst:

SetAttributes[MyData, HoldFirst]

You can now define your function as:

foo[MyData[data_Symbol?AssociationQ]] := (
  data["extraField"] = 1;
  MyData[data]
);

The AssociationQ pattern test will ensure that the symbol data refers to an association. You can use the function like so:

actualData = <|"bigArray" -> {{}}|>;
foo[MyData[actualData]]
actualData

Out[15]= MyData[actualData]

Out[16]= <|"bigArray" -> {{}}, "extraField" -> 1|>

Note that MyData[actualData] does not show the association because of the HoldFirst attribute. You could define custom formatting rules to make it show a preview of the data. You could also try to turn MyData into a sort of constructor that will automatically create a new symbol to hold an association for you. For example:

ClearAll[MyData]
SetAttributes[MyData, HoldFirst]
MyData[stuff : Except[_Symbol]] := With[{
    evaluatedStuff = stuff
  },
   Module[{storeVar = evaluatedStuff},
     MyData[storeVar]
   ] /; AssociationQ[evaluatedStuff]
 ];
MyData[AssociationThread[Range[10], RandomReal[1, 10]]]
First[%]

MyData[storeVar$6198]

<|1 -> 0.236334, 2 -> 0.354161, 3 -> 0.314371, 4 -> 0.738186, 5 -> 0.916299, 6 -> 0.0289776, 7 -> 0.831803, 8 -> 0.533609, 9 -> 0.316124, 10 -> 0.211526|>

All of this is roughly how Association itself actually works. Note the attributes on Association:

In[29]:= Attributes[Association]

{HoldAllComplete, Protected}

This is what is meant when people say that Association is a constructor and a container. You can tell that some sort of construction step takes place when you define an association:

Unevaluated[<||>] === <||>

False

$\endgroup$
  • $\begingroup$ very interesting and clearly explained, thanks $\endgroup$ – Picaud Vincent May 8 at 15:07
  • $\begingroup$ If you are interested, I asked a related question here mathematica.stackexchange.com/q/198105/42847 $\endgroup$ – Picaud Vincent May 10 at 10:47
  • $\begingroup$ You can be even slicker with this (rather than just checking for a Symbol) but checking for the NoEntry or Valid bits. These give you a way to build a constructor that can even take a Symbol as input and check that this is an Association in a very clean and efficient way. These bits can also be applied all over the place to tell that you have a proper constructed object and not just something that syntactically matches. That's a powerful property to have. $\endgroup$ – b3m2a1 May 10 at 19:55
2
$\begingroup$

It all depends on how you want to do the modification. In general, Mathematica is actually very efficient about handling pass-by-value calls. If you look at memory consumption "bigArray" will not be copied:

struct[] :=
  struct[<||>];
struct[a_]@"Add"[field_ , val_] :=
  struct[Append[a, field -> val]];
struct[a_]@"Modify"[field_ , fn_] :=
  struct[ReplacePart[a, field -> fn@a[field]]];

myStruct = struct[];

Block[{arr = RandomReal[{}, {500, 500}]},
 (* done to prevent the memory getting stuck to Out *)
 myStruct = myStruct@"Add"["bigArray", arr];
 {arr // ByteCount, memPrev = MemoryInUse[]}
 ]

{2000152, 974574520}

Block[{},
 (* done to prevent the memory getting stuck to Out *)
 myStruct = myStruct@"Add"["empty", None];
 MemoryInUse[] - memPrev
 ]

3472

And in fact "bigArray" is really stored as an object with an internal ID as can be seen in the many low-level functions that depend on explicit expression identity.

Of course, you could do this with a direct mutable OOP approach, too, but I won't get into that here.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.