4
$\begingroup$

Is there some way to create lookups keyed on Temporary symbols that somehow avoid the referencing of the Temporary symbols, allowing the symbols and the values to still be automatically garbage collected?

(*$HistoryLength must be zero so that In/Out don't set up references to our symbols:*)
$HistoryLength = 0;
Names["Global`$myTemp*"]

ClearAll[CreateTemporarySymbol]

CreateTemporarySymbol[] := With[{p = Module[{pp = Unique["$myTemp", Temporary]}, pp]}, p]

Let's create two lists of these temp symbols. We can of course see the symbols in the Global context:

list[1] = Table[CreateTemporarySymbol[], 5]
list[2] = Table[CreateTemporarySymbol[], 5]
Names["Global`$myTemp*"]

(* -> {$myTemp3,$myTemp4,$myTemp5,$myTemp6,$myTemp7}*)
(* -> {$myTemp8,$myTemp9,$myTemp10,$myTemp11,$myTemp12}*)
(* -> {"$myTemp10","$myTemp11","$myTemp12","$myTemp3","$myTemp4","$myTemp5","$myTemp6","$myTemp7","$myTemp8","$myTemp9"}*)

Because the symbols were created as Temporary, deleting list[2] causes it's variables to be released and automatically cleaned up. Now only the symbols that are in list[1] remain:

list[2] =.
Names["Global`$myTemp*"]
(*-> {"$myTemp3","$myTemp4","$myTemp5","$myTemp6","$myTemp7"}*)

That is all the standard behavior.

Suppose, however, that I want to attach some values to these symbols. I don't want to store the values in the symbols themselves as I intend to use the symbols in many places and don't want to use up memory don't want the symbol to be evaluated for Printing or Saving or Cases or Select (the values might be large expressions the details of which are irrelevant to my purposes).

Is there some way of doing this that preserves their auto-cleanup when I remove the list[1] that contains them?

I had assumed that all of the DownValues/UpValues wouldn't work, but hoped that perhaps TagSet might.

Alas, it does not:

ClearAll[TryToHideSomeValues]

TryToHideSomeValues[myTemp_, value_] := With[{p = myTemp}, p /: HiddenValue[p] = value]

MapThread[TryToHideSomeValues[#1, #2] &, {list[1], RandomReal[{1, 11}, {5, 5}]}]

Map[HiddenValue, list[1]]

(* -> {{8.971, 5.41, 10.13, 8.482, 1.743}, {9.795, 7.289, 10.06, 1.640, 8.272}, {4.788, 5.803, 7.973, 6.589, 2.513}, {9.007, 2.325, 8.298, 7.071, 6.16}, {9.037, 3.004, 2.911, 8.603, 8.603}} *)

Naturally this is because TagSet is no different from anyone else in this regard and also registers any symbols on its LHS: deleting list[1] does not allow its symbols to get automatically released, as it had for list[2]:

list[1] =.
Names["Global`$myTemp*"]

(* -> {"$myTemp3","$myTemp4","$myTemp5","$myTemp6","$myTemp7"}*)

Is there some magic spot that one might hide such values such that the symbols and the values still get cleaned when list[1] is deleted?

I have looked around SE and seen e.g. 'Weak hash maps' and Language`NewExpressionStore which might somehow be solutions, but nothing obvious seemed to pop up.

In fact OwnValues does already operate in this requested fashion. Just because list[1] contains multiple copies of a symbol with OwnValues doesn't mean that the memory footprint is increased for each repetition ('laziness') - not until some sort of 'final' evaluation (like printing or saving) do the OwnValues of the symbols get inserted and expanded. So the memory storage of the values of the symbols is not the issue. It is more an issue of how a list of symbols with massive OwnValues can be saved or displayed without hassle.


In response to @b3m21a's very helpful answer I have made an updated version of my OP using NewExpressionStore.

It seems to me that NewExpressionStore is also incapable of providing the capabilities I desire: separation of symbols from values and the automatic deletion of symbols and values when the (Temporary) symbols are no longer referenced:

As before, I set up Temporary symbols, creating 5 of them inside list[1]:

$HistoryLength = 0;
Names["Global`$myTemp*"]

ClearAll[CreateTemporarySymbol]

CreateTemporarySymbol[] := With[{p = Module[{pp = Unique["$myTemp", Temporary]}, pp]}, p]

list[1] = Table[CreateTemporarySymbol[], 5]

-> {$myTemp3, $myTemp4, $myTemp5, $myTemp6, $myTemp7}

Names["Global`$myTemp*"]

-> {"$myTemp3", "$myTemp4", "$myTemp5", "$myTemp6", "$myTemp7"}

Modifying b3m21a's answer, I create hiddenCache which is a NewExpressionStore from the built-in Language` context:

hiddenCache = Language`NewExpressionStore["someLabel"]

... and 'put' a matrix directly into it:

hiddenCache["put"[HiddenValues, $myTemp3, {{7, 3}, {4, 9}}]]

The matrix is stored in the hiddenCache, associated with the (Temporary) $myTemp3 symbol:

hiddenCache["get"[HiddenValues, $myTemp3]]

-> {{7, 3}, {4, 9}}

... but since the matrix is not stored in the OwnValues of $myTemp3, it doesn't get expressed when we view list[1] (the matrix is also untouchable by Save or Cases or any other function that might operate on list[1]). This achieves my first objective of separating the values from the symbols:

$myTemp3
list[1]

-> $myTemp3

-> {$myTemp3, $myTemp4, $myTemp5, $myTemp6, $myTemp7}

Of course all of the $myTemp vars are in the namespace:

Names["Global`$myTemp*"]

-> {"$myTemp3", "$myTemp4", "$myTemp5", "$myTemp6", "$myTemp7"}

...But because they are Temporary, and because list[1] contains their only reference, they get automatically cleaned up when we delete list[1], except for $myTemp3 which is now unfortunately bound somewhere in the hiddenCache:

list[1] =.
Names["Global`$myTemp*"]

-> {"$myTemp3"}

The values are still being stored in hiddenCache, associated with $myTemp3, and binding to it so that it couldn't be automatically cleaned up. This fails my second objective of having the values being automatically removed at the same time as the symbols are:

hiddenCache["listTable"[]]

-> {{HiddenValues, {{$myTemp3, {{7, 3}, {4, 9}}}}}}

The only way to get $myTemp3 to be removed from the namespace is for HiddenValues to be 'removed':

hiddenCache["remove"[HiddenValues]]

Now the values that had been in hiddenCache are gone:

hiddenCache["listTable"[]]

->  {}

... and now $myTemp is truly cleaned up:

Names["Global`$myTemp*"]

-> {}

However this fails my desire of not having to do my own garbage collection.

$\endgroup$
  • $\begingroup$ You could use strings containing the (full) symbol names as keys... $\endgroup$ – Henrik Schumacher May 20 '19 at 14:15
  • 2
    $\begingroup$ This is exactly what ExpressionStore is built for. $\endgroup$ – b3m2a1 May 20 '19 at 16:31
  • $\begingroup$ Thanks @b3m2a1. I'll have to try out an implementation using it. As you say in your post, it does look a little clunky. $\endgroup$ – berniethejet May 20 '19 at 17:32
  • $\begingroup$ "I don't want to store the values in the symbols themselves as I intend to use the symbols in many places and don't want to use up memory (the values might be large numeric matrices)." - I don't get these memory considerations. You are probably not aware that Mathematica uses lazy copying. $\endgroup$ – Henrik Schumacher May 21 '19 at 1:00
  • $\begingroup$ Yes @HenrikSchumacher you are right, I wasn't thinking clearly on this point. I subsequently edited my post to fix this. But the problem still remains for things like Print or Save which will of course 'expand' all the symbols. Also things like Cases will search all of the OwnValues too, is that right? $\endgroup$ – berniethejet May 21 '19 at 12:00
1
$\begingroup$

Maybe OwnValues can be abused for that.

$HistoryLength = 0;
CreateTemporarySymbol[] := Module[{p = Unique["$myTemp", Temporary]}, p]
AppendToOwnValues[symbol_, key_, value_] := AppendTo[OwnValues[symbol], key :> value]
ExtractOwnValue[symbol_, key_] := key /. ReleaseHold[OwnValues[symbol]]

Creating temporary symbols and storing values in them:

list[1] = Table[CreateTemporarySymbol[], 5];
MapThread[AppendToOwnValues, {list[1], ConstantArray["Key", 5], RandomReal[{1, 11}, {5, 5}]}];

Retrieving a value:

ExtractOwnValue[list[1][[1]], "Key"]

{9.58016, 9.22525, 3.86703, 5.60385, 4.23212}

Deleting the list list[1] and checking that the symbols are really erased:

Names["Global`$myTemp*"]
list[1] =.
Names["Global`$myTemp*"]

{"$myTemp11", "$myTemp12", "$myTemp13", "$myTemp14", "$myTemp15"}

{}

However, I am pretty sure that OwnValues is not meant for that.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks Henrik, I have used this (abusive) method and can report that it is working now without issue for many months now. I really think Wolfram should make TagSet operate in this fashion by default - that is, not having symbols referenced on the LHS of the TagSet values being referenced in the symbol dependency table. $\endgroup$ – berniethejet Nov 7 '19 at 15:41
5
$\begingroup$

Here's an example of how you can use ExpressionStore for this:

tempCache = Language`NewExpressionStore["ilmbiahi"];

bindTempSyms[holder_Symbol, key_, syms_] :=
  tempCache@"put"[holder, key, syms];
getTempSyms[holder_Symbol, key_] :=
  tempCache@"get"[holder, key];

Module[{list},

 Module[{sym1, sym2, sym3},
  bindTempSyms[list, 1, {Hold[sym1], Hold[sym2], Hold[sym3]}];
  sym1 = "dog";
  sym2 = "cat";
  sym3 = "my shattered hopes and dreams";
  ];

 getTempSyms[list, 1][[-1]] // ReleaseHold
 ]

"my shattered hopes and dreams"

Names["sym*"]

{"sym1", "sym1$", "sym2", "sym2$", "sym3", "sym3$", "syms"}

Note that the Module-ized versions of these symbols don't appear.

What's nice about this is whenever list went out of scope—even if it were returned from the Module (and if list wouldn't have been bound to Out)—those bound symbols would go out of scope too.

On the other hand, you indicate that your biggest issue now is in display? I'm not sure exactly what you mean there, but there are many ways to display big values compactly. The standard way is through BoxForm`ArrangeSummaryBox. I won't get into how it works, but there are many, many usages of it across the site. Here's how it can work with a general data structure as implemented here:

RegisterDataStructure[
  pkg`myObj,
  pkg`myObj[Hold[sym_Symbol]]
  ];
pkg`myObj[] := Module[{tmp}, pkg`myObj@Hold[tmp]]

po = pkg`myObj[]

enter image description here

Here's how you could set the data on such an object:

pkg`setObjData[po, RandomReal[{}, {100, 100}]]

enter image description here

Note that the object data has changed:

pkg`myObj`Data@po // ReleaseHold // ByteCount

80152

This can be chained with ExpressionStore to make some very powerful encapsulation strategies.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @b3m2a1, this helps me a lot towards figuring out the NewExpressionStore syntax. I have posted an answer also with an implementation based on this answer of yours. However it still doesn't achieve what I am looking for. I have, however, changed things a bit (using the symbols as keys and not storing the Held-symbols, but the values directly), so perhaps my changes have destroyed your version's capabilities. $\endgroup$ – berniethejet May 21 '19 at 16:30
  • 1
    $\begingroup$ @berniethejet if your only remaining issues are in the display you can easily do that via a custom Format rule. Let me emphasize though that I think you might want to rethink how you’re writing this just a little bit. If something very simple (what you want to do) is becoming so convoluted and difficult, there is probably a simpler more direct path. $\endgroup$ – b3m2a1 May 22 '19 at 5:47
  • $\begingroup$ Thanks @b3m2a1. As everyone has pointed out, my goals weren't clearly stated. To be able to print out the list[1] without seeing the contents of the myTemp is just one part of suppressing myTemp. I also need to Save the list[i] without having the myTemp being expressed, and I operate on the list[i] in all sorts of ways with e.g. Cases which I don't want to search into the myTemp, to save time. $\endgroup$ – berniethejet May 22 '19 at 11:35
3
$\begingroup$

You could use strings containing the (full) symbol names as keys. This function may come in handy:

f = Function[s,
   Context[s] <> SymbolName[Unevaluated[s]],
   HoldFirst
   ];

Examples:

a = 1;
bla`a = 2;
bla`bli`blubb`a = 2;
f[a]
f[bla`a]
f[bla`bli`blubb`a]

"Global`a"

"bla`a"

"bla`bli`blubb`a"

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hmmm, yes, that is a good idea. I'll have to test it out. I have always worried about using the string 'interfaces' to variables as I have felt (but have yet to verify) that e.g. Names["Global$myTemp*'] slows down significantly the longer that MMA runs as it piles up more and more (often leaked) symbols. $\endgroup$ – berniethejet May 20 '19 at 15:33
  • $\begingroup$ The problem with this method is that I still have to do garbage collection on the DownValues of whatever lookup function I create on the stringified names, is that right? $\endgroup$ – berniethejet May 20 '19 at 22:20
  • $\begingroup$ Honestly, I don't understand your point... $\endgroup$ – Henrik Schumacher May 21 '19 at 0:57
  • $\begingroup$ Well, suppose I modified my function above to use your 'f' stringifier: TryToHideSomeValues[myTemp_, value_] := With[{p = f@myTemp}, p /: HiddenValue[p] = value]. Wouldn't I still have to do my own garbage collection on 'HiddenValue[p]'? $\endgroup$ – berniethejet May 21 '19 at 12:24
  • $\begingroup$ p /: HiddenValue[p] = value proablby won't work because p is now a string. Simply use With[{p = f@myTemp}, HiddenValue[p] = value]. $\endgroup$ – Henrik Schumacher May 21 '19 at 15:47
2
$\begingroup$

If it is OK to use OwnValues and you avoid them just to prevent large expressions from appearing as output, then you can wrap the temporary symbols you create in Hold and release it when necessary.

For example,

$HistoryLength = 0;
Names["temp*"]
(*{}*)

sym = Module[
 {temp},
 temp = 1;
 Hold[temp]
]

Names["temp*"]
(*{"temp", "temp$2465"}*)

sym =.
Names["temp*"]
(*{temp}*)

EDIT

By the way, I have once faced myself a task of garbage collecting Temporary symbols with DownValues. Unfortunately, I don't have time to write a full solution at the moment, but the main idea is simple: you introduce an auxiliary Temporary symbol to count references to your main symbol (which now can be not Temporary):

ClearAll[reference, new]
new[] := Module[
  {ref, store},
  reference[store] ^= Context[ref] <> SymbolName[ref];
  reference[ref, store]
]

Now you can attach any definitions to the generated store$123 symbol and use reference[ref$123, store$123] objects in your code to access them. If no reference[ref$123, store$123] objects are referenced any more, then ref$123 will be released, which you can check with Names@reference@store$123. Of course, this approach would require to call garbage collection manually from time to time, but the benefit of calling garbage collection this way is that you can implement a finilizer which will be executed before releasing store$123 symbol.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Alright thanks @AntonSakovich, this sounds interesting, let me try it out and get back to you. $\endgroup$ – berniethejet May 23 '19 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.