Is it possible to create lookups on temporary symbols that still allow the symbols to be automatically removed?

Question

Is there some way to create lookups keyed on Temporary symbols that somehow avoid the referencing of the Temporary symbols, allowing the symbols and the values to still be automatically garbage collected?

(*$HistoryLength must be zero so that In/Out don't set up references to our symbols:*)
$HistoryLength = 0;
Names["Global`$myTemp*"]

ClearAll[CreateTemporarySymbol]

CreateTemporarySymbol[] := With[{p = Module[{pp = Unique["$myTemp", Temporary]}, pp]}, p]

Let's create two lists of these temp symbols. We can of course see the symbols in the Global context:

list[1] = Table[CreateTemporarySymbol[], 5]
list[2] = Table[CreateTemporarySymbol[], 5]
Names["Global`$myTemp*"]

(* -> {$myTemp3,$myTemp4,$myTemp5,$myTemp6,$myTemp7}*)
(* -> {$myTemp8,$myTemp9,$myTemp10,$myTemp11,$myTemp12}*)
(* -> {"$myTemp10","$myTemp11","$myTemp12","$myTemp3","$myTemp4","$myTemp5","$myTemp6","$myTemp7","$myTemp8","$myTemp9"}*)

Because the symbols were created as Temporary, deleting list[2] causes it's variables to be released and automatically cleaned up. Now only the symbols that are in list[1] remain:

list[2] =.
Names["Global`$myTemp*"]
(*-> {"$myTemp3","$myTemp4","$myTemp5","$myTemp6","$myTemp7"}*)

That is all the standard behavior.

Suppose, however, that I want to attach some values to these symbols. I don't want to store the values in the symbols themselves as I intend to use the symbols in many places and don't want to use up memory don't want the symbol to be evaluated for Printing or Saving or Cases or Select (the values might be large expressions the details of which are irrelevant to my purposes).

Is there some way of doing this that preserves their auto-cleanup when I remove the list[1] that contains them?

I had assumed that all of the DownValues/UpValues wouldn't work, but hoped that perhaps TagSet might.

Alas, it does not:

ClearAll[TryToHideSomeValues]

TryToHideSomeValues[myTemp_, value_] := With[{p = myTemp}, p /: HiddenValue[p] = value]

MapThread[TryToHideSomeValues[#1, #2] &, {list[1], RandomReal[{1, 11}, {5, 5}]}]

Map[HiddenValue, list[1]]

(* -> {{8.971, 5.41, 10.13, 8.482, 1.743}, {9.795, 7.289, 10.06, 1.640, 8.272}, {4.788, 5.803, 7.973, 6.589, 2.513}, {9.007, 2.325, 8.298, 7.071, 6.16}, {9.037, 3.004, 2.911, 8.603, 8.603}} *)

Naturally this is because TagSet is no different from anyone else in this regard and also registers any symbols on its LHS: deleting list[1] does not allow its symbols to get automatically released, as it had for list[2]:

list[1] =.
Names["Global`$myTemp*"]

(* -> {"$myTemp3","$myTemp4","$myTemp5","$myTemp6","$myTemp7"}*)

Is there some magic spot that one might hide such values such that the symbols and the values still get cleaned when list[1] is deleted?

I have looked around SE and seen e.g. 'Weak hash maps' and Language`NewExpressionStore which might somehow be solutions, but nothing obvious seemed to pop up.

In fact OwnValues does already operate in this requested fashion. Just because list[1] contains multiple copies of a symbol with OwnValues doesn't mean that the memory footprint is increased for each repetition ('laziness') - not until some sort of 'final' evaluation (like printing or saving) do the OwnValues of the symbols get inserted and expanded. So the memory storage of the values of the symbols is not the issue. It is more an issue of how a list of symbols with massive OwnValues can be saved or displayed without hassle.

---

In response to @b3m21a's very helpful answer I have made an updated version of my OP using NewExpressionStore.

It seems to me that NewExpressionStore is also incapable of providing the capabilities I desire: separation of symbols from values and the automatic deletion of symbols and values when the (Temporary) symbols are no longer referenced:

As before, I set up Temporary symbols, creating 5 of them inside list[1]:

$HistoryLength = 0;
Names["Global`$myTemp*"]

ClearAll[CreateTemporarySymbol]

CreateTemporarySymbol[] := With[{p = Module[{pp = Unique["$myTemp", Temporary]}, pp]}, p]

list[1] = Table[CreateTemporarySymbol[], 5]

-> {$myTemp3, $myTemp4, $myTemp5, $myTemp6, $myTemp7}

Names["Global`$myTemp*"]

-> {"$myTemp3", "$myTemp4", "$myTemp5", "$myTemp6", "$myTemp7"}

Modifying b3m21a's answer, I create hiddenCache which is a NewExpressionStore from the built-in Language` context:

hiddenCache = Language`NewExpressionStore["someLabel"]

... and 'put' a matrix directly into it:

hiddenCache["put"[HiddenValues, $myTemp3, {{7, 3}, {4, 9}}]]

The matrix is stored in the hiddenCache, associated with the (Temporary) $myTemp3 symbol:

hiddenCache["get"[HiddenValues, $myTemp3]]

-> {{7, 3}, {4, 9}}

... but since the matrix is not stored in the OwnValues of $myTemp3, it doesn't get expressed when we view list[1] (the matrix is also untouchable by Save or Cases or any other function that might operate on list[1]). This achieves my first objective of separating the values from the symbols:

$myTemp3
list[1]

-> $myTemp3

-> {$myTemp3, $myTemp4, $myTemp5, $myTemp6, $myTemp7}

Of course all of the $myTemp vars are in the namespace:

Names["Global`$myTemp*"]

-> {"$myTemp3", "$myTemp4", "$myTemp5", "$myTemp6", "$myTemp7"}

...But because they are Temporary, and because list[1] contains their only reference, they get automatically cleaned up when we delete list[1], except for $myTemp3 which is now unfortunately bound somewhere in the hiddenCache:

list[1] =.
Names["Global`$myTemp*"]

-> {"$myTemp3"}

The values are still being stored in hiddenCache, associated with $myTemp3, and binding to it so that it couldn't be automatically cleaned up. This fails my second objective of having the values being automatically removed at the same time as the symbols are:

hiddenCache["listTable"[]]

-> {{HiddenValues, {{$myTemp3, {{7, 3}, {4, 9}}}}}}

The only way to get $myTemp3 to be removed from the namespace is for HiddenValues to be 'removed':

hiddenCache["remove"[HiddenValues]]

Now the values that had been in hiddenCache are gone:

hiddenCache["listTable"[]]

->  {}

... and now $myTemp is truly cleaned up:

Names["Global`$myTemp*"]

-> {}

However this fails my desire of not having to do my own garbage collection.

Answer 1

Maybe OwnValues can be abused for that.

$HistoryLength = 0;
CreateTemporarySymbol[] := Module[{p = Unique["$myTemp", Temporary]}, p]
AppendToOwnValues[symbol_, key_, value_] := AppendTo[OwnValues[symbol], key :> value]
ExtractOwnValue[symbol_, key_] := key /. ReleaseHold[OwnValues[symbol]]

Creating temporary symbols and storing values in them:

list[1] = Table[CreateTemporarySymbol[], 5];
MapThread[AppendToOwnValues, {list[1], ConstantArray["Key", 5], RandomReal[{1, 11}, {5, 5}]}];

Retrieving a value:

ExtractOwnValue[list[1][[1]], "Key"]

{9.58016, 9.22525, 3.86703, 5.60385, 4.23212}

Deleting the list list[1] and checking that the symbols are really erased:

Names["Global`$myTemp*"]
list[1] =.
Names["Global`$myTemp*"]

{"$myTemp11", "$myTemp12", "$myTemp13", "$myTemp14", "$myTemp15"}

{}

However, I am pretty sure that OwnValues is not meant for that.

Answer 2

Here's an example of how you can use ExpressionStore for this:

tempCache = Language`NewExpressionStore["ilmbiahi"];

bindTempSyms[holder_Symbol, key_, syms_] :=
  tempCache@"put"[holder, key, syms];
getTempSyms[holder_Symbol, key_] :=
  tempCache@"get"[holder, key];

Module[{list},

 Module[{sym1, sym2, sym3},
  bindTempSyms[list, 1, {Hold[sym1], Hold[sym2], Hold[sym3]}];
  sym1 = "dog";
  sym2 = "cat";
  sym3 = "my shattered hopes and dreams";
  ];

 getTempSyms[list, 1][[-1]] // ReleaseHold
 ]

"my shattered hopes and dreams"

Names["sym*"]

{"sym1", "sym1$", "sym2", "sym2$", "sym3", "sym3$", "syms"}

Note that the Module-ized versions of these symbols don't appear.

What's nice about this is whenever list went out of scope—even if it were returned from the Module (and if list wouldn't have been bound to Out)—those bound symbols would go out of scope too.

On the other hand, you indicate that your biggest issue now is in display? I'm not sure exactly what you mean there, but there are many ways to display big values compactly. The standard way is through BoxForm`ArrangeSummaryBox. I won't get into how it works, but there are many, many usages of it across the site. Here's how it can work with a general data structure as implemented here:

RegisterDataStructure[
  pkg`myObj,
  pkg`myObj[Hold[sym_Symbol]]
  ];
pkg`myObj[] := Module[{tmp}, pkg`myObj@Hold[tmp]]

po = pkg`myObj[]

Here's how you could set the data on such an object:

pkg`setObjData[po, RandomReal[{}, {100, 100}]]

Note that the object data has changed:

pkg`myObj`Data@po // ReleaseHold // ByteCount

80152

This can be chained with ExpressionStore to make some very powerful encapsulation strategies.

Answer 3

You could use strings containing the (full) symbol names as keys. This function may come in handy:

f = Function[s,
   Context[s] <> SymbolName[Unevaluated[s]],
   HoldFirst
   ];

Examples:

a = 1;
bla`a = 2;
bla`bli`blubb`a = 2;
f[a]
f[bla`a]
f[bla`bli`blubb`a]

"Global`a"

"bla`a"

"bla`bli`blubb`a"

Answer 4

If it is OK to use OwnValues and you avoid them just to prevent large expressions from appearing as output, then you can wrap the temporary symbols you create in Hold and release it when necessary.

For example,

$HistoryLength = 0;
Names["temp*"]
(*{}*)

sym = Module[
 {temp},
 temp = 1;
 Hold[temp]
]

Names["temp*"]
(*{"temp", "temp$2465"}*)

sym =.
Names["temp*"]
(*{temp}*)

EDIT

By the way, I have once faced myself a task of garbage collecting Temporary symbols with DownValues. Unfortunately, I don't have time to write a full solution at the moment, but the main idea is simple: you introduce an auxiliary Temporary symbol to count references to your main symbol (which now can be not Temporary):

ClearAll[reference, new]
new[] := Module[
  {ref, store},
  reference[store] ^= Context[ref] <> SymbolName[ref];
  reference[ref, store]
]

Now you can attach any definitions to the generated store$123 symbol and use reference[ref$123, store$123] objects in your code to access them. If no reference[ref$123, store$123] objects are referenced any more, then ref$123 will be released, which you can check with Names@reference@store$123. Of course, this approach would require to call garbage collection manually from time to time, but the benefit of calling garbage collection this way is that you can implement a finilizer which will be executed before releasing store$123 symbol.