# How to disallow the application of functions outside a white list?

In an earlier post I described a special kind of list (actually, a special kind of multiset) that can be represented more efficiently with a structure like this:

dd = discreteData[<|"scale" -> 1.234,
"bias"  -> 5.678,
"tally" -> {{-5, 2},
{-4, 251},
{-3, 5941},
{-2, 60383},
{-1, 241185},
{ 0, 383613},
{ 1, 241644},
{ 2, 61035},
{ 3, 5686},
{ 4, 259},
{ 5, 1}}|>];


(Basically, the values encoded by this data structure are all of the form $kS + B$, where $k$ is an integer, and $S$ and $B$ are the numbers that the data structure labels "scale" and "bias". The "tally" item is a list of pairs $\{k, n\}$, where $n$ is the number of times that the value $kS + B$ appears in the data.)

Now, let

dd2list[dd_] := dd[[1, "bias"]] + dd[[1, "scale"]] Flatten[ ConstantArray @@@ dd[[1, "tally"]] ];


Since any discreteData object dd contains the same information as the corresponding List object dd2list[dd], it would be great if one could arrange matters such that dd and dd2list[dd] could be used interchangeably in Mathematica programs.

By defining suitable upvalues, one can have dd behave like dd2list[dd] in the specified context. E.g.

discreteData /: Length[ d_discreteData ] := With[
{ assoc = d[] },
Plus @@ ( Last /@ assoc[["tally"]] )
];

Length[dd] == Length[dd2list[dd]]
(* True *)


Unfortunately, this one upvalue covers only a tiny fraction of the expressions one is likely to find dd in. If one mistakenly replaces a list object with its discreteData counterpart in an expression for which no proper upvalue has been defined, the results could change dramatically (and for the worse, of course).

For example, if dd is defined as shown above, the evaluation of Sqrt /@ dd succeeds without warning, and the result even has the correct "shape" (i.e. the right head, the right fields, with values also correctly dimensioned), but it is grossly incorrect.

Of course, one can avoid this one error by defining a suitable upvalue for Map, but in order to make this data structure resistant to such silent errors in general, one would have to implement upvalues for a huge number of list-consuming functions. This does not seem practical to me.

In order to make such a data structure behave like a List safely, one needs at least one of

1. the specification of a complete list interface;

2. a way to make it a visible error to apply to objects of the new class any function that is not included in a white-list of approved functions.

(NB: the white-list mentioned in 2 consists of precisely those functions for which I have defined upvalues.)

Please correct me if I'm wrong, but, AFAIK, Mathematica has nothing like 1.

This leaves 2. Is there a way to implement this?

Clarification: option 2 will not solve the problem of making discreteData behave like a list in any appropriate context; it only solves the problem of silent errors.

• You certainly can do the second, again, using UpValues but I think your whitelist will be necessarily huge. Perhaps even more than the list interface. – b3m2a1 Mar 11 '17 at 22:37
• @MB1965: The whitelist I had in mind consists of precisely those functions for which I have defined upvalues. It will not allow my object to behave like a true list in every possible situation, but at list it will prevent silent errors. – kjo Mar 12 '17 at 0:01
• You'll want things like Message and Hold and HoldComplete and all these things that we don't usually think of as "functions" though, too. There are just so many of them and I think you might find the system emitting more messages that you really want to deal with. – b3m2a1 Mar 12 '17 at 0:03
• @MB1965: I see your point. I guess I see no way to use the upvalue idea to implement this discreteData objects safely. I think the silent errors I illustrated represent to big a risk. – kjo Mar 12 '17 at 0:12
• Might be worth implementing a blacklist instead. What sort of functions are you most concerned about? For instance it seems Listable functions are most apt to cause you trouble. – b3m2a1 Mar 12 '17 at 0:22

## 3 Answers

Since, to the best of my knowledge, is not possible to program a custom interface that is integrated at the level of atomic objects such as a SparseArray I think some compromise is unavoidable.

Edmund's method is the first one that came to mind, and I voted for it. It may still be your best option.

However it is not robust in the manner I think you are seeking. Consider for example:

expr = Hold[{1, discreteData[x, y, {z1, z2, z3}], 3}];

Apply[foo, expr, {2}]
Apply[foo, expr, {3}]
Map[foo, expr, {3}]

Hold[{1, foo[x, y, {z1, z2, z3}], 3}]

Hold[{1, discreteData[x, y, foo[z1, z2, z3]], 3}]

Hold[{1, discreteData[foo[x], foo[y], foo[{z1, z2, z3}]], 3}]


I imagine that none of these are the result you would hope to see from each operation.

Perhaps it would be better for these functions to ignore your construct rather than mutating it errantly.

# Proposal

One compromise to achieve this is to make a separate definition for each Symbol to which a data set is assigned.

• To do this I shall use a public function setDiscreteData and a global variable $DD • All your UpSets should be given in the body of setDiscreteData Attributes[setDiscreteData] = HoldFirst; setDiscreteData[s_Symbol, rhs_] := With[{a :=$DD[s]},

a = rhs;

s=. ^:= (ClearAll[s]; a=.);

Set[s, new_] ^:= a = new;

Length[s]    ^:= a // Query["tally", Tr, Last];

]


Now you would define a data set like this:

setDiscreteData[dd,
<|"scale" -> 1.234, "bias" -> 5.678,
"tally" -> {{-5, 2}, {-4, 251}, {-3, 5941}, {-2, 60383}, {-1, 241185}, {0,
383613}, {1, 241644}, {2, 61035}, {3, 5686}, {4, 259}, {5, 1}}|>
];


The defined rules apply:

Length[dd]

1000000


Every other operation sees dd as an atomic Symbol and handles it as such:

expr = Hold[{1, dd, 3}];

Apply[foo, expr, {2}]
Apply[foo, expr, {3}]
Map[foo, expr, {3}]

Hold[{1, dd, 3}]

Hold[{1, dd, 3}]

Hold[{1, dd, 3}]

Replace[dd, x_ :> foo, 1]

dd + 8

dd

8 + dd


Once a data set Symbol (dd) has been assigned it can be reassigned with Set:

dd =
<|"scale" -> 1.234, "bias" -> 5.678, "tally" -> {{-1, 2}, {0, 7}, {1, 4}}|>;

Length[dd]

13


The Symbol may be reset to normal (all definitions cleared) using Unset:

dd=.


This method is incidentally related to How to Set parts of indexed lists?

• It is not clear to me what is the role of $DD. May you explain. Also, how are you able to reassign a in the With? I thought that the With variables are replaced in the body so this was not possible. Both With[{a = 1}, a = 2] and With[{a := 1}, a = 2] produce error Set::setraw: Cannot assign to raw object 1. for 11.0.1 on Windows. May you explain this as well. – Edmund Mar 12 '17 at 14:13 • @Edmund $DD is my DownValues look-up table. Every data set assignment will be made as e.g. \$DD[foo] = . . . . I am using an undocumented but longstanding syntax for With; please see (121173). Your example is not quite the same; try instead: x = 1; With[{a := x}, a = 2]; x – Mr.Wizard Mar 12 '17 at 14:16
• @Edmund Did I write that in an understandable way or should I try again? – Mr.Wizard Mar 12 '17 at 14:32
• It completely understandable. Thanks. – Edmund Mar 12 '17 at 14:35
• @Mr.Wizard Done. – jkuczm Mar 13 '17 at 22:26

Here's a sample implementation of 2 although I think simply writing a generalizable list-like interface is probably the safer route:

whitelisted =
Association@
Thread[
Thread@
Hold@
{
Hold, HoldPattern, Print
} ->
Null];

discreteData::unknown =
"Result of applying function  to discreteData object is unknown";
discreteData /:
HoldPattern[
func_?(Function[Null, ! KeyMemberQ[whitelisted, Hold[#]],
HoldAllComplete])[d_discreteData]
] :=
(
Message[discreteData::unknown, HoldForm[func]];
d
);
discreteData /:
HoldPattern[
func_?(Function[Null, ! KeyMemberQ[whitelisted, Hold[#]],
HoldAllComplete])[a___, d_discreteData, b___]
] :=
(
Message[discreteData::unknown, HoldForm[func]];
HoldComplete[func[a, d, b]]
);


Then:

In:= List[discreteData]

During evaluation of In:= discreteData::unknown: Result of applying function List to discreteData object is unknown

Out= discreteData


But

In:= Hold@discreteData

Out= Hold[discreteData]


And including Simon Wood's example now:

In:= discreteData + 1

During evaluation of In:= discreteData::unknown: Result of applying function Plus to discreteData object is unknown

Out= HoldComplete[1 + discreteData]

• You'll need to allow for multiple arguments, e.g. consider discreteData + 1 – Simon Woods Mar 11 '17 at 22:51
• Right, I'll toss that in. Although I think that should be a second case, because what the result should be there is less clear. – b3m2a1 Mar 11 '17 at 22:52

You may continue to use UpValues and construct a pattern that raises a Message when a function not in the white list is called.

First create a Message to raise.

discreteData::undefsym = "discreteData not defined for 1.";


UpValue pattern to raise the message when a non-white list function is used.

discreteData /:
f_[___, d_discreteData, ___] /;
MemberQ[{Normal, Format, Length, Part, Set, If, List}, f] == False :=
Message[discreteData::undefsym, f]


White list functions work as expected and others will not.

Sqrt /@ dd


discreteData::undefsym: discreteData not defined for Map.

Note that this will work because discreteData is sufficiently deep in Map and defined for Normal when it is entered as a parameter to Normal.

Normal /@ {dd, dd}
Join[{dd}, {dd}]


This will not work and errors on Sqrt since discreteData is not defined for Sqrt when it is entered as a parameter to Sqrt.

Sqrt /@ {dd, dd}
Join[dd, dd]


Hope this helps.

• I believe you can shorten MemberQ[{Normal, Format, Length, Part, Set, If, List}, f] == False to FreeQ[{Normal, Format, Length, Part, Set, If, List}, f] – Mr.Wizard Mar 12 '17 at 15:49