# Nondeterministic $Failed when mapping over a Dataset Consider the following simplified example (on Mathematica 12): ds = Dataset@{<|"hell" -> "o"|>, <|"hell" -> "no"|>};  Map[ds&,ds]  This produces the following errors: During evaluation of In[2]:= MapAt::partw: Part {All,2} of <|hell->o|> does not exist. During evaluation of In[2]:= MapAt::partw: Part {All,2} of <|hell->no|> does not exist. During evaluation of In[2]:= MapAt::partw: Part {All,2} of <|ID-><someid>|> does not exist. During evaluation of In[2]:= General::stop: Further output of MapAt::partw will be suppressed during this calculation. Out[2]=$Failed


(edited out the DS id out of the output)

Evaluating the second cell a second time produces the expected (pasted as inputform)

Dataset[{Dataset[{<|"hell" -> "o"|>, <|"hell" -> "no"|>}, TypeSystemVector[TypeSystemStruct[{"hell"}, {TypeSystemAtom[String]}], 2],
<|"ID" -> <ID1>|>], Dataset[{<|"hell" -> "o"|>, <|"hell" -> "no"|>}, TypeSystemVector[TypeSystemStruct[{"hell"}, {TypeSystemAtom[String]}],
2], <|"ID" -> <ID1>|>]}, TypeSystemVector[TypeSystemAnyType, 2],
<|"Origin" -> HoldComplete[Map, testds2 & , DatasetDatasetHandle[<ID1>]], "ID" -> <ID2>|>]


If this seems like a contrived example, my current use case is mapping a function that takes the entire dataset and a single row and produces a string.

## Observations

• The error appears to manifest when the function being mapped accesses some dataset. e.g. The following does not produce an error:
Map[Echo,ds]

• A similar error and nondeterministic evaluation occurs by using Query[All, ds&] instead Map produces a similar messages (also only on the first run) but does produce the expected result. Moreover, the Query form and Map form do not appear to affect one another, i.e. running Query twice does not prime the Map form, which still produces messages and Failure when first running.

• I thought there might be an issue with referencing the DS while mapping on it, but the following also fails. Note that the messages in this case refer to the ds2 Dataset, i.e. the one in the function (corrected from previous version where the opposite was claimed):

ds = Dataset@{<|"hell" -> "o"|>, <|"hell" -> "no"|>};
ds2 = Dataset@{<|"hell" -> "yes"|>, <|"hell" -> "no"|>};
Map[ds2&,ds]

• Making any changes to the function makes the errors reappear [observation from comment by @lukas-lang], even something as trivial as (1; ds)&.

• Once a function has been cached, mapping it over other datasets works fine.

• If the dataset does not appear directly in the function argument, the problem is avoided (from comment by @lukas-lang)

## Workarounds

• If the mapped over dataset is wrapped with Normal, the map works as expected.
• Making the dataset not appear lexically in the function expression, e.g. by using Downvalues. e.g. temp[] := ds; Map[temp[]&, ds] works (from comment by @lukas-lang). Edit: Correction - simple SetDelayed won't work, only Downvalues,

## Prior art

I found a SO question that seems to have a related issue (in particular, similar messages produced and similar nondeterminism). However, I think that the question was posed in a manner that obstructs the true issue and the proposed solution does not appear applicable here.

• Looks like a bug in the type system of Dataset - every time you change the function in any way, the error appears for one evaluation, indicating that the result is being cached. Another workaround is to hide the structure of the function by moving the definition into a symbol, e.g. myfunc[x_]:=ds followed by Map[myfunc,ds] works without issues (The reason is that the type deduction system does not peek into the down-values of symbols) – Lukas Lang Oct 25 '19 at 21:11
• Thanks for the workaround! It is nice in that this preserves the Dataset Head as well as other heads accepted by Query. Although I am beginning to be somewhat skeptical as to whether I should continue using Dataset in the first place... – Shwouchk Oct 25 '19 at 21:28
• – WReach Oct 26 '19 at 14:13

Analysis current for Mathematica 12.0.0

Summary

The problem is due to a particular limitation in the Dataset type inferencer in which it cannot determine the type of an operator that contains an association object (as opposed to an association constructor).

The problem occurs only the first time because the inferred type information is stored in the TypeApplyCache. Even though the type inferencer issues error messages, it still places a usable entry into that cache. In subsequent evaluations, this cached type is used so the messages do not appear again. The cache can be cleared using TypeSystemResetTypeApplyCache[].

Workarounds

We can work around this problem by masking the value of ds in the operator so that the type inferencer does not see it. For example:

Map[ds&[]&, ds]


Alternatively, we can extract the data from the dataset and use Query which does not perform type inferencing at all:

ds // Normal // Query[Dataset, ds&]


Details

When a Dataset is queried, the system attempts to infer the resulting data type prior to actually evaluating the query. It does this by inspecting both the query operators and the initial data value. Some discussion of type inferencing (and type deduction) can be found in (87479).

The type inferencer uses many heuristics and cannot handle all data types. In this particular case, as the inferencer inspects the query operator it encounters the association objects within the dataset referenced by ds. But the implementation can only handle constructors. The distinction between association constructors and objects is discussed in (204296).

The inferencer has no problem with an operator that contains an assocation constructor:

Needs["TypeSystem"]

ResetTypeApplyCache[]
TypeApply[<|"a" -> 1|> &, {}]

(* Struct[{"a"}, {Atom[Integer]}] *)


... but it fails when it encounters an association object within the operator:

ResetTypeApplyCache[]
With[{a = <|"a"->1|>}, TypeApply[a&, {}]]

(* >> During evaluation of In[34]:= MapAt::partw:
Part {All,2} of <|a->1|> does not exist.

Struct[MapAt[TypeSystemInferencePackagePrivateexprType,
Unevaluated[<|a->1|>],{All,2}]]
*)


The actual bug lies in the internal function TypeSystemInferencePackagePrivateexprType:

TypeSystemInferencePackagePrivateexprType[<|"a" -> 1|>]
(* Struct[{"a"}, {Atom[Integer]}] *)

With[{a = <|"a" -> 1|>},  TypeSystemInferencePackagePrivateexprType[a]]

(* During evaluation of In[40]:= MapAt::partw:
Part {All,2} of <|a->1|> does not exist.

Struct[MapAt[TypeSystemInferencePackagePrivateexprType,
Unevaluated[<|a->1|>],{All,2}]]
*)


We can use traceTypes from (89081) to see how all of this fits into the original query:

traceTypes[ds][All, ds &]


Notice on the first line how the Map[ds&] operator has the usable inferred type of Vector[UnknownType, 2] despite the error messages being displayed. Also notice how the argument types in the last line contain gibberish due to the MapAt failure.

• Thanks for the detailed answer! – Shwouchk Oct 28 '19 at 13:07
• I thought some more about the answer and got curious about whether it is a feature just of associations. Turns out that ds = Dataset[{1, 2, 3, 4}]; Map[ds &, ds] exhibits a similar failure, so this doesn't appear to be limited to associations. – Shwouchk Oct 28 '19 at 13:19
• It is actually the same problem, except this time the first association it hits is the association that holds the dataset's metadata instead of an association within the data proper. You can see this metadata association in Dataset[{1, 2, 3, 4}] // InputForm. – WReach Oct 28 '19 at 14:09
• Very interesting! I wouldn't expect it to get tripped by the metadata as well... – Shwouchk Oct 28 '19 at 14:17
• I think the whole notion of type inference in WL is a bit of folly. I am not sure that it is possible even in principle to accurately predetermine the result type of an arbitrary WL evaluation. Consider: up-values, held expressions, dynamic definitions, domain-specific sublanguages, atoms/semi-atoms/non-atoms... the inferencer's heuristic approach certainly has its work cut out for it. It is actually quite remarkable that it works as well as it does. For my part, I prefer Query and only use Dataset as a tail-end visualization wrapper (like MatrixForm). – WReach Oct 28 '19 at 18:07