0
$\begingroup$

I have struck a combination of the above-mentioned functions that causes evaluation to fail. I wonder whether someone can help me create a more minimal example in order to understand the issue better.

Assuming the definitions:

dataset1 = Dataset@<|"a" -> 1, "b" -> 2|>;
dataset2 = Dataset@<|"c" -> 3, "d" -> 4|>;
datasets = {dataset1, dataset2};
assoc1 = <|"aa" -> 8, "cc" -> 9|>;
test1[in1_] := in1
test2[in1_, in2_] := {in1, in2}

We can pass in that list of datasets to a Module that passes them, one by one, into another Module that calls a function on it:

Map[Function[localDataset, Module[{},
    Normal@Map[Function[localDatasetEntries, test1[localDatasetEntries]], localDataset]
  ]], datasets]
(* {<|"a" -> 1, "b" -> 2|>, <|"c" -> 3, "d" -> 4|>} *)

Upon switching the interior function to one that also uses the globally-defined association, it no longer works:

Map[Function[localDataset, Module[{},
   Normal@Map[Function[localDatasetEntries,test2[localDatasetEntries, assoc1]], localDataset]
   ]], datasets]
(* MapAt::partw: Part {All,2} of <|aa->8,cc->9|> does not exist. 
MapAt::partw: Part {All,2} of <|aa->8,cc->9|> does not exist.
{$Failed, $Failed} *)

although care should be taken because once the above statements have been passed once to the kernel, the evaluation does work. Assuming a new kernel, running the above statement twice gives:

Map[Function[localDataset, Module[{},
   Normal@Map[Function[localDatasetEntries, test2[localDatasetEntries, assoc1]], localDataset]
   ]], datasets]
Map[Function[localDataset, Module[{},
   Normal@Map[Function[localDatasetEntries, test2[localDatasetEntries, assoc1]], localDataset]
   ]], datasets]
(* MapAt::partw: Part {All,2} of <|aa->8,cc->9|> does not exist.
MapAt::partw: Part {All,2} of <|aa->8,cc->9|> does not exist.
{$Failed, $Failed}
{<|"a" -> {1, <|"aa" -> 8, "cc" -> 9|>}, "b" -> {2, <|"aa" -> 8, "cc" -> 9|>}|>, <|"c" -> {3, <|"aa" -> 8, "cc" -> 9|>}, "d" -> {4, <|"aa" -> 8, "cc" -> 9|>}|>} *)

Further, assuming a new kernel each time, certain ways to insert Normal[] into the expression solve the problem; others don't. This one works:

Map[Function[localDataset, Module[{},
    Normal@Map[Function[localDatasetEntries, test2[localDatasetEntries, assoc1]], Normal@localDataset]
   ]], datasets]
(* {{"a" -> {1, <|"aa" -> 8, "cc" -> 9|>}, "b" -> {2, <|"aa" -> 8, "cc" -> 9|>}}, {"c" -> {3, <|"aa" -> 8, "cc" -> 9|>}, "d" -> {4, <|"aa" -> 8, "cc" -> 9|>}}} *)

and this one does not:

Map[Function[localDataset, Module[{},
    Normal@Map[Function[localDatasetEntries,test2[Normal@localDatasetEntries, assoc1]], localDataset]
   ]], datasets]
(* MapAt::partw: Part {All,2} of <|aa->8,cc->9|> does not exist. 
MapAt::partw: Part {All,2} of <|aa->8,cc->9|> does not exist.
{$Failed, $Failed} *)

My particular issue is that I have partitioned a rather large dataset into smaller groups with some shared values, and I would like to operate one by one on the entries in each subgroup, using a list of globally-defined associations as parameters.

My question is, why is this happening? Why do some computations with structured datasets fail while their "normal" association forms work fine? Should we expect to generally encounter this type of issue in certain particular cases? Is there a general fix?

$\endgroup$
  • 2
    $\begingroup$ I don't fully understand your problem, but I think you suffer from some misunderstanding about what Module and especially Print do: Print will print its argument to the notebook but return Null, and most errors you see are due to that. Module with an empty list of local variables is a noop, so you could just remove it without any change in what the code does... $\endgroup$ – Albert Retey Mar 27 '17 at 10:46
  • $\begingroup$ Removing Print[] and Module[] has no effect. I removed them above for clarity. (Also removed a typo semicolon that was causing a Null output, thanks.) $\endgroup$ – ninemileskid Mar 27 '17 at 14:21
  • $\begingroup$ So what is your goal: do you want to append assoc1 to both datasets as shown as a list? $\endgroup$ – gwr Mar 27 '17 at 14:30
  • $\begingroup$ In my actual code project, the nested Modules are serving a purpose, localizing the scope of other calculations. Removing them entirely makes everything work, but my question is about why they won't work in this case. $\endgroup$ – ninemileskid Mar 27 '17 at 14:30
  • $\begingroup$ @gwr Yes, I'd like to operate locally on the entries of each dataset - appending the association to the existing dataset is the most basic example I could think of. $\endgroup$ – ninemileskid Mar 27 '17 at 14:33
0
$\begingroup$

I am not quite sure why this is happening, but a quick solution is to wrap Association around assoc1. So this (using the OP's) example will work out without any error messages:

Map[ 
    Function[ localDataset, Module[ {}, 
       Normal @ Map[  
           Function[ localDatasetEntries, 
               test2[ localDatasetEntries, Association[assoc1] ] 
           ], 
           localDataset
       ]
      ]
    ], 
    datasets
]

{<|"a" -> {1, <|"aa" -> 8, "cc" -> 9|>}, "b" -> {2, <|"aa" -> 8, "cc" -> 9|>}|>, <|"c" -> {3, <|"aa" -> 8, "cc" -> 9|>}, "d" -> {4, <|"aa" -> 8, "cc" -> 9|>}|>}

But those Module, Map, Function and Normal concoctions look like the "most ugly code" I have seen in a long time :) So note, that something in this line will work (and may also be localized imo):

With[
    {assoc = assoc1},
    Dataset@datasets // Query[ All, All, List[#, Association[assoc]] & ]
]

datasets

$\endgroup$
  • $\begingroup$ @ninemileskid Does this help? $\endgroup$ – gwr Mar 27 '17 at 15:27
  • $\begingroup$ Yes, it does. Using Association[assoc1] is slightly slower than using the version with Normal in my original post, despite it preserving the Dataset heads - but it does work. It seems the Query solution sometimes takes an excessively long time to finish, although other times it is just as fast as the others. (It seems to do this most often upon a freshly started kernel.) $\endgroup$ – ninemileskid Mar 27 '17 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.