3
$\begingroup$

If I have a dataset ds and perform an operation such as ds[All, f] does f get applied in the order the rows exist (the order they were placed into the dataset or as the result of a SortBy clause) or is there no guarantee of such an order?

If there is no guarantee using this form is there another way of controlling the order f processes each row?

I can't find anything explicit in the documentation concerning this. I'm not sure in my current work out of expected order execution is happening but it would explain some of the odd results I'm getting.

$\endgroup$
2
$\begingroup$

Let's try it:

ds = Dataset[AssociationThread[{"number", "letter"} -> #] & /@ Transpose[{
      ConstantArray[0, 10],
      CharacterRange["a", "j"]
      }]];

i = 0;
ds[All, f[i++, #letter] &] // Normal

{f[0, "a"], f[1, "b"], f[2, "c"], f[3, "d"], f[4, "e"], f[5, "f"],
f[6, "g"], f[7, "h"], f[8, "i"], f[9, "j"]}

So it does apply the function to the rows in order. You could have asked the same question about Map. Does it apply the function in order? The reason it's not in the documentation I suppose is because the definition of Map is that f /@ {a,b,c} turns into {f[a], f[b], f[c]}. The reason why the functions get evaluated in order has nothing to do with Map itself, it's just because that's how the evaluator work. You can try for example

i = 0;
{i++, i++, i++, i++}

{0, 1, 2, 3}

to see that this is the case. I suspect the reason here is similar, the documentation writers may have been thinking about ds[All, f] as f /@ Normal@ds and therefore did not think it necessary to write about it, since it's really a detail about how the evaluator works.

The answers to this question reminds me that what I've done here is done in other situations as well. The answers that I linked to use assumptions about the order in which DeleteDuplicates handles elements and the order in which GatherBy collects elements. This is not in the documentation either, but through tests it has been found out to be so.

TL;DR: Dataset does traverse the rows from the top.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.