Transpose is faster when it distributes Keys at the 2nd level of a Dataset than when it factors them out.

 ds[len_] := <|"a" -> Range[len], "b" -> (1/Range[len] // N)|> // Dataset

(Using N to work around an unrelated bug - will post separately)

dst[len_] := ds[len][Transpose]


{ds[5], dst[5]}

enter image description here

Normal form:

{ds[5] // Normal, dst[5] // Normal} // Column

Timing study:

{10, 1000, 10000, 50000} // 
     "ds time" -> (ds[#] // Transpose // Timing // First), 
     "dst time" -> (dst[#] // Transpose // Timing // 
        First)|> &] // Dataset

enter image description here

Though timing is similar for smaller datasets, it grows to ~40x at 50k rows. By adding more rows this Timing ratio increases further; I've observed 100x in real world time series.

Note Transpose in dst constructor using SetDelayed contributes a small amount that doesn't affect the conclusion.

What explains this imbalance? If anything I expected that distributing Keys would take longer.

  • $\begingroup$ Related: (83838). Possibly related, at least in concept: (25641) $\endgroup$
    – Mr.Wizard
    May 27 '15 at 22:30
  • $\begingroup$ I'm not quite sure why yet, but replacing your Transpose with Transpose[#, AllowedHeads -> All]& and the timings are much closer... $\endgroup$
    – Stefan R
    Jun 2 '15 at 19:36
  • 1
    $\begingroup$ Also, the slowness seems to come from the code you use for the timing, i.e. I can leave your definitions of ds and dst unchanged but in the timing study I replace the transposes as I mentioned previously, and things are miraculously faster. $\endgroup$
    – Stefan R
    Jun 2 '15 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.