18
$\begingroup$

Why Dataset doesn't serialize its contents, isn't it the whole point? I expect it to behave similar to python's pandas.DataFrame, R's data.frame and similar tabular data abstractions in other languages. Why on earth importing 100 Mb csv file with Wolfram results in 1 Gb Dataset expression in memory, which in turn results in terrible Data Science and Machine Learning experience for everyone. Is this issue being addressed somehow?

$\endgroup$
  • 3
    $\begingroup$ Minor remark, not related to the main issue in the question: I would say Dataset objects are more like R's data.table, not so much as data.frame. Anyone willing to share an opinion on this? $\endgroup$ – Anton Antonov Mar 14 '18 at 15:39
  • 1
    $\begingroup$ @AntonAntonov You're right, in terms of usage their respective DSLs are indeed somewhat similar. $\endgroup$ – swish Mar 14 '18 at 15:43
  • 6
    $\begingroup$ I think this is the kind of thing that should be forwarded to WRI support. It’s not bug per se but it does show a lack of performance WRI should be reminded of. $\endgroup$ – b3m2a1 Mar 14 '18 at 16:19
  • 5
    $\begingroup$ The Streaming project was supposed to address that. However, it's been stalled for some time, unfortunately. My development time is currently dedicated to a different project. But the more users request this functionality, the better are the chances that we will get needed dedicated time to bring this project to production. $\endgroup$ – Leonid Shifrin Mar 14 '18 at 20:16
  • 5
    $\begingroup$ @LeonidShifrin I honestly rather have a release dedicated to performance enhancements and bug fixes than plethora of experimental features. It's nice and all to see all that interesting functionality coming up, as I like playing with new stuff, but it feels more and more bulky at an expense of smooth experience. $\endgroup$ – swish Mar 14 '18 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.