21
$\begingroup$

Why Dataset doesn't serialize its contents, isn't it the whole point? I expect it to behave similar to python's pandas.DataFrame, R's data.frame and similar tabular data abstractions in other languages. Why on earth importing 100 Mb csv file with Wolfram results in 1 Gb Dataset expression in memory, which in turn results in terrible Data Science and Machine Learning experience for everyone. Is this issue being addressed somehow?

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ Minor remark, not related to the main issue in the question: I would say Dataset objects are more like R's data.table, not so much as data.frame. Anyone willing to share an opinion on this? $\endgroup$
    – Anton Antonov
    Commented Mar 14, 2018 at 15:39
  • 2
    $\begingroup$ @AntonAntonov You're right, in terms of usage their respective DSLs are indeed somewhat similar. $\endgroup$
    – swish
    Commented Mar 14, 2018 at 15:43
  • 7
    $\begingroup$ I think this is the kind of thing that should be forwarded to WRI support. It’s not bug per se but it does show a lack of performance WRI should be reminded of. $\endgroup$
    – b3m2a1
    Commented Mar 14, 2018 at 16:19
  • 7
    $\begingroup$ The Streaming project was supposed to address that. However, it's been stalled for some time, unfortunately. My development time is currently dedicated to a different project. But the more users request this functionality, the better are the chances that we will get needed dedicated time to bring this project to production. $\endgroup$
    – Leonid Shifrin
    Commented Mar 14, 2018 at 20:16
  • 8
    $\begingroup$ @LeonidShifrin I honestly rather have a release dedicated to performance enhancements and bug fixes than plethora of experimental features. It's nice and all to see all that interesting functionality coming up, as I like playing with new stuff, but it feels more and more bulky at an expense of smooth experience. $\endgroup$
    – swish
    Commented Mar 14, 2018 at 20:39

1 Answer 1

Reset to default
9
$\begingroup$

You may use ResourceFunction TableSet from the Wolfram Function Repository for tabular data.

Dataset is optimized for hierarchical data with varying structure per row/node and has some overhead to manage that.

TableSet is for tabular data so has a much smaller footprint.

enter image description here

enter image description here

TableSet is also compatible with SQL-like functions such as Query, Select, and others.

As the contributor of the function any feedback on its utility is welcome.

Hope this helps.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I've seen that function posted recently. Waiting for an opportunity to use it :) $\endgroup$
    – swish
    Commented Aug 17, 2021 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.