Timeline for Inefficient memory storage for Dataset
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 17, 2021 at 11:47 | answer | added | Edmund | timeline score: 9 | |
| Mar 14, 2018 at 22:57 | comment | added | Hans | @swish Does your CSV file contain header(s). Have you tried constructing this dataset without the header(s)? Also tangential does any one use external SQL database as a data store for WL? | |
| Mar 14, 2018 at 22:56 | comment | added | Leonid Shifrin |
A really big problem with Dataset is that it does not have a formal query language, which would be restricted to operations which one could at least hope to implement efficiently. Right now, it is more like a nice formatting wrapper around hierarchical data, plus the rather nice Query machinery. But the main point is, it essentially allows one to use any Mathematica function on that data. And implementing out-of-core versions for the entire variety of such operations is simply not feasible. It might be easier to define / implement a different data-frame-like object altogether.
|
|
| Mar 14, 2018 at 20:39 | comment | added | swish | @LeonidShifrin I honestly rather have a release dedicated to performance enhancements and bug fixes than plethora of experimental features. It's nice and all to see all that interesting functionality coming up, as I like playing with new stuff, but it feels more and more bulky at an expense of smooth experience. | |
| Mar 14, 2018 at 20:16 | comment | added | Leonid Shifrin | The Streaming project was supposed to address that. However, it's been stalled for some time, unfortunately. My development time is currently dedicated to a different project. But the more users request this functionality, the better are the chances that we will get needed dedicated time to bring this project to production. | |
| Mar 14, 2018 at 16:50 | history | tweeted | twitter.com/StackMma/status/973964608850587648 | ||
| Mar 14, 2018 at 16:19 | comment | added | b3m2a1 | I think this is the kind of thing that should be forwarded to WRI support. It’s not bug per se but it does show a lack of performance WRI should be reminded of. | |
| Mar 14, 2018 at 15:55 | comment | added | Anton Antonov | Thanks for confirming... :) | |
| Mar 14, 2018 at 15:43 | comment | added | swish | @AntonAntonov You're right, in terms of usage their respective DSLs are indeed somewhat similar. | |
| Mar 14, 2018 at 15:39 | comment | added | Anton Antonov |
Minor remark, not related to the main issue in the question: I would say Dataset objects are more like R's data.table, not so much as data.frame. Anyone willing to share an opinion on this?
|
|
| Mar 14, 2018 at 15:01 | history | asked | swish | CC BY-SA 3.0 |