11
$\begingroup$

There's a performance bottleneck when using these types of functions. If I had a large CSV file and were to import it as a regular 'unstructured' CSV, I save almost half the time importing the file, and half the RAM.

I get it, Dataset is a cute little function that might do well for tiny datasets of 10 to 20 mbytes. But just imagine there are actually people out there that deal with really big files. Not toy sized files.

Here's what Dataset is doing. It amplifies the size of the data by a factor of 2 in my RAM as opposed to had I imported it as CSV. Additionally, it takes almost twice as long to import. Also, there doesn't seem to be an intrinsic speed advantage to using the Dataset structure when selecting or extracting elements from the 'structured list'.

Hence, I'm trying to find convincing arguments to use it but to my dismay have not. Can someone provide a list of good reasons to use Dataset functions for performance reasons ? In what context or situations does Dataset perform much better ...

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ It is certainly unfortunate that Dataset[] has trouble with "big data", since otherwise the ability to use database-like operations on it is a pretty good convenience. $\endgroup$
    – J. M.'s missing motivation ♦
    Commented Nov 13, 2019 at 6:54
  • $\begingroup$ There doesn't seem to any question in this post. $\endgroup$
    – m_goldberg
    Commented Nov 13, 2019 at 8:52
  • $\begingroup$ @m_goldberg I think there is. read it again, please. $\endgroup$
    – goodheart
    Commented Nov 13, 2019 at 9:04
  • 7
    $\begingroup$ To the closers: I think this is a totally valid question. $\endgroup$
    – Leonid Shifrin
    Commented Nov 13, 2019 at 11:12
  • 2
    $\begingroup$ Does this answer your question? Inefficient memory storage for Dataset $\endgroup$
    – Edmund
    Commented Aug 17, 2021 at 12:13

1 Answer 1

Reset to default
13
$\begingroup$

Why would you limit yourself to Database-like operations as J. M. nicely put it, when you can have true out-of-core SQL?

In version 12 we have introduced integration of SQL-backed databases into the Entity Framework and we've been able to successfully query TB-scale data sets.

Admittedly, queries done through the Entity Framework are more constrained than what Dataset allows, but that's a very important tradeoff. By limiting the expressiveness of the language, we've been able to have a small set of primitives that keep the data in a tabular form, which is in turn amenable to compilation into SQL queries.

I personally think that the kind of hybrid workflows where one first slices and dices data that is too large to keep in memory (by filtering and aggregating), and then brings the smaller data set in memory to be treated with a much more expressive language is what most data scientists are moving towards. This is also happening in they Python community, just to name one.

If you're interested in learning more there is this excellent tutorial authored by none other than Leonid Shifrin.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ sorry but that's not an answer to my question which specifically asks for Dataset. an inconvenient truth is that Python does a far better job at a host of things, including handling large datasets. $\endgroup$
    – goodheart
    Commented Nov 13, 2019 at 13:11
  • $\begingroup$ Well... I'm giving you a good reason not to use Dataset. So in a sense it is a negative answer to your question. It also gives you some insight on why it is very difficult to have proper out-of-core Datasets (the Dataset domain specific language is not constrained enough). $\endgroup$
    – Carlo
    Commented Nov 13, 2019 at 13:33
  • $\begingroup$ The please use a comment section for this, dont take up answer slots for what otherwise would be missing the point. $\endgroup$
    – goodheart
    Commented Nov 13, 2019 at 13:35
  • $\begingroup$ @goodheart Even though this answer may be perceived as a tangential one, it may still be useful - if not for you, then for others coming to this discussion. It is also too large for a comment. In general, we do allow such answers on the site, and also your question was broad enough to allow such interpretation too. You don't have to vote it up if you feel it is irrelevant, and the others have the same freedom. $\endgroup$
    – Leonid Shifrin
    Commented Nov 13, 2019 at 13:40
  • 8
    $\begingroup$ But the sad truth is, that Dataset doesn't add any efficiency to the raw processing using Lists and Associations - more on the opposite, as you noted. The new entity-based framework for working with relational databases is currently the only efficient and high-level / well integrated into the language way to work with really large datasets in Mathematica, which is why I feel that this answer is still relevant. $\endgroup$
    – Leonid Shifrin
    Commented Nov 13, 2019 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.