3
$\begingroup$

I'm updating my package for Dirichlet Characters (https://github.com/truculentmath/DirichletCharacters) to include precomputed zeros of the associated $L$-series. That's about 15 meg of data, currently parsed into about 900 files. Most users of the package (there aren't many, yet) would not use that data, but for those that would, the data will be a valuable time saver. A particular user may only use the data from 1 file, or 5 files. In my own work, I need all 900.

Where is the correct location (to instruct the user) to put the data? FileNameJoin[{$UserBaseDirectory, "Applications", "CharacterData"}]? Is there a convenient (or standard) way for the package to load the data if the user has the data files and tries to use the function, but to not load those files that aren't called for, or that the user doesn't have? Is there a reasonable way to (within the package) have the user download the data?

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ Is all this trouble worth it just to avoid bundling the data with the package? 15 MB is nothing today. You probably use more data than that during a short web browsing session. You could even store it compressed, which would significantly reduce the size. You can bundle it together into a single file to avoid a mess. $\endgroup$
    – Szabolcs
    Commented Apr 24, 2020 at 17:14
  • 1
    $\begingroup$ This is what I do in my package: github.com/szhorvat/IGraphM/blob/master/IGraphM/IGData.m#L17 Notice that the data is only loaded upon first use. It does not slow down package loading. Here's the data: github.com/szhorvat/IGraphM/blob/master/IGraphM/IGData.mz In this case I used a string which can be passed to Uncompress. There are more efficient ways to store it too, e.g. a gzipped MX file (though I'm not sure how future-proof that is). $\endgroup$
    – Szabolcs
    Commented Apr 24, 2020 at 17:16
  • 1
    $\begingroup$ If you absolutely insist on not bundling all the data with your package, and instead asking your users to install it themselves if they need it (which is more work for them), I would recommend using $UserBaseDirectory/ApplicationData/DirichletCharacters. Some people use the package's own installation directory, but I don't like that approach: what if the package is updated or reinstalled or repackaged into paclet form? $\endgroup$
    – Szabolcs
    Commented Apr 24, 2020 at 17:20
  • 2
    $\begingroup$ If you clarify your needs further, I'd be happy to help come up with the best solution. I'd like to understand if bundling all data is a no-go, and if so, why? $\endgroup$
    – Szabolcs
    Commented Apr 24, 2020 at 17:31
  • 2
    $\begingroup$ Can you share the data so we can experiment? (I assume this is no problem since you plan to make it public anyway.) I am not a big fan of Dataset, as the performance is not always great. Would you be okay with using other formats to manipulate the data in your package? $\endgroup$
    – Szabolcs
    Commented Apr 25, 2020 at 19:39

1 Answer 1

Reset to default
3
$\begingroup$

I have combined all the data into 1 file, and stored it in ApplicationData/DirichletCharacter/dataset.mx

In the package I have a CharacterData command, defined as

CharacterData := CharacterData = Normal[Import[FileNameJoin[{$UserBaseDirectory,"ApplicationData","DirichletCharacters","dataset.mx"}]]]

"Normal" converts from a Dataset to an Association, which I do because Dataset doesn't like keys that aren't strings, for some reason. I have a separate command LSeriesZeroHeights[char] that pulls the data relevant to char out of CharacterData.

This is smooth enough, and fast enough, for my purposes. I hope to grow the dataset over time, but it is much more expensive to compute than to store, so I doubt I'll ever really have a memory headache.

Thanks to Szabolcs for the advice, I profit from your experience!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.