# How can I efficiently save multidimensional data that is retrievable by key?

I am doing some simulations in Mathematica, where I calculate a spectrum intensity(frequency) for a variation of parameters in my samples. So in the following image, for example, I vary the charge carrier concentration of a layer and get the following plot:

What I have to do now is do a variation of 5-10 different parameters (charge carrier concentration, mobility, effective mass, thickness, ...) and see the influence of each, depending on each. So I don't want to get 5 of these plots, because in the plot all other parameters are constant; instead, I want to change all parameters and then do e.g. 3D plots to show the influence of any two parameters at once.

As this is going to be a lot of simulations, I want to chuck in a list of values for each parameter, do all the individual simulations over e.g. a week of computation, fill the results in a big n-dimensional hypercube of data and save that. Later on, I would like to be able to extract data from this saved data hypercube in a controlled manner by asking e.g. for all data points with: mobility = 5, effective mass = 3, etc.

As I would then like to use this setup for different samples, or maybe it will be used by other people, or the parameters themselves might also change, ... so I would prefer to use keys instead of just list indices like data[[1,1,1,All]].

I was thinking of making a dataset with every data point added separately like the example from the documentation:

    dataset = Dataset[{
<|"a" -> 1, "b" -> "x", "c" -> {1}|>,
<|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
<|"a" -> 3, "b" -> "z", "c" -> {3}|>,
<|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
<|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
<|"a" -> 6, "b" -> "z", "c" -> {}|>}]


and then I would use all the parameters as key-value pairs to find a row, while the list shown in "c" in this example would then be a list of lists {{x1, x2,...},{y1, y2,...}} with the intensity(frequency) data I want to plot for this parameter set.

Does that seem plausible? It will probably result in billions of data points and maybe I would like to interpolate across part of the data later. Is there a better way to do this?

I would be happy for some input and could not really find an answer to this question via search for multidimensional data and similar requests. Thanks a lot in advance!

• Why not write a parser for your request? It seems to me your data may be so large such that an association applied to the whole set would not be advantageous. May 22, 2020 at 21:09
• I usually label my files with useful informations regarding their content such that I can reconstruct the format of the parameters that make up the block of data. May 22, 2020 at 21:15
• @CATrevillian Thanks for your reply! Do I understand you correctly, that you would save the simulation results for each parameter set in a separate file and give it a header that contains the parameters? May 23, 2020 at 22:45
• yeah so I use the mx file format and basically construct the file name according to the parameters I put into it, then go from there to autocorrelate what values belong to what parts of the file name. I’m explaining it badly but it should also be noted that this is using only inputted values, not those that are computed within the simulation. May 23, 2020 at 23:24
• @CATrevillian so let's assume that I did a variation of the electron mobility and saved this in files which contain this information in their filename, like "mobility=5.mx", "mobility=10.mx", etc. .. How would I open those in a way that I can control something similar to "get me all the files of the mobility variation with mobility values between 1 and 100"? May 27, 2020 at 6:53