# Properly appending actual data values to a null list spanning full parameter space

I have five independent variables and one dependent "output". I am scanning across the independent variables and generating a nested list of these and the corresponding output (denoted here by 'f'). For example, {{a1,b1,c1,d1,e1,f},{a2,b1,c1,d1,e1,f},...,{an,b1,c1,d1,e1,f},{a1,b2,c1,d1,e1,f},...} i.e., each variable is being scanned across in turn with each permutation yielding a different output 'f'. In matrix form,

As you can see, not all values are scanned through. This is because the system is a set of differential equations, and I believe whenever NDSolveValue encounters a singularity it skips that set of values.

In order to interpolate the data, however, I need it to be on a regular grid. One way around this would be to create a nested list of zeroes corresponding to the size of the full parameter space -- {{an,bn,cn,0}} -- and 'properly' append to this the nested list of solutions from NDSolve, for the choice of parameters which do have a solution. How do I do this?

More info: I am trying to scan across {a, PowerRange[0.00001, 0.1, 10]},{b, 200, 1600, 200}, {c, PowerRange[1, 10000, 10]}, {d, PowerRange[1, 10000, 10]}, {e, 0.05, 0.25, 0.05}, if it helps. The actual data set is a lot bigger (in case you are worried about having enough points to interpolate), but still isnt 'regular' in this way.

• If you scan across the proper range, why not just add a 0 instead of nothing during this step? Then the data are already regular without having to fix anything. Just note that the interpolation will be very strange if you introduce zeros between the proper values Jul 4, 2018 at 13:17

extras = Table[{a, b, c, d, e, 0},
{a, PowerRange[0.00001, 0.1, 10]},
{b, 200, 1600, 200},
{c, PowerRange[1, 10000, 10]},
{d, PowerRange[1, 10000, 10]},
{e, 0.05, 0.25, 0.05}
]~Flatten~4;

combined = Join[yourdata, extras];

onfullgrid = DeleteDuplicatesBy[combined, Most]


This code does the following:

• generates your desired set of “fake points” with the dependent variable set to zero. These are generated over the full grid. I would caution you about the choice of the dependent variable value here. A zero might be far out of the trend and lead to funny interpolation results. You will have to take that into consideration though as you know your data, and you haven’t shared it,
• joins your calculated points with the “fake” ones. In this step it is critical that your points are positioned first in the resulting list for the next step to work.
• DeleteDuplicatesBy removes all list members that return the same values when a selector function is applied to each. In this case we use Most as a selector function, which returns all members of a list but the last. Note that only the first-occurring one of these duplicates is retained by DeleteDuplicatesBy in the result. This is why it is critical that you have your good calculated points first in the joined set.
• Thank you very much, this is very helpful. However -- DeleteDuplicatesBy doesn't seem to be working. The set of "fake points" is appended to the end of the data with Join, but after that the duplicates remain and neither is it ordered with the data. So I'm guessing the problem is with the selector? Jul 5, 2018 at 14:20
• @Sarah Regarding the ordering, that will not hurt further use: Interpolation wouldn’t mind if the points are “out of order”. I tried reproducing the data you had in the image and this method seemed to work fine. Unfortunately I cannot say anything about your data, because you didn’t provide it for me to test. It is difficult to answer questions precisely when the implementation details might depend on features of your data that I cannot inspect. Jul 5, 2018 at 14:33
• Hmm. When I try it on a small test set: test = {{0.1, 1000, 1, 1, 0.25, 0.00042}, {0.1, 1000, 1, 10, 0.25, 0.00041}, {0.01, 1200, 1, 1, 0.25, 0.01013}, {0.1, 1200, 1, 1, 0.2, 0.00028}, {0.1, 1200, 1, 1, 0.25, 0.00018}}; null = Table[{a, b, c, d, e, 0}, {a, PowerRange[0.01, 0.1, 10]}, {b, 1000, 1200, 200}, {c, PowerRange[1, 10, 10]}, {d, PowerRange[1, 10, 10]}, {e, 0.2, 0.25, 0.05}]~Flatten~4; combined = Join[test, null]; onfullgrid= DeleteDuplicatesBy[combined, Most] I find that I am still getting duplicates? Jul 5, 2018 at 14:52
• I would be glad to share my full data set, I am not sure of the etiquette -- how exactly do I post this? Jul 5, 2018 at 14:54
• @Sarah. I'll take a look at the sample you sent. Meanwhile, you could add it as text in the question, but if that's too big, you can provide a link to an online code sharing service. I tend to use pastebin. You paste your data as text there (e.g. as a MMA expression). Jul 5, 2018 at 15:49