Reorganizing large list

I have a large list of lists (1.8 million sublists) where the first three entries define the list entry, and the other entries depend on these three entries.

list={{x11,x21,x31,f1[x11,x21,x31],f2[x11,x21,x31],...},
{x12,x22,x32,f1[x12,x22,x32],f2[x12,x22,x32],...},
...}


I want to reorganize this list completely and split it up. I want separate, 3d lists for each function, such that f1list[[1,1,1]]=f1[x11,x21,x31] and so on for every function. I already have a 1D list that contains all entries for the x values, i.e. x1list={x11,x12,x13,...}

What is the most efficient way to do this? Looping trough it three times with IF statements sounds super stupid. I have been toying around with approaches using Sort and Cases but couldn't make it work efficiently.

• Comments are not for extended discussion; this conversation has been moved to chat.
– Kuba
Nov 7, 2022 at 8:17
• @Kuba by moving those comments to chat, you've removed all of the necessary information needed to actually solve the problem. I mean, maybe it was just me, but I don't think the original post had nearly enough explanation in it to come up with a solution. So, while I do appreciate not having the extended commentary, I think we need an improved question before we can do without the commentary. I guess I'm just reacting to my questions being described as "extended discussion" rather than "getting to the point". Nov 7, 2022 at 21:39
• @Noldig, per the above, can you update your question and flesh it out with some of the detail we "discovered"? That's assuming we actually did get to the root of the problem. Nov 7, 2022 at 21:40
• @lericr They should be in the question then. OP or you can edit it. No one is expected to read a long conversation just to get the point.
– Kuba
Nov 8, 2022 at 5:53

Okay, maybe there's some organization of list you could take advantage of, but without knowing that, you can recover the temperature, density and particleFraction like this:

temperatureValues = Union[list[[All, 1]]];
densityValues = Union[list[[All, 2]]];
particleValues = Union[list[[All, 3]]];


Now you can use Table to get the data structured by parameter:

newList =
Table[
{f1[t, d, p], f2[t, d, p], f3[t, d, p]},
{t, temperatureValues},
{d, densityValues},
{p, particleValues}]


(Of course, you'll have as many fns as you need.)

You could also use it for just f1list:

f1list =
Table[
f1[t, d, p],
{t, temperatureValues},
{d, densityValues},
{p, particleValues}]


You might also want the following, which gives you the three fNlist lists from newList:

Flatten[newList, {{4}, {2, 3}}]


(I think that's what you want, but maybe that's not the right order. Just play with the Flatten parameters until you get what you want.)

The point of this answer is that the organization of list is similar to ArrayRules and that therefore one approach to this problem is to call SparseArray, because it is the inverse of ArrayRules. This makes sense as a step in the computation even if the final 3d arrays are actually dense.

I will use this as an example:

list={{2,2,4,a,b},{2,3,3,c,d}};
x1list={2};
x2list={2,3};
x3list={3,4};


Code:

(* more convenient numbering *)
xlist[1] = x1list;
xlist[2] = x2list;
xlist[3] = x3list;

(* create a list of 3d positions *)
positions = Transpose[Table[
Replace[list[[;;,k]],
{k,1,3}]];

(* generate sparse 3d arrays *)


Use Normal to get ordinary arrays:

f1list = Normal[f1list]
(* {{{0,a},{c,0}}} *)

f2list = Normal[f2list]
(* {{{0,b},{d,0}}} *)


• An alternative approach would be to turn list into an Association, aka hash table, instead of 3d arrays, which allows for fast lookup. Use something like

f1association = Association[Map[(#[[;;3]]->#[[3+1]])&,list]];
f2association = Association[Map[(#[[;;3]]->#[[3+2]])&,list]];


and retrieve a value using, for example,

f1association[{2,2,4}]
(* a *)

• If OP does not have list yet and wants to evaluate f1, f2, ... from scratch, they can use

f1list = Outer[f1,x1list,x2list,x3list];
f2list = Outer[f2,x1list,x2list,x3list];

• Is there an advantage to the SparseArray approach over Association ? Personally which approach do you prefer? Nov 7, 2022 at 11:09
• Since OP has data that fills 3d arrays, I think OP's plan of constructing 3d arrays is reasonable. I proposed using SparseArray as an intermediate step to construct that array. I mentioned associations because they are easier to construct in OP's case, given list. On the other hand, throwing a big dense array into an association makes limited sense. It also comes with a memory overhead that I think is not so small in the case of 1.8 million entries and keys equal to lists 3 numbers. If would certainly measure that memory overhead, for example using ByteCount. Nov 7, 2022 at 13:17