Reorganizing large list

Question

I have a large list of lists (1.8 million sublists) where the first three entries define the list entry, and the other entries depend on these three entries.

list={{x11,x21,x31,f1[x11,x21,x31],f2[x11,x21,x31],...},
      {x12,x22,x32,f1[x12,x22,x32],f2[x12,x22,x32],...},
      ...}

I want to reorganize this list completely and split it up. I want separate, 3d lists for each function, such that f1list[[1,1,1]]=f1[x11,x21,x31] and so on for every function. I already have a 1D list that contains all entries for the x values, i.e. x1list={x11,x12,x13,...}

What is the most efficient way to do this? Looping trough it three times with IF statements sounds super stupid. I have been toying around with approaches using Sort and Cases but couldn't make it work efficiently.

Answer 1

Okay, maybe there's some organization of list you could take advantage of, but without knowing that, you can recover the temperature, density and particleFraction like this:

temperatureValues = Union[list[[All, 1]]];
densityValues = Union[list[[All, 2]]];
particleValues = Union[list[[All, 3]]];

Now you can use Table to get the data structured by parameter:

newList = 
  Table[
    {f1[t, d, p], f2[t, d, p], f3[t, d, p]}, 
    {t, temperatureValues}, 
    {d, densityValues}, 
    {p, particleValues}]

(Of course, you'll have as many fns as you need.)

You could also use it for just f1list:

f1list = 
  Table[
    f1[t, d, p], 
    {t, temperatureValues}, 
    {d, densityValues}, 
    {p, particleValues}]

You might also want the following, which gives you the three fNlist lists from newList:

Flatten[newList, {{4}, {2, 3}}]

(I think that's what you want, but maybe that's not the right order. Just play with the Flatten parameters until you get what you want.)

Answer 2

The point of this answer is that the organization of list is similar to ArrayRules and that therefore one approach to this problem is to call SparseArray, because it is the inverse of ArrayRules. This makes sense as a step in the computation even if the final 3d arrays are actually dense.

I will use this as an example:

list={{2,2,4,a,b},{2,3,3,c,d}};
x1list={2};
x2list={2,3};
x3list={3,4};

Code:

(* more convenient numbering *)
xlist[1] = x1list;
xlist[2] = x2list;
xlist[3] = x3list;

(* create a list of 3d positions *)
positions = Transpose[Table[
   Replace[list[[;;,k]],
      Dispatch[Thread[xlist[k]->Range[Length[xlist[k]]]]],{1}],
         {k,1,3}]];

(* generate sparse 3d arrays *)
f1list = SparseArray[MapThread[Rule,{positions,list[[;;,3+1]]}]];
f2list = SparseArray[MapThread[Rule,{positions,list[[;;,3+2]]}]];

Use Normal to get ordinary arrays:

f1list = Normal[f1list]
(* {{{0,a},{c,0}}} *)

f2list = Normal[f2list]
(* {{{0,b},{d,0}}} *)

---

Two comments.

• An alternative approach would be to turn list into an Association, aka hash table, instead of 3d arrays, which allows for fast lookup. Use something like

  f1association = Association[Map[(#[[;;3]]->#[[3+1]])&,list]]; 
  f2association = Association[Map[(#[[;;3]]->#[[3+2]])&,list]]; 
  

  and retrieve a value using, for example,

  f1association[{2,2,4}]
  (* a *)
  

• If OP does not have list yet and wants to evaluate f1, f2, ... from scratch, they can use

  f1list = Outer[f1,x1list,x2list,x3list];
  f2list = Outer[f2,x1list,x2list,x3list];