Given list1
and list2
whose elements are vectors of a certain (fixed) dimension, I am interested in the behaviour of a scalar function cfn[list1[[i]],list2[[j]]]
.
As I defined it this function outputs for me a list of entries formatted as {list1[[i]], list2[[j]], scalar}
since I would also like to keep track of which vectors the scalar number came from.
The fastest way to generate such a list is of course to use Outer to map my function over the two lists
Outer[cfn[#1 ,#2]&, list1 ,list2,1]
However this will produce for me a complete list of values for the function after scanning through my lists and since my lists are very long the process is long and I run out of memory if I increase the dimension (typically list1*list2
is a function also of the fixed dimension and I seem to be fine doing order of 10 million computations).
If I am uninterested in all the function values, but say only those in a certain range cmin < cfn[list1,list2] < cmax
is there an efficient way to scan the lists and pick out just these?
I tried the obvious nested For loops and as expected ended up slowing down the computation significantly.
Thanks!
Edit: As requested in the comments I am attaching a simplified version of my code which only computes the inner product of the two vectors after some redefinitions.
n
is a number that we specify and list1
and list2
are lists whose entries are integer valued vectors of length (n-1)
e.g., list1[[1]] = {0, 0, 0, 4}
etc.
cfn[list1_List, list2_List, n_] := Module[{rhow},
rhow = Table[1, {i, 1, n - 1}];
Lambda = list1 - list2 + rhow;
hdim = Lambda.Lambda;
Return[{list1, list2, hdim}]];
Timing[data = Flatten[Outer[cfn[#1 , #2, n] &, list1 , list2, 1], 1];
selectdata = Select[data, cmin < #[[3]] < cmax &];]
The second code which nests the For loops:
rhow = Table[1, {i, 1, n - 1}];
sampledataAlt[list1_List, list2_List, n_] := Module[{},
sampledata2 = {};
For[i = 1, i <= Length[list1], i++,
For[j = 1, j <= Length[list2], j++,
Lambda = list1[[i]] - list2[[j]] + rhow;
hdim = Lambda.Lambda;
If[cmin < hdim < cmax,
dataNEW = {list1[[i]], list2[[j]], hdim}, dataNEW = {}];
sampledata2 = Join[sampledata2, dataNEW];]];
Return[sampledata2];]
Timing[test = Partition[sampledataAlt[list1, list2, 5], 3];]
where I have made some list manipulations to split things up.
For my trial with Length[list1] = 126
and Length[list2] = 210
the second code is marginally faster, but it slows down when I increase the list sizes.
Outer[cfn[#1, #2] &, list1[[Range[2, 4]]], list2[[Range[3, 5]]]]
$\endgroup$Compile
. If there are certain properties of yourcfn
function such as monotonous behavior, or anything else that would allow you to guess the ordering of your results, then you may consider a similar question $\endgroup$