speed up iteration with conditionals plus optimize memory usage

Question

Given list1 and list2 whose elements are vectors of a certain (fixed) dimension, I am interested in the behaviour of a scalar function cfn[list1[[i]],list2[[j]]]. As I defined it this function outputs for me a list of entries formatted as {list1[[i]], list2[[j]], scalar} since I would also like to keep track of which vectors the scalar number came from.

The fastest way to generate such a list is of course to use Outer to map my function over the two lists

Outer[cfn[#1 ,#2]&, list1 ,list2,1] 

However this will produce for me a complete list of values for the function after scanning through my lists and since my lists are very long the process is long and I run out of memory if I increase the dimension (typically list1*list2 is a function also of the fixed dimension and I seem to be fine doing order of 10 million computations).

If I am uninterested in all the function values, but say only those in a certain range cmin < cfn[list1,list2] < cmax is there an efficient way to scan the lists and pick out just these?
I tried the obvious nested For loops and as expected ended up slowing down the computation significantly.

Thanks!

Edit: As requested in the comments I am attaching a simplified version of my code which only computes the inner product of the two vectors after some redefinitions. n is a number that we specify and list1 and list2 are lists whose entries are integer valued vectors of length (n-1) e.g., list1[[1]] = {0, 0, 0, 4} etc.

cfn[list1_List, list2_List, n_] := Module[{rhow}, 
    rhow = Table[1, {i, 1, n - 1}]; 
        Lambda = list1 - list2 + rhow; 
hdim = Lambda.Lambda; 
    Return[{list1, list2, hdim}]]; 

Timing[data = Flatten[Outer[cfn[#1 , #2, n] &, list1 , list2, 1], 1];
       selectdata = Select[data, cmin < #[[3]] < cmax &];]

The second code which nests the For loops:

rhow = Table[1, {i, 1, n - 1}];
sampledataAlt[list1_List, list2_List, n_] := Module[{},
            sampledata2 = {};
            For[i = 1, i <= Length[list1], i++,
            For[j = 1, j <= Length[list2], j++,
                Lambda = list1[[i]] - list2[[j]] + rhow;
                hdim  = Lambda.Lambda;
                If[cmin < hdim < cmax, 
 dataNEW = {list1[[i]], list2[[j]], hdim}, dataNEW = {}];
                sampledata2 = Join[sampledata2, dataNEW];]];
                Return[sampledata2];]
 Timing[test = Partition[sampledataAlt[list1, list2, 5], 3];]

where I have made some list manipulations to split things up.

For my trial with Length[list1] = 126 and Length[list2] = 210 the second code is marginally faster, but it slows down when I increase the list sizes.

Answer 1

A possible approach is to calculate all the function values, but don't attach the elements of list1 and list2 until after filtering out the unwanted values. This should save quite a bit of memory. In the code below I redefine cfn to return only the dot product, use Outer as you did, then Position to locate the elements between cmin and cmax. Your final data structure is pulled together from list1, list2 and data using the output from Position.

cfn[l1_, l2_] := (#.#) &[l1 - l2 + 1]

list1 = Developer`ToPackedArray[list1];
list2 = Developer`ToPackedArray[list2];

data = Outer[cfn, list1, list2, 1];
p = Position[data, x_ /; 10 < x < 15, {2}, Heads -> False];
selectdata = Transpose[{list1[[p[[All, 1]]]], list2[[p[[All, 2]]]], Extract[data, p]}];