# speed up iteration with conditionals plus optimize memory usage

Given list1 and list2 whose elements are vectors of a certain (fixed) dimension, I am interested in the behaviour of a scalar function cfn[list1[[i]],list2[[j]]]. As I defined it this function outputs for me a list of entries formatted as {list1[[i]], list2[[j]], scalar} since I would also like to keep track of which vectors the scalar number came from.

The fastest way to generate such a list is of course to use Outer to map my function over the two lists

Outer[cfn[#1 ,#2]&, list1 ,list2,1]


However this will produce for me a complete list of values for the function after scanning through my lists and since my lists are very long the process is long and I run out of memory if I increase the dimension (typically list1*list2 is a function also of the fixed dimension and I seem to be fine doing order of 10 million computations).

If I am uninterested in all the function values, but say only those in a certain range cmin < cfn[list1,list2] < cmax is there an efficient way to scan the lists and pick out just these?
I tried the obvious nested For loops and as expected ended up slowing down the computation significantly.

Thanks!

Edit: As requested in the comments I am attaching a simplified version of my code which only computes the inner product of the two vectors after some redefinitions. n is a number that we specify and list1 and list2 are lists whose entries are integer valued vectors of length (n-1) e.g., list1[[1]] = {0, 0, 0, 4} etc.

cfn[list1_List, list2_List, n_] := Module[{rhow},
rhow = Table[1, {i, 1, n - 1}];
Lambda = list1 - list2 + rhow;
hdim = Lambda.Lambda;
Return[{list1, list2, hdim}]];

Timing[data = Flatten[Outer[cfn[#1 , #2, n] &, list1 , list2, 1], 1];
selectdata = Select[data, cmin < #[[3]] < cmax &];]


The second code which nests the For loops:

rhow = Table[1, {i, 1, n - 1}];
sampledataAlt[list1_List, list2_List, n_] := Module[{},
sampledata2 = {};
For[i = 1, i <= Length[list1], i++,
For[j = 1, j <= Length[list2], j++,
Lambda = list1[[i]] - list2[[j]] + rhow;
hdim  = Lambda.Lambda;
If[cmin < hdim < cmax,
dataNEW = {list1[[i]], list2[[j]], hdim}, dataNEW = {}];
sampledata2 = Join[sampledata2, dataNEW];]];
Return[sampledata2];]
Timing[test = Partition[sampledataAlt[list1, list2, 5], 3];]


where I have made some list manipulations to split things up.

For my trial with Length[list1] = 126 and Length[list2] = 210 the second code is marginally faster, but it slows down when I increase the list sizes.

-
Would it be possible to have full, executable examples please? I find that I am much more likely to attempt to answer questions like this if I can copy and paste something into Mathematica, run it and then start playing around –  WalkingRandomly Nov 28 '12 at 11:56
what's wrong with Outer[cfn[#1, #2] &, list1[[Range[2, 4]]], list2[[Range[3, 5]]]] –  chris Nov 28 '12 at 12:44
I have added a sample code which is a simplified version of what I am looking for. Thanks! –  bartimaeus Nov 28 '12 at 14:00
One thing I would definitely try for numerical lists is to use Compile. If there are certain properties of your cfn function such as monotonous behavior, or anything else that would allow you to guess the ordering of your results, then you may consider a similar question –  Leonid Shifrin Nov 28 '12 at 14:47
Thanks for the code but would you mind going one step further and create some random input test lists please? –  WalkingRandomly Nov 28 '12 at 16:19

## 1 Answer

A possible approach is to calculate all the function values, but don't attach the elements of list1 and list2 until after filtering out the unwanted values. This should save quite a bit of memory. In the code below I redefine cfn to return only the dot product, use Outer as you did, then Position to locate the elements between cmin and cmax. Your final data structure is pulled together from list1, list2 and data using the output from Position.

cfn[l1_, l2_] := (#.#) &[l1 - l2 + 1]

list1 = DeveloperToPackedArray[list1];
list2 = DeveloperToPackedArray[list2];

data = Outer[cfn, list1, list2, 1];
p = Position[data, x_ /; 10 < x < 15, {2}, Heads -> False];
selectdata = Transpose[{list1[[p[[All, 1]]]], list2[[p[[All, 2]]]], Extract[data, p]}];

-
This is a nice trick; I will check if it works for my problem. –  bartimaeus Nov 29 '12 at 9:17