How to efficiently take complement of two big lists?

Question

I have a giant list l1 and some smaller lists l21, l21, ..., l2n. The l1 is the superset and has $\tilde{}10^7$ elements. l2? is subset of l1 (if my other code does correct job). All lists only contain numbers (again, if my code does correct job). I need to take complement of l1 and each of the l2?. The number of subsets (i.e. n) is several hundreds. What's the most memory/time efficient way of doing this? I run this on 4GB machine and usually have 1.* GB of free memory when the program reaches that stage. I'd probably save numbers in file and use some command-line tool like grep to do this. But the original lists are all prepared in a Mathematica program. If there is a good solution in Mathematica then i'd like to avoid going out.

Answer 1

This depends.. How many numbers are in the subsets? In which range are you numbers? What do you want to do with each complement? Can you give a small example using RandomInteger to create sample-data? Generally, you could first try to calculate one complement by using something like this

l1 = RandomInteger[{0, 10^6}, {10^7}];
l21 = RandomInteger[{0, 10^6}, {10^5}];

compl = Complement[l1, l21];

and see whether you memory is sufficient.

To see how much memory is used you can tryMemoryInUse[]. ByteCount[expr] is able to find out how much memory is used by a variable (or expression in general). After the above command, I have wasted

MemoryInUse[]/2^20.

(* Out[9]= 101.952 *)

about 100 MB here.