Skip to main content
Tweeted twitter.com/StackMma/status/1269282936634912770
edited tags
Link
Source Link
JP-Ellis
  • 1k
  • 7
  • 15

Random Sample from Very Large Table

I'm currently trying to create some test data to check a numerical implementation against Mathematica's implementation of some functions.

In order to do this, I've been generating a broad range of arguments with a (Parallel)Table which works fine for a function with a few arguments, but I have one function I want to test which takes 4 integer arguments and 10 floating point arguments:

f[n1, n2, n3, n4, x0, x1, x2, x3, x4, x5, x6, x7, x8, x9]

The thing is creating a broad range of values use a table results in a simply enormous amount of data which cannot be held in memory. For example even:

realSample[n_] := Join[-10^Subdivide[-10, 10, n], 10^Subdivide[-10, 10, n], Subdivide[-10, 10, n]];
Table[{n1, n2, n3, n4, a, b, c, d, e, f, w, x, y, z},
  {n1, 0, 1}, {n2, 0, 1}, {n3, 0, 1}, {n4, 0, 1},
  {x0, realSample[1]}, ...
]

results in something that is just unmanageable even for n = 1 simply because of the sheer number of arguments.

So instead, I'd like to take a random sample of the whole table (say a million); however, RandomSample doesn't stream the table and the same problem as before still comes up.

Short of implementing a reservoir sampling, is there anything in Mathematica that allows me to take a random sample of the full table without loading the whole table into memory?