6
$\begingroup$

I have a process that returns an unpredictable number of data points, and I'd like to run it repeatedly until I have a certain number of points.

My actual code is too complicated to use an illustration, so I wrote this toy example. fakeData[] will return 1-21 data points, and I want to run it until I have at least 100. But this code doesn't work because you can't take the so-far Length[] of a list that you're still building.

fakeData[n_] := RandomReal[1, 1 + RandomInteger[n]];
big = Reap[
   While[Length[big] < 100, (* this doesn't work*)
    Sow[fakeData[20]]]][[2, 1]]

I could just allocate 'big' as a Table with length 100 and copy each new small list into it, but then I'd have to discard some perfectly good data points I laboriously calculated, which is distasteful. Is there a better way?

$\endgroup$
  • $\begingroup$ Will NestWhile[(Join[#, fakeData[20]]) &, {}, Length[#] < 100 &] work for you? $\endgroup$ – Shadowray Jul 24 at 20:19
  • $\begingroup$ @Shadowray That will work, but I've been told that using Join[] repeatedly is very, very slow because of all the recopying necessary every time you add a chunk. $\endgroup$ – Jerry Guern Jul 25 at 0:05
7
$\begingroup$

How about:

SeedRandom[1]
Reap[NestWhile[Join[#, Sow@fakeData[20]] &, {}, LessThan[100]@*Length]][[2, 1]]

{{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649}, {0.337261, 0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, 0.964289}, {0.741987, 0.819242}, {0.539713}, {0.012502, 0.439595, 0.169709, 0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584}, {0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, 0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, 0.245655, 0.562932}, {0.370393, 0.934574, 0.550753, 0.136193, 0.390665, 0.941924, 0.743334, 0.296465}, {0.114065, 0.612737, 0.596194, 0.32461, 0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, 0.129602}, {0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, 0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, 0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614}, {0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, 0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, 0.315747, 0.367579, 0.521331, 0.36944, 0.566759}}

Another similar possibility:

SeedRandom[1]
Reap[NestWhile[Length @ Sow @ fakeData[20] &, 0, LessThan[100] @* Plus, All]][[2, 1]]

same answer

$\endgroup$
  • $\begingroup$ Okay, thank you, that seems to do what exactly I wanted, now I just have to study docs for a while to understand how/why it works. :-) May I ask, why did you put that SeedRandom[1] in there? I don't see it's purpose, but I assume you had expert-level reasons. $\endgroup$ – Jerry Guern Jul 23 at 18:26
  • $\begingroup$ Because fakeData calls RandomReal and RandomInteger and these random functions can have reproducible results if you specify the seed. $\endgroup$ – rhermans Jul 23 at 18:28
  • $\begingroup$ One issue with this is that it doubles the memory cost, no? With large Reaps that could be prohibitive $\endgroup$ – b3m2a1 Jul 23 at 20:15
6
$\begingroup$

The straight forward solution is to simply count the number of points you have sown, i.e.:

big = Module[
    {count = 0},
    Reap[
        While[ count < 100, count += Length@Sow[fakeData[20]] ]
    ][[2,1]]
]
$\endgroup$
  • $\begingroup$ Sorry, I misread your question. See updated answer. $\endgroup$ – sakra Jul 23 at 18:31
  • 1
    $\begingroup$ Oh, I see, you're right, I can just manually track the length as I as to it. Thanks. $\endgroup$ – Jerry Guern Jul 23 at 22:26
5
$\begingroup$

Here's a method that just uses Bag since I think effectively that's what Reap and Sow are using. It's probably a bit slower than adding the lists directly and flattening after, but it's conceptually how you were thinking about the original problem:

bag = Internal`Bag[];
SeedRandom[1]
While[Internal`BagLength[bag] < 100,
  Internal`StuffBag[bag, #] & /@ fakeData[20]
  ];
Internal`BagPart[bag, All]

{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649, 0.337261, \
0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, \
0.964289, 0.741987, 0.819242, 0.539713, 0.012502, 0.439595, 0.169709, \
0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584, \
0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, \
0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, \
0.245655, 0.562932, 0.370393, 0.934574, 0.550753, 0.136193, 0.390665, \
0.941924, 0.743334, 0.296465, 0.114065, 0.612737, 0.596194, 0.32461, \
0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, \
0.129602, 0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, \
0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, \
0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614, \
0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, \
0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, \
0.315747, 0.367579, 0.521331, 0.36944, 0.566759}
$\endgroup$
  • $\begingroup$ Oh, thank you. SE didn't notify me of this Answer for some reason, so I just saw it today. Yes, this is conceptually what I was trying to do, and I appreciate this glimpse into what MMA is doing 'under the hood'. $\endgroup$ – Jerry Guern Oct 2 at 19:41
  • $\begingroup$ I'm wondering though, if Bag is what Reap/Sow use internally, why would this be slower than adding lists, rather than much faster? I didn't understand that part of your Answer. $\endgroup$ – Jerry Guern Oct 2 at 19:43
  • $\begingroup$ @JerryGuern You’re making Mathematica loop a bunch of times rather than letting that happen at the C level when Flatten is called. Basically I’m saying that Sakra’s answer will be the best in all likelihood. In general, you want to maintain a nested List structure when constructing your dataset because this is stored as a linked list. Next best is list of list since that will be pointers. Then when you need it cast to the proper array you’ll want to use. $\endgroup$ – b3m2a1 Oct 3 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.