# How to Sow[] until I've Reap[]'d enough?

I have a process that returns an unpredictable number of data points, and I'd like to run it repeatedly until I have a certain number of points.

My actual code is too complicated to use an illustration, so I wrote this toy example. fakeData[] will return 1-21 data points, and I want to run it until I have at least 100. But this code doesn't work because you can't take the so-far Length[] of a list that you're still building.

fakeData[n_] := RandomReal[1, 1 + RandomInteger[n]];
big = Reap[
While[Length[big] < 100, (* this doesn't work*)
Sow[fakeData[20]]]][[2, 1]]


I could just allocate 'big' as a Table with length 100 and copy each new small list into it, but then I'd have to discard some perfectly good data points I laboriously calculated, which is distasteful. Is there a better way?

• Will NestWhile[(Join[#, fakeData[20]]) &, {}, Length[#] < 100 &] work for you? – Shadowray Jul 24 '19 at 20:19
• @Shadowray That will work, but I've been told that using Join[] repeatedly is very, very slow because of all the recopying necessary every time you add a chunk. – Jerry Guern Jul 25 '19 at 0:05

SeedRandom[1]
Reap[NestWhile[Join[#, Sow@fakeData[20]] &, {}, LessThan[100]@*Length]][[2, 1]]


{{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649}, {0.337261, 0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, 0.964289}, {0.741987, 0.819242}, {0.539713}, {0.012502, 0.439595, 0.169709, 0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584}, {0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, 0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, 0.245655, 0.562932}, {0.370393, 0.934574, 0.550753, 0.136193, 0.390665, 0.941924, 0.743334, 0.296465}, {0.114065, 0.612737, 0.596194, 0.32461, 0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, 0.129602}, {0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, 0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, 0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614}, {0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, 0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, 0.315747, 0.367579, 0.521331, 0.36944, 0.566759}}

Another similar possibility:

SeedRandom[1]
Reap[NestWhile[Length @ Sow @ fakeData[20] &, 0, LessThan[100] @* Plus, All]][[2, 1]]


• Okay, thank you, that seems to do what exactly I wanted, now I just have to study docs for a while to understand how/why it works. :-) May I ask, why did you put that SeedRandom[1] in there? I don't see it's purpose, but I assume you had expert-level reasons. – Jerry Guern Jul 23 '19 at 18:26
• Because fakeData calls RandomReal and RandomInteger and these random functions can have reproducible results if you specify the seed. – rhermans Jul 23 '19 at 18:28
• One issue with this is that it doubles the memory cost, no? With large Reaps that could be prohibitive – b3m2a1 Jul 23 '19 at 20:15

The straight forward solution is to simply count the number of points you have sown, i.e.:

big = Module[
{count = 0},
Reap[
While[ count < 100, count += Length@Sow[fakeData[20]] ]
][[2,1]]
]

• Oh, I see, you're right, I can just manually track the length as I as to it. Thanks. – Jerry Guern Jul 23 '19 at 22:26

Here's a method that just uses Bag since I think effectively that's what Reap and Sow are using. It's probably a bit slower than adding the lists directly and flattening after, but it's conceptually how you were thinking about the original problem:

bag = InternalBag[];
SeedRandom[1]
While[InternalBagLength[bag] < 100,
InternalStuffBag[bag, #] & /@ fakeData[20]
];
InternalBagPart[bag, All]

{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649, 0.337261, \
0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, \
0.964289, 0.741987, 0.819242, 0.539713, 0.012502, 0.439595, 0.169709, \
0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584, \
0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, \
0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, \
0.245655, 0.562932, 0.370393, 0.934574, 0.550753, 0.136193, 0.390665, \
0.941924, 0.743334, 0.296465, 0.114065, 0.612737, 0.596194, 0.32461, \
0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, \
0.129602, 0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, \
0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, \
0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614, \
0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, \
0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, \
0.315747, 0.367579, 0.521331, 0.36944, 0.566759}

• Oh, thank you. SE didn't notify me of this Answer for some reason, so I just saw it today. Yes, this is conceptually what I was trying to do, and I appreciate this glimpse into what MMA is doing 'under the hood'. – Jerry Guern Oct 2 '19 at 19:41
• I'm wondering though, if Bag is what Reap/Sow use internally, why would this be slower than adding lists, rather than much faster? I didn't understand that part of your Answer. – Jerry Guern Oct 2 '19 at 19:43
• @JerryGuern You’re making Mathematica loop a bunch of times rather than letting that happen at the C level when Flatten is called. Basically I’m saying that Sakra’s answer will be the best in all likelihood. In general, you want to maintain a nested List structure when constructing your dataset because this is stored as a linked list. Next best is list of list since that will be pointers. Then when you need it cast to the proper array you’ll want to use. – b3m2a1 Oct 3 '19 at 2:52