# How to Sow[] until I've Reap[]'d enough?

I have a process that returns an unpredictable number of data points, and I'd like to run it repeatedly until I have a certain number of points.

My actual code is too complicated to use an illustration, so I wrote this toy example. fakeData[] will return 1-21 data points, and I want to run it until I have at least 100. But this code doesn't work because you can't take the so-far Length[] of a list that you're still building.

fakeData[n_] := RandomReal[1, 1 + RandomInteger[n]];
big = Reap[
While[Length[big] < 100, (* this doesn't work*)
Sow[fakeData[20]]]][[2, 1]]


I could just allocate 'big' as a Table with length 100 and copy each new small list into it, but then I'd have to discard some perfectly good data points I laboriously calculated, which is distasteful. Is there a better way?

• Will NestWhile[(Join[#, fakeData[20]]) &, {}, Length[#] < 100 &] work for you? Commented Jul 24, 2019 at 20:19
• @Shadowray That will work, but I've been told that using Join[] repeatedly is very, very slow because of all the recopying necessary every time you add a chunk. Commented Jul 25, 2019 at 0:05

SeedRandom[1]
Reap[NestWhile[Join[#, Sow@fakeData[20]] &, {}, LessThan[100]@*Length]][[2, 1]]


{{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649}, {0.337261, 0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, 0.964289}, {0.741987, 0.819242}, {0.539713}, {0.012502, 0.439595, 0.169709, 0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584}, {0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, 0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, 0.245655, 0.562932}, {0.370393, 0.934574, 0.550753, 0.136193, 0.390665, 0.941924, 0.743334, 0.296465}, {0.114065, 0.612737, 0.596194, 0.32461, 0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, 0.129602}, {0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, 0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, 0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614}, {0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, 0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, 0.315747, 0.367579, 0.521331, 0.36944, 0.566759}}

Another similar possibility:

SeedRandom[1]
Reap[NestWhile[Length @ Sow @ fakeData[20] &, 0, LessThan[100] @* Plus, All]][[2, 1]]


• Okay, thank you, that seems to do what exactly I wanted, now I just have to study docs for a while to understand how/why it works. :-) May I ask, why did you put that SeedRandom[1] in there? I don't see it's purpose, but I assume you had expert-level reasons. Commented Jul 23, 2019 at 18:26
• Because fakeData calls RandomReal and RandomInteger and these random functions can have reproducible results if you specify the seed. Commented Jul 23, 2019 at 18:28
• One issue with this is that it doubles the memory cost, no? With large Reaps that could be prohibitive Commented Jul 23, 2019 at 20:15

The straight forward solution is to simply count the number of points you have sown, i.e.:

big = Module[
{count = 0},
Reap[
While[ count < 100, count += Length@Sow[fakeData[20]] ]
][[2,1]]
]

• Oh, I see, you're right, I can just manually track the length as I as to it. Thanks. Commented Jul 23, 2019 at 22:26

Here's a method that just uses Bag since I think effectively that's what Reap and Sow are using. It's probably a bit slower than adding the lists directly and flattening after, but it's conceptually how you were thinking about the original problem:

bag = InternalBag[];
SeedRandom[1]
While[InternalBagLength[bag] < 100,
InternalStuffBag[bag, #] & /@ fakeData[20]
];
InternalBagPart[bag, All]

{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649, 0.337261, \
0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, \
0.964289, 0.741987, 0.819242, 0.539713, 0.012502, 0.439595, 0.169709, \
0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584, \
0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, \
0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, \
0.245655, 0.562932, 0.370393, 0.934574, 0.550753, 0.136193, 0.390665, \
0.941924, 0.743334, 0.296465, 0.114065, 0.612737, 0.596194, 0.32461, \
0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, \
0.129602, 0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, \
0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, \
0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614, \
0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, \
0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, \
0.315747, 0.367579, 0.521331, 0.36944, 0.566759}

• Oh, thank you. SE didn't notify me of this Answer for some reason, so I just saw it today. Yes, this is conceptually what I was trying to do, and I appreciate this glimpse into what MMA is doing 'under the hood'. Commented Oct 2, 2019 at 19:41
• I'm wondering though, if Bag is what Reap/Sow use internally, why would this be slower than adding lists, rather than much faster? I didn't understand that part of your Answer. Commented Oct 2, 2019 at 19:43
• @JerryGuern You’re making Mathematica loop a bunch of times rather than letting that happen at the C level when Flatten is called. Basically I’m saying that Sakra’s answer will be the best in all likelihood. In general, you want to maintain a nested List structure when constructing your dataset because this is stored as a linked list. Next best is list of list since that will be pointers. Then when you need it cast to the proper array you’ll want to use. Commented Oct 3, 2019 at 2:52