6
$\begingroup$

I have a process that returns an unpredictable number of data points, and I'd like to run it repeatedly until I have a certain number of points.

My actual code is too complicated to use an illustration, so I wrote this toy example. fakeData[] will return 1-21 data points, and I want to run it until I have at least 100. But this code doesn't work because you can't take the so-far Length[] of a list that you're still building.

fakeData[n_] := RandomReal[1, 1 + RandomInteger[n]];
big = Reap[
   While[Length[big] < 100, (* this doesn't work*)
    Sow[fakeData[20]]]][[2, 1]]

I could just allocate 'big' as a Table with length 100 and copy each new small list into it, but then I'd have to discard some perfectly good data points I laboriously calculated, which is distasteful. Is there a better way?

Improve this question
$\endgroup$
2
  • $\begingroup$ Will NestWhile[(Join[#, fakeData[20]]) &, {}, Length[#] < 100 &] work for you? $\endgroup$
    – Ray Shadow
    Commented Jul 24, 2019 at 20:19
  • $\begingroup$ @Shadowray That will work, but I've been told that using Join[] repeatedly is very, very slow because of all the recopying necessary every time you add a chunk. $\endgroup$
    – Jerry Guern
    Commented Jul 25, 2019 at 0:05

3 Answers 3

Reset to default
7
$\begingroup$

How about:

SeedRandom[1]
Reap[NestWhile[Join[#, Sow@fakeData[20]] &, {}, LessThan[100]@*Length]][[2, 1]]

{{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649}, {0.337261, 0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, 0.964289}, {0.741987, 0.819242}, {0.539713}, {0.012502, 0.439595, 0.169709, 0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584}, {0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, 0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, 0.245655, 0.562932}, {0.370393, 0.934574, 0.550753, 0.136193, 0.390665, 0.941924, 0.743334, 0.296465}, {0.114065, 0.612737, 0.596194, 0.32461, 0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, 0.129602}, {0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, 0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, 0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614}, {0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, 0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, 0.315747, 0.367579, 0.521331, 0.36944, 0.566759}}

Another similar possibility:

SeedRandom[1]
Reap[NestWhile[Length @ Sow @ fakeData[20] &, 0, LessThan[100] @* Plus, All]][[2, 1]]

same answer

Improve this answer
$\endgroup$
3
  • $\begingroup$ Okay, thank you, that seems to do what exactly I wanted, now I just have to study docs for a while to understand how/why it works. :-) May I ask, why did you put that SeedRandom[1] in there? I don't see it's purpose, but I assume you had expert-level reasons. $\endgroup$
    – Jerry Guern
    Commented Jul 23, 2019 at 18:26
  • $\begingroup$ Because fakeData calls RandomReal and RandomInteger and these random functions can have reproducible results if you specify the seed. $\endgroup$
    – rhermans
    Commented Jul 23, 2019 at 18:28
  • $\begingroup$ One issue with this is that it doubles the memory cost, no? With large Reaps that could be prohibitive $\endgroup$
    – b3m2a1
    Commented Jul 23, 2019 at 20:15
6
$\begingroup$

The straight forward solution is to simply count the number of points you have sown, i.e.:

big = Module[
    {count = 0},
    Reap[
        While[ count < 100, count += Length@Sow[fakeData[20]] ]
    ][[2,1]]
]
Improve this answer
$\endgroup$
2
  • $\begingroup$ Sorry, I misread your question. See updated answer. $\endgroup$
    – sakra
    Commented Jul 23, 2019 at 18:31
  • 1
    $\begingroup$ Oh, I see, you're right, I can just manually track the length as I as to it. Thanks. $\endgroup$
    – Jerry Guern
    Commented Jul 23, 2019 at 22:26
6
$\begingroup$

Here's a method that just uses Bag since I think effectively that's what Reap and Sow are using. It's probably a bit slower than adding the lists directly and flattening after, but it's conceptually how you were thinking about the original problem:

bag = Internal`Bag[];
SeedRandom[1]
While[Internal`BagLength[bag] < 100,
  Internal`StuffBag[bag, #] & /@ fakeData[20]
  ];
Internal`BagPart[bag, All]

{0.00683794, 0.0936818, 0.474619, 0.310422, 0.153631, 0.31649, 0.337261, \
0.470877, 0.32728, 0.124887, 0.113682, 0.988692, 0.970078, 0.908979, \
0.964289, 0.741987, 0.819242, 0.539713, 0.012502, 0.439595, 0.169709, \
0.771071, 0.998221, 0.179295, 0.901812, 0.661701, 0.162254, 0.85584, \
0.00132041, 0.784942, 0.693806, 0.687592, 0.525913, 0.842108, 0.203219, \
0.495244, 0.909835, 0.464522, 0.115059, 0.443676, 0.712994, 0.439824, \
0.245655, 0.562932, 0.370393, 0.934574, 0.550753, 0.136193, 0.390665, \
0.941924, 0.743334, 0.296465, 0.114065, 0.612737, 0.596194, 0.32461, \
0.713441, 0.225573, 0.387218, 0.55637, 0.336226, 0.90315, 0.333871, 0.188398, \
0.129602, 0.265823, 0.750065, 0.757875, 0.679856, 0.0740267, 0.691003, \
0.571181, 0.921954, 0.559011, 0.341209, 0.757399, 0.856246, 0.578542, \
0.866321, 0.641392, 0.474307, 0.197374, 0.172371, 0.448029, 0.122614, \
0.146429, 0.0648023, 0.514557, 0.320289, 0.510485, 0.00828315, 0.346533, \
0.0588742, 0.436849, 0.305532, 0.767718, 0.254158, 0.345529, 0.208461, \
0.315747, 0.367579, 0.521331, 0.36944, 0.566759}
Improve this answer
$\endgroup$
3
  • $\begingroup$ Oh, thank you. SE didn't notify me of this Answer for some reason, so I just saw it today. Yes, this is conceptually what I was trying to do, and I appreciate this glimpse into what MMA is doing 'under the hood'. $\endgroup$
    – Jerry Guern
    Commented Oct 2, 2019 at 19:41
  • $\begingroup$ I'm wondering though, if Bag is what Reap/Sow use internally, why would this be slower than adding lists, rather than much faster? I didn't understand that part of your Answer. $\endgroup$
    – Jerry Guern
    Commented Oct 2, 2019 at 19:43
  • $\begingroup$ @JerryGuern You’re making Mathematica loop a bunch of times rather than letting that happen at the C level when Flatten is called. Basically I’m saying that Sakra’s answer will be the best in all likelihood. In general, you want to maintain a nested List structure when constructing your dataset because this is stored as a linked list. Next best is list of list since that will be pointers. Then when you need it cast to the proper array you’ll want to use. $\endgroup$
    – b3m2a1
    Commented Oct 3, 2019 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.