Reputation
70,603
Next tag badge:
215/100 score
20/20 answers
Badges
7 212 301
Impact
~449k people reached

16h
revised Is there a built-in equivalent to Python's enumerate?
edited tags
19h
revised Efficient lazy weak compositions
edited tags
1d
comment Efficient lazy weak compositions
I'd say you did an excellent job here, given that you had to dig a lot of things and there is no documentation. In any case, thanks a bunch for such a great non-trivial test of Streaming functionality! It is great to see this stuff being put to use.
1d
comment Efficient lazy weak compositions
In particular, you should be able to wrap you entire benchmark code in LazyListBlock, and then you shouldn't need any explicit chunk cleanup, since (if everything works correctly), all generated LazyLists will then be automatically destroyed.
1d
comment Efficient lazy weak compositions
One more tidbit is that if you want to destroy some intermediate LazyLists, generated in the process, you can use one of LazyListBlock, LazyListBlockReturn. The former destroys all LazyLists, created inside of it, while the latter keeps the one that is being returned, if it is the value of the code run inside of it, and otherwise works the same as LazyListBlock, destroying all the other LazyList objects, generated inside of it. There is also LazyListBlockProtect, which you can wrap around some LazyLists created inside LazyListBlock, to protect them from destruction.
1d
comment Efficient lazy weak compositions
Virtual chunks have some bug which, for instance, prevents my implementation of LazyTuples (Virtual chunk branch of it) to work correctly. I should really fix it, it is good to be reminded about it. As to chunk destruction, I'd also be grateful for reports in cases when some chunks aren't collected using Scan[LazyListDestroy, LazyLists[]] code.
1d
comment Efficient lazy weak compositions
Also, while using DeleteFile@FileNames["*.mx", $StreamingCacheBase] will indeed delete the chunks from disk, this will leave Streaming in an invalid state. The better way is to use something like Scan[LazyListDestroy, LazyLists[]], which should (if the Streaming garbage collector works correctly, which I hope is true) then delete chunks from disk as a part of the lazy list destruction procedure.
1d
comment Efficient lazy weak compositions
I have to mention, that there are types of chunks called Virtual, for which there is no caching on disk. I did not demonstrate how to use them, and also there seems to be a bug there which I still have to fix, but for example infinite Range is built with those. They can be used any time when you can generate chunks purely by calling some function of the chunk index. In the implementation of LazyTuples, there is a branch that uses virtual chunks. They might be better suited for this case. I will have to revisit the bug there though.
1d
comment Efficient lazy weak compositions
Wow. I am impressed. Really. It will take me some time to absorb that. But I will certainly set out the time and do that. Big +1, of course.
1d
revised Monitor the computation only if a certain Option is given to the function
Ditched the verbose version
1d
comment What kind of creature is h[c] for definition h[c_][x_] := (* function of x and c *)
@murray One useful thing here is to think how would you alternatively implement such a construct. If you want it to have exactly the same semantics, you can do something like h[c_]:=Function[x, (x + c/x)/2]. This reveals that you create a closure, closed over the value of c. So, this is one way to think also about h[c][x] construct when defined with rules, from the functional programming (operational) viewpoint - although the mechanism is technically very different.
1d
comment What kind of creature is h[c] for definition h[c_][x_] := (* function of x and c *)
It is simply an expression which by itself does not get rewritten non-trivially, so stays inert. It is not different from any other expression that doesn't have global rewrite rules applicable to it. You can make it behave as a function, however. Technically, this is realized via SubValues. This allows you to pass the "arguments" collected in h (like 2 in this case), to the body of the "function" (r.h.s. of the rule). This is useful, since it allows you to separate parameters which are fixed externally (behaves like constants, 2 here), from those which are passed dynamically.
1d
comment Monitor the computation only if a certain Option is given to the function
@MarcoB Thanks! Glad you liked it. I hesitated a bit to post it, since I am not fully convinced that the issue with leaking of local symbols can be always easily avoided. But objectively, the general formulation of the problem requires state - sharing by more than one function, so there is no way to avoid this issue in the general formulation of this problem. The problem caught my eye also because I do need similar things myself.
1d
answered Monitor the computation only if a certain Option is given to the function
1d
awarded  string-manipulation
1d
awarded  Enlightened
1d
awarded  Nice Answer
1d
awarded  Great Answer
1d
awarded  Nice Answer
2d
revised How to specify that Quiet should silence foo::shdw messages for ANY symbol foo?
Added a few extra bits and examples