Manipulate not working as advertised

Question

I have gone through the reference titled Advanced Manipulate Functionality, and I quote from the section titled Dealing with Slow Evaluations:

There is no five-second limit to such nonblocking evaluations, so by using the ContinuousAction -> False option, arbitrarily long evaluations can be used.

This appears to be incorrect. Even with ContinuousAction -> False my Manipulate evaluation is aborting after a few seconds. My only recourse is to pre-compute everything, but this consumes a lot of memory.

Unfortunately, I cannot provide a direct sample of the code, because it involves loading large CSV files residing on my disk. Each selection of Manipulate should have loaded all the files in the selected directory and done some computations on them. Instead, I have to cycle through the directories and load all the files first, and then call the loaded lists in Manipulate. An individual file takes about 10 to 15 seconds to load, but the entire collection takes over 20 minutes. It would have been better if I could have loaded the files on demand.

Are there any other hidden options for Manipulate that I might use here?

EDIT

In response to @Edmund's request, I am providing a pared-down version of the code. First, the helper functions:

ClearAll[customManipulate, nestedManipulate, loadCSVFiles,];

customManipulate[cf_, lbl_] := With[{d = #, k = Sort@Keys@#}, Manipulate[
    cf[d[s]], {{s, First@k, Style[lbl, "Subsubsection"]}, k},
    Paneled -> False, FrameMargins -> None, ControlType -> PopupMenu,
    ContinuousAction -> False
]] &;

nestedManipulate[ifn_, lbls_] := Fold[customManipulate, ifn, Reverse@lbls];

loadCSVFiles = RightComposition[
    FileNames["*.csv", #] &, AssociationMap[Import[#, "CSV"] &], KeyMap[FileBaseName]
];

The function nestedManipulate is simply a convenience function that allows me to call customManipulate on a deeply nested Association.

Now, say that I have a folder called csv-dat, which contains CSV files, and a data-processing function dfn. I can use the following lines of code to view the files:

{"path", "to", "csv-dat"} // FileNameJoin // loadCSVFiles // customManipulate[dfn, "File"]

This should give me a Manipulate view from where I can select the particular file whose result I want to view.

I use this setup quite often in internal demonstrations. Let us say that the folder csv-dat has about 600 files, and the function dfn takes about 10 seconds to run on each file. During the demonstration, I don't mind waiting for 10 seconds each time, especially since I can talk about other aspects of the data in the meantime. But I don't want to wait for over 6,000 seconds to preprocess every file before starting the meeting. Nor can I necessarily preprocess only a small subset, because at various times, different teams want to view different files (usually based on their names).

The other thing is that often I use more than one data processing function. And I want to be able to select between dfn1 and dfn2 without needing to preprocess everything.

So, this is the predicament I face. And I would be grateful if someone could help me find a set-up so that customManipulate did not abort.

Answer 1

The issue you are bumping into is the action taken after the dynamic variable is updated takes a very long time.

ContinuousAction -> False makes Manipulate wait until user is not manipulating the control before it updates the dynamic variable's value. This prevents continuous updates but does not prevent the time-out of long running evaluations once the dynamic variable is updated.

SynchronousUpdating -> False sets the evaluation that occurs after the dynamic variable is updated to run asynchronously. This prevents long running evaluations from timing out.

You need both of the above options set on your Manipulate. Add SynchronousUpdating -> False to the list of options for the Manipulate in the customManipulate function.

Hope this helps.

Answer 2

The key is what is "nonblocking."

The following works no matter what you move the slider to:

Manipulate[Pause[x]; x, {x, 0, 10, Appearance -> "Labeled"}, 
 ContinuousAction -> False]

This does not, because the initial evaluation of the body, which blocks the Front End, times out (aborts).

Manipulate[Pause[7.62]; x, {x, 0, 10, Appearance -> "Labeled"}, 
 ContinuousAction -> False]