how can I avoid CPU waste (by master kernel) using WaitNext?

Question

I'm using WaitNext in Mathematica for parallel processing on Mac OS X. I have 4 processors on my MacBook and LaunchKernels[] has launched 4 kernels in Mathematica, in addition to the master kernel. However, each of these 4 kernels gets approximately 75-80% of a processor, while the master kernel gets 70-75%. It seems to me that the master kernel should be just waiting, instead of using CPU.

The relevant code is

While[Length[parallelJobs] > 0,
 {result, pJ, parallelJobs} = WaitNext[parallelJobs];
 Write[outputstream, result];
 setofresults = Join[setofresults, result]
]

where parallelJobs is a list of process IDs of the form ParallelSubmit[{...},...], and each of the jobs should take about an hour. Thus, there is no reason for the master kernel to be running. A similar code with WaitAll instead of WaitNext leaves the master kernel with 0% CPU and gives 100% to each of the launched parallel kernels. But the reason I want to use WaitNext instead is because I want to write the results as they're generated. I'd be grateful for any explanation of what's going on or ideas for getting around this waste of CPU. Thanks.

Later:

Here's a fix. I bundle more stuff into the ParallelSubmit[...] parts, in order to Write there, and then I use WaitAll on the list of ParallelSubmit items. This seems to result in considerably less overhead CPU used by the master kernel. It now seems to be an academic question: why does WaitNext (seem to?) use so much more overhead than WaitAll?

Answer 1

This may be system or version dependent, as I don't get the same behavior. On Mathematica 8, OS X, using this line on a fresh kernel:

$ProcessID
WaitAll[{ParallelSubmit[While[True, Null]]}]

I get the same cpu utilization (70% main kernel, 100% parallel kernel), as if WaitAll is replaced by WaitNext.

In the following code, the parallel kernel gets a stack trace from the main kernel while the main kernel is running WaitNext:

seeStack[] := Stack[_];
SetSharedFunction[seeStack];
wait = ParallelSubmit[seeStack[]]
WaitNext[{wait}][[1]]

Based on the output, it looks to me like WaitNext checks for completed processes in a While[True,...] loop, hence the CPU usage. Very curiously, if WaitNext is changed to WaitAll the code does not terminate. If other expressions are submitted between the third and fourth line, they will execute normally (though I won't see the results because seeStack[] does not terminate).

My guess is that WaitAll is somehow integrated into the kernel and your implementation is more efficient than mine.

I learned about SetSharedFunction from this answer.