Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm currently doing work on processing a number of images taken frame-by-frame from a video. As a result, I have a directory of around 16K PNG images that at most ~300K.

That said, I have a routine, analyze which I map in the following way:

analyse /@ FileNames[ "captures\\*.png" , nbDir];

analyse is a Module with calls to a number of image-processing functions that are computationally bound:

That said, I'm heavily computationally bound for the the process I perform on each image, but also heavily disk/IO bound because there are so many images.

To that end, I've started using ParallelMap like so:

ParallelMap[analyse, FileNames["captures\\*.png" , nbDir]];

However, I know of the Method parameter which allows one to specify just how much parallelization should take place in ParallelMap.

My gut instinct here is to go with the CoarsestGrained option for the Method parameter but I can't really put together why the approach would be valid (or invalid, if it were not).

For this situation, what approach can I take to tune this call to ParallelMap in order to get the best performance (to be specific, I'd like it to be done as soon as possible, with a disregard for resource consumption if sacrifices are to be made).

share|improve this question
add comment

2 Answers

up vote 13 down vote accepted

In this case, you would want to favor Method->FinestGrained, not Method->CoarsestGrained.

FinestGrained is more useful when the workload has items that may take a very long time to process, or which may have wildly varying computation times.

CoarsestGrained is useful when the workload consists of many fast, easy to compute items. You don't want to have heavy synchronization and dispatch when you're adding 1 to 10^10 numbers, do you?

I'll add some real examples shortly. But in general:

  • FinestGrained: Computation bound or varying completion time
  • CoarsestGrained: Fast computations. Really fast ones. Communication is expensive so you want to limit it.

You can really view it the two options as tradeoffs between TComm (cost of communication) vs TComp (cost of computation. In other words:

If[TComm > TComp,
    Method->"CoarsestGrained",
    Method->"FinestGraned"]

If you want some more control over the grain, you can utilize Method->ItemsPerEvaluation->n. In this case, we hand out n items to each kernel when they come for work. You can use this to further tweak how fine Mathematica grains your calculations.

As a note, the default option of Method->Automatic (which is implicit any time you use Parallel*) performs load balancing we may be expensive to perform if your workload is better suited to fine or coarse grained.

Remember though -- Always measure!

share|improve this answer
    
I agree, always measure; still at the point where I'm trying to figure this all out. =) –  casperOne Jan 19 '12 at 5:10
    
@casperOne: I'll give you examples soon - I promise! –  Mike Bantegui Jan 19 '12 at 5:14
    
It's turning out to be extremely difficult to find examples where you would prefer finest grained over coarsest grained.. –  Mike Bantegui Jan 19 '12 at 6:00
    
In this case or in general? –  casperOne Jan 19 '12 at 6:01
    
In general. I'm trying all kinds of cases and it's turning out that neither makes a huge difference. I'm finding all the cases where Mathematica does fine using any of the methods. –  Mike Bantegui Jan 19 '12 at 6:25
show 2 more comments

As another approach, you might want to consider using ParallelSubmit and WaitAll or WaitNext for this task:

tasks = Table[ParallelSubmit[{i}, i^2], {i, 100}];

WaitAll[tasks]

which gives {1, 4, 9, ..., 10000}

share|improve this answer
    
Is this applicable to Mike's question on asynchronous evaluation? (I'm out of upvotes.) –  rcollyer Jan 19 '12 at 19:16
    
Do I have to worry about overloading a queue? We're talking submitting tens of thousands of items to be process here, and I don't necessarily need to wait (although that's good to know), for now, I just need to know when done. –  casperOne Jan 19 '12 at 19:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.