Timeline for Effective use of idle cores when using parallelization
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 18 at 16:31 | vote | accept | seva011 | ||
| Mar 7 at 17:07 | comment | added | seva011 |
In fact, the approach you suggested, might be described as following: Parallelize of the stack {ParallelMap[F1,L1],ParallelMap[F2,L2]}. I did not manage to handle with the syntaxis # $@ yet, pardon. It is likely, I will need ParallelSubmit, it sounds close to what I think of. But it is not clear how to handle it.
|
|
| Mar 7 at 16:50 | comment | added | seva011 |
1) Correct me, you suggested gathering all the functions in a pool and apply ParallelMap to them. Whether we can interrupt ParallelMap and withdraw an intermediate result ? If you have a solution I will appreciate it if you share it. I do utilize Save[...] after each ParallelMap. 2) I am working with a cluster of 8 nodes (PC's) with 20 cores per node. I am not sure, one can safely address the nodes memory while they are in the work process. 3) I do believe one can do it simpler. There has to be a Mathematica command to run the next command if a core is idle after the previous.
|
|
| Mar 7 at 14:37 | comment | added | Domen | Why don't you write the results to a file? | |
| Mar 7 at 12:57 | comment | added | seva011 |
Your response helped me to specify the issue I concerned of: I would like to load the CPU's and keep tasks distinguished to get the output timely when calculations of each task is finished. What you propose should definitely work, thank you ! However, I would like to have access to the output of the finished calculations made with ParallelMap[F1,L1], while ParallelMap[F2,L2] is still being run. Sounds insignificant, but my ParalellMap’s takes about a few weeks each, and I prefer to work with intermediate results, not waiting for the end of the full computations. Thank you !
|
|
| Mar 7 at 11:51 | history | answered | Domen | CC BY-SA 4.0 |