I have to run multiple tasks simultaneously:

  1. Data download from stock exchange during live market session and append it to a global variable.
  2. Process the data by taking the data from the front of the global variable then push it into another global variable and subsequently write it to a file.
  3. Run the primary computation by taking the processed data from the front of the global variable and then delete it from the front to free the memory.

Question is, how do I run these three different tasks simultaneously, written in three different shells, so that one operation doesn't get affected/delayed by another. If I use ParallelSubmit for three tasks then the main kernel remains unused. I want to run the primary computation in the main kernel since it is optimized to use maximum number of available cores whereas parallel kernels are using one core at a time which will slow down the primary computation as it is a time consuming task. Here's some simple code for ParallelSubmit:

tsk[i_] := If[i == 1,
    Print["task1 ", StringSplit[DateString[AbsoluteTime[]]][[-1]], " ", 
     j], {j, 1, 10}],
   If[i == 2,
      Print["task2 ", StringSplit[DateString[AbsoluteTime[]]][[-1]], " ",
        j], {j, 1, 10}],
     If[i == 3,
         Print["task3 ", StringSplit[DateString[AbsoluteTime[]]][[-1]], 
          " ", j], {j, 1, 10}];
WaitAll[Table[ParallelSubmit[{i}, tsk[i]], {i, 1, 3}]]

ParallelEvaluate is running sequentially making it ineffective.

My major concern is to assign one of the tasks to the main kernel. Alternatively if I could assign multiple cores to one of the tasks within ParallelSubmit then the purpose would be served. Ideally one should be able to assign specific kernel and specific core/s (with kernel IDs and core/s IDs) to specific tasks.

Any suggestions?

  • 1
    $\begingroup$ hope this question gets some attention; parallelism in MMA is tricky! I think it should be retitled, though: something like "How do I run parallel processes without tying up the main kernel?" or "How do I allow one process to take advantage of all available cores while other processes run in parallel?" Basically, something that gets at your "major concern" at the bottom. I think the question in the title can be resolved with SetSharedVariable (and maybe SetSharedFunction), which should let all subkernels access the same global variable. CriticalSection is likely also useful for safety. $\endgroup$
    – thorimur
    Sep 14, 2021 at 4:03
  • $\begingroup$ @thorimur: changed the title based on your suggestion $\endgroup$
    – csk 7
    Sep 14, 2021 at 5:40
  • $\begingroup$ @csk7 The simplest way is playing with processor core priority to boost certain kernel that already created. You can use the Operation System tools for it through the Run[]. $\endgroup$
    – Rom38
    Sep 22, 2021 at 6:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.