I have to run multiple tasks simultaneously:
- Data download from stock exchange during live market session and append it to a global variable.
- Process the data by taking the data from the front of the global variable then push it into another global variable and subsequently write it to a file.
- Run the primary computation by taking the processed data from the front of the global variable and then delete it from the front to free the memory.
Question is, how do I run these three different tasks simultaneously, written in three different shells, so that one operation doesn't get affected/delayed by another. If I use ParallelSubmit
for three tasks then the main kernel remains unused. I want to run the primary computation in the main kernel since it is optimized to use maximum number of available cores whereas parallel kernels are using one core at a time which will slow down the primary computation as it is a time consuming task. Here's some simple code for ParallelSubmit
:
tsk[i_] := If[i == 1,
Do[Pause[i];
Print["task1 ", StringSplit[DateString[AbsoluteTime[]]][[-1]], " ",
j], {j, 1, 10}],
If[i == 2,
Do[Pause[i];
Print["task2 ", StringSplit[DateString[AbsoluteTime[]]][[-1]], " ",
j], {j, 1, 10}],
If[i == 3,
Do[Pause[i];
Print["task3 ", StringSplit[DateString[AbsoluteTime[]]][[-1]],
" ", j], {j, 1, 10}];
];
];
];
DistributeDefinitions[tsk]
StringSplit[DateString[AbsoluteTime[]]][[-1]]
WaitAll[Table[ParallelSubmit[{i}, tsk[i]], {i, 1, 3}]]
ParallelEvaluate
is running sequentially making it ineffective.
My major concern is to assign one of the tasks to the main kernel. Alternatively if I could assign multiple cores to one of the tasks within ParallelSubmit
then the purpose would be served. Ideally one should be able to assign specific kernel and specific core/s (with kernel IDs and core/s IDs) to specific tasks.
Any suggestions?
SetSharedVariable
(and maybeSetSharedFunction
), which should let all subkernels access the same global variable.CriticalSection
is likely also useful for safety. $\endgroup$Run[]
. $\endgroup$