I have a program that makes many http requests (gathering data from APIs), and computes on that data. Speed is a priority and using the most up to date data in the computations is a priority. Repeating these tasks as fast as they can be repeated is the goal. Gather then compute and repeat as fast as possible.
The http requests are parallelized to speed up the data gathering. I put that data in a variable that is overwritten. I have that as one scheduled task.
I then want to have 12 scheduled tasks that take that data and compute on it in 12 different ways but due to the computation time, would always like the computations have the most up to date data. Simple reason to not map on the Computation function as during that computation time, data might have been updated.
task1=SessionSubmit[ScheduledTask[Computation1[data,args],5]] task2=SessionSubmit[ScheduledTask[Computation2[data,args],5]] ...
When I run all of this, Mathematica crashes. I would even like to parallelize the work of the computations but that really crashes Mathematica.
paralleltask1=SessionSubmit[ScheduledTask[ParallelizedComputation1[data,args],5]] paralleltask2=SessionSubmit[ScheduledTask[ParallelizedComputation2[data,args],5]] ...
Is there a limit to how many scheduled tasks? I have tried the
LocalSubmit function to create a new thread but that function is not working.