3
$\begingroup$

I have a program that makes many http requests (gathering data from APIs), and computes on that data. Speed is a priority and using the most up to date data in the computations is a priority. Repeating these tasks as fast as they can be repeated is the goal. Gather then compute and repeat as fast as possible.

The http requests are parallelized to speed up the data gathering. I put that data in a variable that is overwritten. I have that as one scheduled task.

SessionSubmit[ScheduledTask[data=ParallelizedHttpRequests[args],2]]

I then want to have 12 scheduled tasks that take that data and compute on it in 12 different ways but due to the computation time, would always like the computations have the most up to date data. Simple reason to not map on the Computation function as during that computation time, data might have been updated.

task1=SessionSubmit[ScheduledTask[Computation1[data,args],5]]
task2=SessionSubmit[ScheduledTask[Computation2[data,args],5]]
...

When I run all of this, Mathematica crashes. I would even like to parallelize the work of the computations but that really crashes Mathematica.

paralleltask1=SessionSubmit[ScheduledTask[ParallelizedComputation1[data,args],5]]
paralleltask2=SessionSubmit[ScheduledTask[ParallelizedComputation2[data,args],5]]
...

Until the sockets is fixed, see here and here repeated http requests is the only way to get the data.

Is there a limit to how many scheduled tasks? I have tried the LocalSubmit function to create a new thread but that function is not working.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.