1
$\begingroup$

I'd like to be able to create a list of max temperature TimeSeries for the 300 stations closest to KAVL from 2007 to 2017 in a time efficient manner -- a few seconds would be nice.

I'm finding nearby stations using

nearavl = WeatherData[{"KAVL", 300}];

and then mapping that list using

max = (List[{#["Latitude"], #["Longitude"]}, 
  WeatherData[#["Name"], 
   "MaxTemperature", {{2001}, {2017}, "Year"}]])&/@nearavl;

to generate a list of coordinates paired with yearly max temperature TimeSeries.

This works but is terribly slow on my machine (takes around five minutes). Is there a way to speed things up?

EDIT: It seems that the bottleneck was, by and large, CPU use. When I used ParallelTable, the procedure was sped up significantly. I watched network usage in a monitoring program during the time the code was running and didn't see downloads go above 100 KB/s. After a while, downloading tailed off completely, but CPU usage was still near 100% for a while.

$\endgroup$
  • $\begingroup$ I think that the speed of this task is limited by how fast it is to download the data from the wolfram servers. There is probably not much that you can do to speed things up, parallelizing the downloads as suggested might help. Another thing that would at least help when you are rerunning this is to cache data so you don't have to redownload everytime you are needing it. But to some extend the WeatherData-framework will do that already... $\endgroup$ – Albert Retey Mar 7 '18 at 14:20
  • 1
    $\begingroup$ Yes, there was quite a bit of variance in run times, which I suspect is due to caching. However, quitting and restarting Mathematica did not lead to consistently reproducible results. Any idea how Mathematica caches WeatherData? $\endgroup$ – Lee Mar 7 '18 at 15:07
1
$\begingroup$

I created a list of 300 stations near KAVL and ran the following:

Table[{WeatherData[s,"Coordinates"], WeatherData[s,"MaxTemperature",
  {{2007},{2017},"Year"}]}, {s, stations}]

It runs in about 39 seconds. When I ran the same code with ParallelTable[], it took about 20 seconds. That's on a 4 core machine. If you have a multicore machine, ParallelTable[] should reduce the run time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.