4
$\begingroup$
Normal[WeatherData[
                WeatherData[
                 Entity["City", {"Birmingham", "Alabama", "UnitedStates"}]], 
                "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]][[All, 
               2]] // Length

Gives 730 which makes sense because DateObject[{2020, 12, 31}] - DateObject[{2019, 1, 1}]=730

but

2.

Normal[WeatherData[
    WeatherData[
     Entity["City", {"Huntsville", "Alabama", "UnitedStates"}]], 
    "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]][[All, 
   2]] // Length

Gives 731, why?

3.

 Normal[WeatherData[
    WeatherData[
     Entity["City", {"Auburn", "Alabama", "UnitedStates"}]], 
    "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]][[All, 
   2]] // Length

Gives 681, why?

4.

Normal[WeatherData[
    Entity["City", {"Auburn", "Alabama", "UnitedStates"}], 
    "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]][[All, 
   2]] // Length

Gives 731, why?

Is there a way to get consistent data i.e. 730 temperature data for each city in the US?

$\endgroup$
4
  • 7
    $\begingroup$ WeatherData often has missing data. I showed how to identify what days are missing in this post on the Wolfram Community site. $\endgroup$ Commented Jan 23, 2021 at 0:44
  • $\begingroup$ Thanks @RohitNamjoshi your approach will certainly come in handy. $\endgroup$
    – sra
    Commented Jan 23, 2021 at 20:17
  • $\begingroup$ @RohitNamjoshi is there a way to just get a list of missing dates rather than date intervals? $\endgroup$
    – sra
    Commented Jan 23, 2021 at 21:08
  • 1
    $\begingroup$ You might use Complement[DateRange[{2019, 1, 1}, {2020, 12, 31}], DateList /@ WeatherData[Entity["WeatherStation", "KAUO"], "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]["Dates"]] to get a list of missing dates. $\endgroup$
    – creidhne
    Commented Jan 23, 2021 at 21:53

1 Answer 1

4
$\begingroup$

The results given by WeatherData are not inconsistent. WeatherData gives different results for different location specifications, and that is what's happening in your examples. Additionally, individual weather stations may not have data for every date.

Here's how WeatherData works for different location specifications:

  • When the location in WeatherData[loc, ...] is a weather station identifier, the result is the data for the weather station.

  • When the location is a city, given as WeatherData[loc], the result is the nearest weather station to the city for which data has ever been available.

  • When the location is a city, given as WeatherData[loc, ...], the result is data for the nearest high-reliability weather station.

This means that if you want the "best" data for a city, use the city as the location, not the nearest weather station.

Why are there different numbers of data samples?

Let's use Auburn as an example. When the location is a city, WeatherData[city] gives the nearest station, but WeatherData[city, ...] returns data for the nearest high-relability station. For example,

city = Entity["City", {"Auburn", "Alabama", "UnitedStates"}];
nearestStation = WeatherData[city];

This gives the nearest weather station to Auburn, which (currently) is KAUO, and WeatherData for the station is:

WeatherData[nearestStation, "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]["PathLength"]

which shows that station KAUO has 681 temperature samples for the dates. Instead, when the location is a city, WeatherData returns data for the nearest high-relability station.

WeatherData[city, "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}]["PathLength"]

which shows that the nearest high-reliability station has 731 temperature samples.

$\endgroup$
8
  • $\begingroup$ Thanks @creidhne for clearly explaining it. Even though there are 730 days with that time span, why am I getting 731 data points? For the kind of data set that I am trying to create, I would like to have exactly the same number of data points for all the cities. So, are there ways to do so? $\endgroup$
    – sra
    Commented Jan 23, 2021 at 19:12
  • $\begingroup$ @psimeson Ah, yes, but Today - Yesterday is 1 day, yet there are two samples, one for today, and one for yesterday, which shows that there is 1 more sample than the difference in the dates. WeatherData returns a TimeSeries. When there are missing date samples, you can interpolate values for the missing dates with ts = TimeSeries[WeatherData[...], TemporalRegularity -> True]. $\endgroup$
    – creidhne
    Commented Jan 23, 2021 at 20:05
  • $\begingroup$ @psimeson I'll update the answer with some extra examples, but to get the full range of dates, you need ts[dates] where dates is a range from a start date to an end date. See the Basic Uses section of TimeSeries for more methods, e.g., TimeSeriesRescale, TimeSeriesInsert, etc. $\endgroup$
    – creidhne
    Commented Jan 23, 2021 at 21:29
  • $\begingroup$ thanks again. That would be helpful. I am also looking more into it. I have about 1250 US cities and for most of them I get 731 but there are a few where I get <731. I am trying to fix that. I looked to TimeSeriesResample with interval 1 but Mathematica crashed. $\endgroup$
    – sra
    Commented Jan 23, 2021 at 21:36
  • $\begingroup$ city = Entity["City", {"Auburn", "Alabama", "UnitedStates"}]; nearestStation = WeatherData[city]; ts = TimeSeries[ WeatherData[nearestStation, "MeanTemperature", {{2019, 1, 1}, {2020, 12, 31}, "Day"}], TemporalRegularity -> True]; md1 = Complement[DateRange[{2019, 1, 1}, {2020, 12, 31}], DateList /@ td["Dates"]]; nl1 = Table[{md1[[i]], ts[md1[[i]]]}, {i, Length[md1]}] nll1 = TimeSeriesInsert[ts, nl1]; Intersection[Normal[nll1][[All, 2]][[All, 1]], Normal[td][[All, 2]][[All, 1]]] // Length -- > Should be 681 but it' s 362 $\endgroup$
    – sra
    Commented Jan 23, 2021 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.