2
$\begingroup$

I can get weather station location:

WMOStations = ResourceObject["WMO Meteorological Stations"];

and select for the US:

cdata = Values@
  ResourceData[WMOStations][Select[#Country == "United States" &]]

Define contiguous states:

kstates = {"AL", "FL", "GA", "MS", "TN", "AR", "LA", "MO", "OK", "TX",
    "AZ", "CA", "NM", "NV", "UT", "OR", "CO", "KS", "NE", "WY", "CT", 
   "MA", "NY", "RI", "DC", "MD", "VA", "DE", "NJ", "PA", "NC", "SC", 
   "IA", "IL", "MN", "SD", "WI", "ID", "MT", "WA", "IN", "KY", "MI", 
   "OH", "WV", "NH", "VT", "ME", "ND"};

Select stations for those states:

data1 = Table[
   cdata[Select[#StateCode == kstates[[i]] &], "Position"], {i , 
    Length[kstates]}];
WMOLocationsUS = Flatten[Normal[data1]];
WMOLocationsUSc = DeleteDuplicates[WMOLocationsUS];

This gives me 397 Weather stations

From these 397 stations, I want to get daily MeanTemperature for the last 10 years. Here's what I did:

weatherstation = Map[WeatherData, WMOLocationsUSc]; 
meanwdata = 
  Table[{weatherstation[[i]], 
    WeatherData[weatherstation[[i]], 
     "MeanTemperature", {{2010, 1, 1}, {2020, 12, 31}, "Day"}]}, {i, 
    1, Length[weatherstation]}];

The problem is that I get lots of missing values and lots of dates where there is no data. I tried to interpolate the time series and it kinda works but for certain cases, I have to extrapolate and the result is not good.

enter image description here

So, is there a way to filter the weather stations that can give me consistent data? I am fine with have about 300 weather stations.

$\endgroup$
5
  • $\begingroup$ Why not just exclude stations that have more than some threshold percent of missing data, or ones where extrapolation is needed. Does that result in less than 300? $\endgroup$ Jan 28 at 2:50
  • $\begingroup$ @RohitNamjoshi ya I will lose lots of data that way (I am will have stations << 300). $\endgroup$
    – psimeson
    Jan 28 at 4:45
  • $\begingroup$ I'm confused. What do you mean by consistent data? The data from the weather stations is not inconsistent. The missing observations are "real world" events. There's no explanation of individual missing observations, but power failures, sensor maintenance problems, and storms are likely reasons. Every station will have different missed observations, but this does not mean their data is inconsistent. Some stations may be more reliable than others in that they have fewer missed observations. Do you want to ignore stations that have a large proportion of missing observations? $\endgroup$
    – creidhne
    Jan 31 at 18:27
  • $\begingroup$ Thanks @creidhne for the clear explanation for the reasons behind missing data. All I am looking for is about 250-300 weather stations that have data for everyday for 5 years or so. That's what I was trying to mean by consistency. I kinda tried to fill in some data by interpolation but as shown above, it didn't work. Sorry for the ambiguity. $\endgroup$
    – psimeson
    Feb 1 at 4:07
  • $\begingroup$ Is the issue just that the extrapolation breaks down? You can fix that in Interpolation by using the "ExtrapolationHandler" argument $\endgroup$
    – b3m2a1
    Feb 5 at 4:22
1
$\begingroup$

Using meanwdata, first, we need to remove stations that have no observations. Let's define beginning- and end-dates so we can find the largest number of observations that a station might have. Use a short list of stations for demonstration; otherwise use weatherstations instead of sampleStations.

(*Note: 2011-1-1 to 2020-12-31 is 10 years of daily observations*)
{startDate, endDate} = {{2011, 1, 1}, {2020, 12, 31}};
observations = 
  QuantityMagnitude[DateDifference[startDate, endDate]] + 1;
sampleStations = {Entity["WeatherStation", "KAYS"], 
   Entity["WeatherStation", "KAQR"], 
   Entity["WeatherStation", "KSSI"], 
   Entity["WeatherStation", "KMRF"], 
   Entity["WeatherStation", "KSVN"]};
meanwdata = {#, WeatherData[Entity["WeatherStation", #],
   "MeanTemperature", {startDate, endDate, "Day"}]} & /@ 
      sampleStations; (*use weatherstations for all stations*)

Remove stations that have no observations. DeleteMissing removes 1 of the sample stations from meanwdata.

meanwdata = DeleteMissing[meanwdata, 1, Infinity];
EntityValue[meanwdata[[All, 1]], "Name"]
(*{"KAYS", "KSSI", "KMRF", "KSVN"}*)

One way to filter meanwdata is to set a threshold for the number of missing observations. We might want stations that have 95% or more of the expected number of observations. This eliminates station KMRF from the sample stations:

Select[meanwdata, Last[#]["PathLength"]/observations > .95 &]

Missing Daily Observations

If we define consistent to mean that multiple stations have common sets of observation dates, I don't think there will be a significant number of stations that meet that requirement. There will be few, if any, weather stations that have aligned data for every day for 5 years, or any significant number of consecutive days.

Missing observations are caused by real-world events. Some examples of these events are power failures, equipment malfunctions, storms, and downtime for repair, calibration or maintenance. We can inspect the number of consecutive daily observations with two functions:

dateIntervals[dates_?ListQ] := 
 DateInterval /@ 
  DateBounds /@ 
   Split[dates, Subtract /* EqualTo[Quantity[-1, "Days"]]]
intervalsToDurations[intervals_?ListQ] := 
 Floor@#["Duration"] & /@ intervals + Quantity[1, "Days"]

Here's a list of the longest and shortest number of consecutive daily observations for the sample stations:

TableForm[
 Table[Flatten@{First[st]["Name"], 
    MinMax[intervalsToDurations[dateIntervals[Last[st]["Dates"]]]]},
  {st, meanwdata}],
 TableDepth -> 2, 
 TableHeadings -> {None, {"station", "shortest", "longest"}}]
station shortest longest
KAYS 22 days 1206 days
KSSI 133 days 1072 days
KMRF 1 days 510 days
KSVN 96 days 1434 days

The KSVN station has the longest number of consecutive daily observations, about 4 years, but the other stations have far fewer consecutive observations. The observation intervals for the stations are not likely to align with other stations. Here's how the intervals "stack up" for the sample stations (I've added the stationTimeline function below). You can mouse-over the intervals to see the start and end dates.

Summary

These timelines show that although stations have intervals of consecutive daily observations, no single station's observations align with the observations from other stations. The real-world observations don't happen in long, consecutive, daily intervals. It might be helpful to select stations that have useful intervals by using dateIntervals and intervalsToDurations. Interpolated observations are necessary to fill in for missing daily values.

Table[
  stationTimeline[s], {s, meanwdata}] // Column

observation timelines

Functions for Timeline Plots

timelines[timeSeries_] := 
 Module[{d1, d2, dr, missingDates, observationDates},
  {d1, d2} = DateBounds[timeSeries];
  dr = DateRange[d1, d2];
  missingDates = Complement[dr, timeSeries["Dates"]];
  observationDates = Complement[dr, missingDates];
  {dateIntervals[missingDates], dateIntervals[observationDates]}
]

stationTimeline[stationData_?ListQ] := 
 Module[{ts, nm, missingDateIntervals, observationDateIntervals, 
   intervals, min, max},
  ts = Last@stationData;
  nm = First[stationData]["Name"];
  If[MissingQ@ts, Return["No data for " <> nm]];
  {missingDateIntervals, observationDateIntervals} = timelines[ts];
  intervals = 
   observationDateIntervals;(*or missingDateIntervals*)
  {min, max} =
    MinMax[intervalsToDurations[intervals]];
  TimelinePlot[intervals, PlotLayout -> "Stacked", 
   PlotLabel -> 
    ToString[ts["PathLength"]] <> " Observations for " <> nm <>
     "\nLongest interval: " <> ToString[max] <>
     ", Shortest interval: " <> ToString[min]
  ]
]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.