# FinancialData anomaly

As I learned from this quantitative finance question, 1981-04-17 was a trading holiday. This is in agreement with the 0 volume in that day

  "Volume", {"1981-04-17", "1981-04-17"}]["Values"]


gives {Quantity[0, "Shares"]}

However looking at the (close) prices, via

DateListPlot[
FinancialData["NASDAQ:AAPL", "Close", {"1981-04-10", "1981-04-20"}],
PlotRange -> All]


I see an abnormally large move

Would love to know why Mathematica gives such a high price on a non-trading day. Thanks.

## Possible Reasons

It might be because Mathematica does not know that "1981-04-17" is a non-trading day when building the dataset.

BusinessDayQ[DateObject["1981-04-17"],HolidayCalendar->"UnitedStates"] (*True*)


Or it might be an error from an old dataset that Mathematica acquired.

## Examples of another anomalous data

It is interesting to note that "IBM" has a price even though it didn't trade, whilst "GE" is unaffected.

FinancialData["IBM", "OHLCV", {"1981-04-10", "1981-04-20"}] // Normal
FinancialData["GE", "OHLCV", {"1981-04-10", "1981-04-20"}] // Normal


Some firms have workflows to find and correct these errors in their datasets. Either through user submission or algorithmically. For example, Yahoo Finance does not have this error.

## Data Cleaning

This highlights the importance of having a process to clean and sanity-check any data we work with. For example, we can filter to select only days where "Volume" is larger than zero. From there we can create a new TimeSeries that handles missing dates by using the value of the previous date (to avoid introducing lookahead bias to the dataset). This can be done with the option ResamplingMethod ie TimeSeries[cleanedData, ResamplingMethod-> {"Interpolation", InterpolationOrder -> 0}].

First, we need to convert FinancialData into something we can easily manipulate. Using the workflow for converting FinancialData into a flat list of assoications from my answer here:

(*helper function*)
joinColumnToData[column_?VectorQ,data_?VectorQ]:=Transpose@{column,data};
joinColumnToData[column_?VectorQ,data_?MatrixQ]:=ArrayFlatten@{{List/@column,data}};

(*helper function*)
Options[convertFinancialDataToFlatDatePath]={"DateConversionFunction"->Function[{date},DateString[date,"ISODate"]]};
convertFinancialDataToFlatDatePath[timeSeries_TemporalData,OptionsPattern[]]:=Module[{datesRaw,valuesRaw,dates,values},datesRaw=timeSeries["Dates"];
valuesRaw=timeSeries["Values"];
dates=OptionValue["DateConversionFunction"]/@datesRaw;
values=QuantityMagnitude@valuesRaw;
joinColumnToData[dates,values]];

(* set data universe *)
tickers={"AAPL","IBM","GE"};
dataStart=DateString["1981-04-10","ISODate"];
dataEnd=DateString["1981-04-20","ISODate"];

tickerProperties="OHLCV";
tickerPropertiesHeader={"Open","High","Low","Close","Volume"};(*change if using something other than "OHLCV"*)

(*get list of TimeSeries from Mathematica*)
financialDataRaw=FinancialData[tickers,tickerProperties,{dataStart,dataEnd}];

(*format it*)
financialDataFlat=convertFinancialDataToFlatDatePath/@financialDataRaw;



Secondly, we determine the steps we are going to use to clean and sanity check the raw data. In this case we select only days with positive trading volume.

(*data cleaning*)
financialDataClean = Select[financialData, #Volume > 0 &];


Finally, we convert the clean data into a new TimeSeries object, letting the TimeSeries function handle the missing dates we removed.

dataByDate=Values/@Query[GroupBy[#Date&],#[[All,{"Open","High","Low","Close","Volume"}]]&]@financialDataClean;
formatDataForTimeSeries=Transpose[{Keys@dataByDate,Values@dataByDate}];

cleanTimeSeries =
TimeSeries[
formatDataForTimeSeries,
ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0},
MetaInformation -> {"ComponentNames" -> tickers}
];


Also note that formatting the TimeSeries data in this way allows us to make use of the MetaInformation option to call the TimeSeries we want by ticker.

This is incredibly useful functionality when dealing with multiple assets.

cleanTimeSeries["PathComponent", "AAPL"]["1981-04-16"]
cleanTimeSeries["PathComponent", "AAPL"]["1981-04-17"]
cleanTimeSeries["PathComponent", "AAPL"]["1981-04-20"]

(*
{0.111607, 0.112168, 0.111607, 0.111607, 106600.}
{0.111607, 0.112168, 0.111607, 0.111607, 106600.}
{0.114957, 0.115514, 0.114957, 0.114957, 157800}
*)