34
$\begingroup$

In the recent new in Mathematica 11 webinar one user asked during the Q&A session wether the current implementation of neural networks in Mathematica 11 could be used for processing time series. As far as I remember Taliesin Beynon answer was something along the line of

Recurrent Neural Networks will eventually come to Mathematica in a later version. Until then one could use a Convolutional Neural Network on a time series as if it were an image and "fake" the extra dimensions

I'd like to do exactly that but am unsure how to proceed. Take for an (somewhat silly) example stock prices of various companies from either tech or financial industry from 2007 to now. One interesting question might be wether a network can be trained to discern which industry a company belongs to based on its stock's course over the years (including the 2008/2009 financial crisis). The stocks I am looking at are

tech = {"GE", "AAPL", "MSFT", "MRK", "IBM"};
banks = {"DB", "C", "BAC", "HSBC", "BCS"};

A display of the stock prices (resized to values between 0 and 1) can be generated with 

Callout[FinancialData[#, "Jan. 1, 2007"] // 
 Replace[Transpose@#, {a_, b_} :> 
    Transpose@{a, Rescale@b} ] &, #] & /@ tech // DateListPlot

stocks of tech companies

Callout[FinancialData[#, "Jan. 1, 2007"] // 
 Replace[Transpose@#, {a_, b_} :> 
    Transpose@{a, Rescale@b} ] &, #] & /@ banks // DateListPlot

stocks of financial companies

For the network I'd like to use 100 input nodes (corresponds to a $1\times 100$ or $100\times 1$ pixel image). The following function gives a downsampled version of a stock course with 100 data points and stock prices rescaled to values between 0 and 1.

financialDataToList[stock_] := 
FinancialData[stock, "Jan. 1, 2007"] // Transpose // Last // 
Rescale // ArrayResample[#, 100] &

For visualization ArrayPlot can be used

myarrayplot[stock_] := 
{financialDataToList@stock // List //ArrayPlot[#, AspectRatio -> 1/4] &, stock} 

Grid[#, Alignment -> Left] & @*Map[myarrayplot] /@ {tech, banks} // 
List // Grid[#, Spacings -> {5, 1}] &

time series as images

The following function gives the stock price time series as an image (note that ArrayPlot and Image transform numbers to grayscale differently; to get the same visualization one has to invert black and white (via a :> (1-a) )

toImage[stock_] := financialDataToList@stock //List //Replace[#, a_ :> (1 - a)] & //Image

The net I'd to use may look like this (one or more convolutions/poolings followed by a fully connected layer and outputs that indicate the likelihood of belonging to the respective branch of industry). Feel free to make any sensible adjustments. 

net = NetChain[{
ConvolutionLayer[5, {1, 20}], 
PoolingLayer[{1, 5}, "Stride" -> {4, 4}], 
FlattenLayer[], 
DotPlusLayer[2]}, 
"Input" -> NetEncoder[{"Image", {100, 1}, "Grayscale"}], 
"Output" -> NetDecoder[{"Class", {"tech", "banks"}}]]

The training data can be generated with 

trainingdata = Join[
Thread[(toImage /@ tech) -> "tech" ], 
Thread[(toImage /@ banks) -> "banks"] ]

Calling NetTrain[net, trainingdata] does result in an error I suspect is a direct cause of me not really knowing how to correctly specify the dimensions of input, output and the layers. I am not that familiar with neural networks in practice but have a general understanding of how the individual layers work in principle. Sadly the documentation is lacking of more full examples showing how to connect individual layers to form a working/useful net.

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ I've got this working, but am downloading a much bigger dataset (~2468 individual stocks from 12 industries) to make sure that the network isn't overfitting -- it rather easily gets 100% accuracy on your examples. Should take around 1.5 hours to download, will post the code when that's finished. $\endgroup$
    – Taliesin Beynon
    Aug 25, 2016 at 13:31
  • $\begingroup$ @TaliesinBeynon I browsed through a few stocks manually to see if there was actually a consistent trend. In some cases (JP Morgen I think) the "banksters" seem to have been lucky and their stock prices looks nothing like the other companies from the finance field. $\endgroup$
    – Sascha
    Aug 25, 2016 at 13:37
  • $\begingroup$ @TaliesinBeynon Will Mathematica support Convolution1D as Keras? Currently EmbeddingLayer cannot be connected to ConvolutionLayer. I would like to replicate this: github.com/fchollet/keras/blob/master/examples/imdb_cnn.py $\endgroup$
    – Alexey Golyshev
    Aug 25, 2016 at 14:49
  • 2
    $\begingroup$ @AlexeyGolyshev Yes, we'll make a 1D convolution (though its not hard to use ReshapeLayer to get around this). More importantly, EmbeddingLayer will be able to operate on sequences of inputs to produce sequences of embeddings. So basically you'll set up a NetSequenceEncoder[{"Text", <alphabet>}], and then you'll be able to feed strings into the net, they'll get encoded as sequences of integers, those will get embedded, and you can convolve them. But frankly, the embedding layer is redundant, it's equivalent to using one hot embedding and letting the convolution learn the basis. $\endgroup$
    – Taliesin Beynon
    Aug 25, 2016 at 15:27
  • $\begingroup$ @TaliesinBeynon Thanks for your answer. $\endgroup$
    – Alexey Golyshev
    Aug 25, 2016 at 15:34

2 Answers 2

Reset to default
19
$\begingroup$

Here's the code. What I mentioned in the Q&A session is using ReshapeLayer to turn the input vector into a 1-channel, flat tensor that ConvolutionLayer can operate on, not to actually use images, per se.

I've limited things here just to the original industries, but you can try more if you want. Downloading takes a while, so I have an Export there you can use to reload the data later, after you quit your kernel.

However, there is a big problem with this whole idea, as you'll see, which is overfitting -- there just don't seem to be very good patterns for the net to pick up on. The more powerful you make the net, the more easily it can simply memorize particular time series rather than finding 'deeper' patterns. This will often happen with timeseries classification, because each timeseries has a lot of data, and you typically don't have that many timeseries. The only way to overcome this is to use extreme quantities of data, say 10x or 100x time as much data as we have here.

When running NetTrain, you'll see the loss on the validation set plateaus after just 30 seconds or so. At that point further training is just pointless, so you can click Stop - NetTrain will always return the net with the lowest validation loss.

And if you had tried this without a validation set, and left this net to run for 5 minutes, you'd get an apparent 95% accuracy on your training data. Which would be completely misleading. The actual accuracy maxes out around 70%. Fiddling around with this net isn't going to do much: the overfitting is so severe, the net is in some sense already too complicated.

(*industries = {"Basic Industries", "Capital Goods", 
   "Consumer Durables", "Consumer Non-Durables", "Consumer Services", 
   "Energy", "Finance", "Health Care", "Miscellaneous", 
   "Public Utilities", "Technology", "Transportation"};*)    
In[1]:= industries = {"Finance", "Technology"};

In[2]:= importIndustryStocks[name_] := 
  Import["http://www.nasdaq.com/screening/companies-by-industry.aspx?\
industry=" <> StringReplace[name, " " -> "+"] <> "&render=download", 
    "CSV"][[2 ;;, 1]];

stocksByIndustry = AssociationMap[importIndustryStocks, industries];
Length /@ stocksByIndustry

Out[6]= <|"Finance" -> 558, "Technology" -> 373|>

In[4]:= stocks = Catenate[stocksByIndustry];
Length[stocks]

Out[5]= 931

In[8]:= stock2industry = 
  Association @ Reverse[Flatten[Thread /@ Normal[stocksByIndustry]], 2];

stockData = ParallelMap[
   Replace[FinancialData[#, "Jan. 1, 2007"], 
     Except[_List] -> $Failed] &, stocks]; (* this takes a while *)

In[9]:= Export["industry_stocks.mx", {industries, stocks, stock2industry, stockData}];
FileByteCount["industry_stocks.mx"]

Out[10]= 39651493

In[11]:= toList[ts_List] := Rescale @ ArrayResample[ts[[All, 2]], 256];
toList[_] := $Failed;
rules = DeleteCases[$Failed -> _] @ Thread[Map[toList, stockData] -> Lookup[stock2industry, stocks]];
Length[rules]

Out[14]= 880

In[18]:= TestTrainSplit[rules_, frac_] := Module[{grouped},
   grouped = GroupBy[RandomSample[rules], Last, split[frac]];
   Map[Catenate, Transpose @ Values[grouped]]];
split[frac_][list_] := Block[{n, tn},
   n = Length[list]; tn = Ceiling[n * frac];
   TakeDrop[list, tn]]; 
{testData, trainingData} = TestTrainSplit[rules, 0.2];

module[n_, sz_] := 
  Sequence[ConvolutionLayer[n, {1, sz}, "Dilation" -> 4], 
   PoolingLayer[{1, 16}], Ramp];
net = NetChain[{ReshapeLayer[{1, 1, 256}], module[4, 8], (* can add more modules here *) 
   FlattenLayer[], DotPlusLayer[], SoftmaxLayer[]}, "Input" -> 256, 
  "Output" -> NetDecoder[{"Class", industries}]]

trained = 
  NetTrain[net, trainingData, MaxTrainingRounds -> 100, 
   Method -> "ADAM", ValidationSet -> testData]; (* overfitting happens VERY quickly *)

In[25]:= ListLinePlot@NetExtract[trained, {2, "Weights"}][[All, 1, 1]] (* plot the kernels, they don't look particularly nice though *)

In[26]:= measure = ClassifierMeasurements[trained, testData]

In[27]:= measure["Accuracy"]

Out[27]= 0.711864
Improve this answer
$\endgroup$
6
  • $\begingroup$ Have you tried any regularization e.g. DropoutLayer? $\endgroup$
    – Sascha
    Aug 25, 2016 at 16:20
  • $\begingroup$ @Sascha that certainly helps, accuracy goes up to 77%. $\endgroup$
    – Taliesin Beynon
    Aug 25, 2016 at 16:40
  • $\begingroup$ Can you elaborate on what kind of regularization you tried? $\endgroup$
    – Sascha
    Aug 25, 2016 at 19:59
  • $\begingroup$ @Sascha Oh just a DropoutLayer right after the ConvolutionLayer. $\endgroup$
    – Taliesin Beynon
    Aug 26, 2016 at 16:19
  • $\begingroup$ This is actually a nice example for playing around with BayesianMaximization $\endgroup$
    – Sascha
    Aug 30, 2016 at 21:12
6
$\begingroup$ 
tech = {"GE", "AAPL", "MSFT", "MRK", "IBM"};
banks = {"DB", "C", "BAC", "HSBC", "BCS"};

financialDataToList[stock_] := 
 FinancialData[stock, "Jan. 1, 2007"] // Transpose // Last // 
   Rescale // ArrayResample[#, 100] &

toImage[stock_] := 
 financialDataToList@stock // List // Replace[#, a_ :> (1 - a)] & // 
  Image

trainingdata = 
  Join[Thread[(toImage /@ tech) -> "tech"], 
   Thread[(toImage /@ banks) -> "banks"]];

width = 100;

height = 1;

net = NetChain[
  {
   ConvolutionLayer[16(*number of filters*), {height, 3(*kernel size*)}],
   PoolingLayer[{height, 2(*pooling size*)}],
   FlattenLayer[],
   DotPlusLayer[10],
   DotPlusLayer[2(*number of classes*)],
   SoftmaxLayer["Output" -> NetDecoder[{"Class", {"tech", "banks"}}]]
   },
  "Input" -> NetEncoder[{"Image", {width, height}, ColorSpace -> "Grayscale"}]
  ]

enter image description here

net = NetTrain[net, trainingdata, MaxTrainingRounds -> 5]

ClassifierMeasurements[net, trainingdata, "Accuracy"]

1.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.