4
$\begingroup$

Essentially I have used the SemanticImport[] function on a csv file with headers for each column. Now I understand that each of these headers is referred to 'key' in mathematica. *

I have no problem doing Predict[]. But I understand that NetTrain requires a different input format I was looking under Association, but have not found clarity in doing this.

Thank you and appreciate your advice.


Following reply by aardvark2012:

I ran the following lines:

net = NetChain[{5, 1}, "Input" -> 8, "Output" -> "Scalar"];
trained = NetTrain[net, inputdata]

But an error is shown:

NetTrain::invdataset: Datasets provided to NetTrain must consist of a list of associations with fixed keys.

UPDATE: solved by xslittlegrass. Thank you

$\endgroup$
5
$\begingroup$

This seems to work. Set up some data:

assocdata = 
 Table[Association["Latitude" -> RandomInteger[{-90, 90}], 
   "Longitude" -> RandomInteger[{-180, 180}], 
   "Temperature" -> RandomReal[{-40, 49}]], 5]

{<|"Latitude" -> 64, "Longitude" -> 25, "Temperature" -> -24.8889|>, <|"Latitude" -> -49, "Longitude" -> -101, "Temperature" -> -28.0145|>, <|"Latitude" -> 9, "Longitude" -> -112, "Temperature" -> 22.6383|>, <|"Latitude" -> -65, "Longitude" -> 150,
"Temperature" -> 13.6052|>, <|"Latitude" -> 25, "Longitude" -> 110,
"Temperature" -> 29.0704|>}

Then

listdata = {#[[1]], #[[2]]} -> #[[3]] & /@ assocdata

gives

{{64, 25} -> -24.8889, {-49, -101} -> -28.0145, {9, -112} -> 22.6383, {-65, 150} -> 13.6052, {25, 110} -> 29.0704}

Is that what you meant?

$\endgroup$
  • $\begingroup$ yes, your answer gives me the correct format I am looking for, but somehow NetTrain doesn't like it. Could you have a look at my updated question and would appreciate your advice. Thank you $\endgroup$ – Corse Aug 3 '17 at 3:26
  • $\begingroup$ @Corse Unfortunately I can't play around with it since I'm on 10.4 and NetTrain is 11+. The only guess I can make is that it wants an input of the form Association["Lattiude" -> assocdata[[;; , 1]], "Longitude" -> assocdata[[;; , 2]], "Temperature" -> assocdata[[;; , 3]]]' (where assocdata` has the same form as in my answer). Not convinced it'll work, though, since it's not a "list of associations with fixed keys". Not even sure what a fixed key is. Is it perhaps interpreting your latitude, etc as variable names? Sorry, but I'm just guessing now. $\endgroup$ – aardvark2012 Aug 3 '17 at 4:52
  • $\begingroup$ I would trawl through the documentation on all the different *Layers (Linear, Elementwise, etc), play around with them and see if you can figure out what they're doing. $\endgroup$ – aardvark2012 Aug 3 '17 at 4:53
  • $\begingroup$ @Corse No problem. Thanks for the accept. $\endgroup$ – aardvark2012 Aug 3 '17 at 7:04
3
$\begingroup$

I think aardvark2012's answer is correct, this is just a comment on your follow-up questions. Your follow-up questions can be addressed by providing NetTrain with the compatible training data formats.

You can either use a Dataset or a list as training data. If a Dataset is used, then the format should be like Dataset[{<|"Input" -> {64, 25}, "Output" -> -24.8889|>, <|"Input" -> {-49, -101}, "Output" -> -28.0145|>}]. For example

inputdata = 
 Dataset@Table[
   Association["Input" -> RandomReal[{0, 1}, {2}], 
    "Output" -> RandomReal[{0, 1}]], {10}]

enter image description here

then you can train the network like

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = NetTrain[net, inputdata]

If using a list, then the training data should be like {{64, 25} -> -24.8889, {-49, -101} -> -28.0145, {9, -112} -> 22.6383, {-65, 150} -> 13.6052, {25, 110} -> 29.0704}. For example

inputdata = 
 Table[RandomReal[{0, 1}, {2}] -> RandomReal[{0, 1}], {10}];

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = NetTrain[net, inputdata]

You can also specify the input and output ports with NetGraph, which then allow you to train on a Dataset with named keys directly. For example

inputdata = Dataset[Association[Thread[header -> #]] & /@ data]

enter image description here

net = NetGraph[{ReshapeLayer[{1}], ReshapeLayer[{1}], CatenateLayer[],
    LinearLayer[5], LinearLayer[1]}, {NetPort["lat"] -> 1 -> 3, 
   NetPort["lon"] -> 2 -> 3 -> 4 -> 5 -> NetPort["temperature"]}]

enter image description here

NetTrain[net, inputdata, Method -> {"ADAM", "LearningRate" -> 0.01}]

Hope it helps.


Here is an example of how to read in and train on a csv dataset

path = 
 Export["~/Downloads/test.csv", 
  Table[{RandomReal[{-90, 90}], RandomReal[{-180, 180}], 
    RandomReal[{-50, 50}]}, {100}]]
(* "~/Downloads/test.csv" *)

inputdata = #[[1 ;; 2]] -> #[[3]] & /@ Import[path, "Data"];

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = 
 NetTrain[net, inputdata, Method -> {"ADAM", "LearningRate" -> 0.01}]

another example using dataset

path = 
 Export["~/Downloads/test.txt", 
  StringRiffle[
   Prepend[Table[
     StringRiffle[
      ToString[NumberForm[#, {3, 4}]] & /@ {RandomReal[{-90, 90}], 
        RandomReal[{-180, 180}], RandomReal[{-50, 50}]}, 
      "\t"], {100}], 
    StringRiffle[{"lat", "lon", "temperature"}, "\t"]], "\n"]]
(* "~/Downloads/test.txt" *)

tmp = Import[path, "Table"];


inputdata = 
  Association["Input" -> #[[1 ;; 2]], "Output" -> #[[3]]] & /@ 
   Rest@tmp;

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = 
 NetTrain[net, inputdata, Method -> {"ADAM", "LearningRate" -> 0.01}]
$\endgroup$
  • $\begingroup$ thank you for the clarification. I refer to this: Association["Input" -> RandomReal[{0, 1}, {2}], "Output" -> RandomReal[{0, 1}]], {10}]. How could I replace the RandomReal part using the Keys in my dataset? say I have header as "lat", "lon" and "temperature" from the CSV file. $\endgroup$ – Corse Aug 3 '17 at 14:55
  • $\begingroup$ @Corse See the updates. $\endgroup$ – xslittlegrass Aug 3 '17 at 16:16
  • $\begingroup$ this works well and that was very enlightening and helpful for my understanding on the input formats of NetTrain/NetGraph. Thank you very much! $\endgroup$ – Corse Aug 4 '17 at 2:10
  • $\begingroup$ a minor question to seek your advice: for the purposes of regression/prediction, is it normally recommended to put a BatchNormalizationLayer[] as the first hidden layer? Does it have the effect of scaling the values of each variable and is it at all necessary in Mathematica? $\endgroup$ – Corse Aug 4 '17 at 2:33
  • $\begingroup$ one more query if you don't mind, if i have an additional column (input variable) that is of a class vector, how could I modify the part: inputdata = #[[1 ;; 2]] -> #[[3]] & /@ Import[path, "Data"]; to utilize the NetEncoder function for 'UnitVector'? $\endgroup$ – Corse Aug 4 '17 at 3:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.