Regarding input to NetTrain function

Question

Essentially I have used the SemanticImport[] function on a csv file with headers for each column. Now I understand that each of these headers is referred to 'key' in mathematica. *

I have no problem doing Predict[]. But I understand that NetTrain requires a different input format I was looking under Association, but have not found clarity in doing this.

Thank you and appreciate your advice.

---

Following reply by aardvark2012:

I ran the following lines:

net = NetChain[{5, 1}, "Input" -> 8, "Output" -> "Scalar"];
trained = NetTrain[net, inputdata]

But an error is shown:

NetTrain::invdataset: Datasets provided to NetTrain must consist of a list of associations with fixed keys.

---

UPDATE: solved by xslittlegrass. Thank you

Answer 1

This seems to work. Set up some data:

assocdata = 
 Table[Association["Latitude" -> RandomInteger[{-90, 90}], 
   "Longitude" -> RandomInteger[{-180, 180}], 
   "Temperature" -> RandomReal[{-40, 49}]], 5]

{<|"Latitude" -> 64, "Longitude" -> 25, "Temperature" -> -24.8889|>, <|"Latitude" -> -49, "Longitude" -> -101, "Temperature" -> -28.0145|>, <|"Latitude" -> 9, "Longitude" -> -112, "Temperature" -> 22.6383|>, <|"Latitude" -> -65, "Longitude" -> 150,
"Temperature" -> 13.6052|>, <|"Latitude" -> 25, "Longitude" -> 110,
"Temperature" -> 29.0704|>}

Then

listdata = {#[[1]], #[[2]]} -> #[[3]] & /@ assocdata

gives

{{64, 25} -> -24.8889, {-49, -101} -> -28.0145, {9, -112} -> 22.6383, {-65, 150} -> 13.6052, {25, 110} -> 29.0704}

Is that what you meant?

Answer 2

I think aardvark2012's answer is correct, this is just a comment on your follow-up questions. Your follow-up questions can be addressed by providing NetTrain with the compatible training data formats.

You can either use a Dataset or a list as training data. If a Dataset is used, then the format should be like Dataset[{<|"Input" -> {64, 25}, "Output" -> -24.8889|>, <|"Input" -> {-49, -101}, "Output" -> -28.0145|>}]. For example

inputdata = 
 Dataset@Table[
   Association["Input" -> RandomReal[{0, 1}, {2}], 
    "Output" -> RandomReal[{0, 1}]], {10}]

then you can train the network like

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = NetTrain[net, inputdata]

If using a list, then the training data should be like {{64, 25} -> -24.8889, {-49, -101} -> -28.0145, {9, -112} -> 22.6383, {-65, 150} -> 13.6052, {25, 110} -> 29.0704}. For example

inputdata = 
 Table[RandomReal[{0, 1}, {2}] -> RandomReal[{0, 1}], {10}];

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = NetTrain[net, inputdata]

You can also specify the input and output ports with NetGraph, which then allow you to train on a Dataset with named keys directly. For example

inputdata = Dataset[Association[Thread[header -> #]] & /@ data]

net = NetGraph[{ReshapeLayer[{1}], ReshapeLayer[{1}], CatenateLayer[],
    LinearLayer[5], LinearLayer[1]}, {NetPort["lat"] -> 1 -> 3, 
   NetPort["lon"] -> 2 -> 3 -> 4 -> 5 -> NetPort["temperature"]}]

NetTrain[net, inputdata, Method -> {"ADAM", "LearningRate" -> 0.01}]

Hope it helps.

---

Here is an example of how to read in and train on a csv dataset

path = 
 Export["~/Downloads/test.csv", 
  Table[{RandomReal[{-90, 90}], RandomReal[{-180, 180}], 
    RandomReal[{-50, 50}]}, {100}]]
(* "~/Downloads/test.csv" *)

inputdata = #[[1 ;; 2]] -> #[[3]] & /@ Import[path, "Data"];

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = 
 NetTrain[net, inputdata, Method -> {"ADAM", "LearningRate" -> 0.01}]

another example using dataset

path = 
 Export["~/Downloads/test.txt", 
  StringRiffle[
   Prepend[Table[
     StringRiffle[
      ToString[NumberForm[#, {3, 4}]] & /@ {RandomReal[{-90, 90}], 
        RandomReal[{-180, 180}], RandomReal[{-50, 50}]}, 
      "\t"], {100}], 
    StringRiffle[{"lat", "lon", "temperature"}, "\t"]], "\n"]]
(* "~/Downloads/test.txt" *)

tmp = Import[path, "Table"];


inputdata = 
  Association["Input" -> #[[1 ;; 2]], "Output" -> #[[3]]] & /@ 
   Rest@tmp;

net = NetChain[{5, 1}, "Input" -> 2, "Output" -> "Scalar"];
trained = 
 NetTrain[net, inputdata, Method -> {"ADAM", "LearningRate" -> 0.01}]