8
$\begingroup$

I have a dataset of some (grayscale) images (stored as a list in inputImages) varying in size and content (this is example data from Wolfram, so you can run the code, too)

inputImages = ExampleData /@ ExampleData["TestImage"];
inputImages = ColorConvert[#, "Grayscale"] & /@ inputImages;(*convert to Grayscale for simplicity*)

I just went ahead and assigned each of those images a individual number via

trainingset = inputImages[[#]] -> # & /@ Range[Length[inputImages]];

I then used Predict to create a predictor function via

predict = Predict[trainingset, PerformanceGoal -> "Quality"]

For simple testing purposes I used the training data as input via

out = predict /@ inputImages

{24.5, 21.35, 21.35, 21.35, 27.05, . . . 27.35, 27.55, 27.45, 23.8, 26.7}

To my surprise I do not get an output close to the Range[Length[inputImages]] vector. I tried most available methods for the Predict function with completely different results but none of them close to the simple ascending list the Predict function was trained with.

Any suggestions why the results are not closer to Range[Length[inputImages]] or how to get better predictions?

|improve this question
$\endgroup$
  • $\begingroup$ I am willing to bet that if you make loading the list of images easy you will get more answers. $\endgroup$ – Mr.Wizard♦ Oct 10 '14 at 0:47
  • 2
    $\begingroup$ I think your problem can be illustrated using: inputImages = ExampleData /@ ExampleData["TestImage"]; $\endgroup$ – Mr.Wizard♦ Oct 10 '14 at 0:51
  • 1
    $\begingroup$ @Mr.Wizard: Thanks, the problem does not seem to be dependent on my image dataset, so your suggestion works fine to present the problem. I edited the post, so everyone can try on their own now. $\endgroup$ – Wizard Oct 10 '14 at 9:54
  • 2
    $\begingroup$ Perhaps you're looking for Classify instead of Predict? $\endgroup$ – Chip Hurst Oct 10 '14 at 20:22
  • 1
    $\begingroup$ Or maybe Nearest is better for you since you only have one example for each class. $\endgroup$ – Chip Hurst Oct 10 '14 at 20:29
4
$\begingroup$

Disclaimer: I wouldn't feel confident enough to put this as an answer, but it since you asked for "suggestions of why the results aren't closer", it may be valuable and it is too long for a comment. Also, I haven't had the time to test even the question code so, this will be deleted or modified if it ends up not making too much sense

Answer: I don't think its fair to expect a reasonable output from Predict with an unreasonable input like this one. We are training it with images assigned a value that isn't easily correlateable to the image. Predicting is not finding the best fit: it is finding the best prediction. For this, overfitting is a big issue. Very complex models that make random stuff fit tend to work poorly with new data. What matters is how it performs for new images, not for the ones with which it was trained. To do this, it optimizes for the training data but only after choosing hyperparameters according to how predictions work on a validation set. If you want it to perform well on the input, you should explicitly pass the input also as ValidationSet. Otherwise, it will choose some method of partitioning the input, some number of times, in some proportion, to get the validation set from the input, according to your performance goal, number of inputs, or probably other stuff. Say it didn't shuffle the input, trained with your first 30 images and validated with the last 20. I don't think any sensible method or hyperparameters would give a good prediction for those last 20 (they were all assigned higher values than all the training data), so perhaps Predict ends up having to choose between very bad options. Bottom line is, to evaluate how good a preditor function is, you should split your input, train with some, and test with the others. In this case, you would probably realise easily that if you trained with imgs=ExampleData/@ExampleData["TestImage"];imgs~Take~20, it makes no sense to expect a good result on imgs~Drop~20 simply because your sets don't share information.

|improve this answer
$\endgroup$
  • $\begingroup$ "Predicting is not finding the best fit" unless your using Method -> "LinearRegression", then it is a best fit. But, that's only a minor nit. I, otherwise, agree. $\endgroup$ – rcollyer Oct 13 '14 at 19:55
  • $\begingroup$ @rcollyer, ok, but it is minimizing the error with some regularization, that might be any crazy thing with this input, right? $\endgroup$ – Rojo Oct 13 '14 at 20:05
  • $\begingroup$ With this input, yes. PredictorInformation[predict, "L2Regularization"] returns 100000 for me, but that may vary depending on what the random number generator tells it too. $\endgroup$ – rcollyer Oct 13 '14 at 20:37
  • $\begingroup$ @Rojo your analysis is spot on here. There is no actual relationship between the inputs and the outputs that could be learned in the first place. Predict is doing the right thing by not producing a phony PredictionFunction which overfits to the data (which would be Nearest under the hood, anyway). $\endgroup$ – Taliesin Beynon Nov 5 '14 at 2:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.