Using Predict on images

Question

I have a dataset of some (grayscale) images (stored as a list in inputImages) varying in size and content (this is example data from Wolfram, so you can run the code, too)

inputImages = ExampleData /@ ExampleData["TestImage"];
inputImages = ColorConvert[#, "Grayscale"] & /@ inputImages;(*convert to Grayscale for simplicity*)

I just went ahead and assigned each of those images a individual number via

trainingset = inputImages[[#]] -> # & /@ Range[Length[inputImages]];

I then used Predict to create a predictor function via

predict = Predict[trainingset, PerformanceGoal -> "Quality"]

For simple testing purposes I used the training data as input via

out = predict /@ inputImages

{24.5, 21.35, 21.35, 21.35, 27.05, . . . 27.35, 27.55, 27.45, 23.8, 26.7}

To my surprise I do not get an output close to the Range[Length[inputImages]] vector. I tried most available methods for the Predict function with completely different results but none of them close to the simple ascending list the Predict function was trained with.

Any suggestions why the results are not closer to Range[Length[inputImages]] or how to get better predictions?

Answer 1

Disclaimer: I wouldn't feel confident enough to put this as an answer, but it since you asked for "suggestions of why the results aren't closer", it may be valuable and it is too long for a comment. Also, I haven't had the time to test even the question code so, this will be deleted or modified if it ends up not making too much sense

Answer: I don't think its fair to expect a reasonable output from Predict with an unreasonable input like this one. We are training it with images assigned a value that isn't easily correlateable to the image. Predicting is not finding the best fit: it is finding the best prediction. For this, overfitting is a big issue. Very complex models that make random stuff fit tend to work poorly with new data. What matters is how it performs for new images, not for the ones with which it was trained. To do this, it optimizes for the training data but only after choosing hyperparameters according to how predictions work on a validation set. If you want it to perform well on the input, you should explicitly pass the input also as ValidationSet. Otherwise, it will choose some method of partitioning the input, some number of times, in some proportion, to get the validation set from the input, according to your performance goal, number of inputs, or probably other stuff. Say it didn't shuffle the input, trained with your first 30 images and validated with the last 20. I don't think any sensible method or hyperparameters would give a good prediction for those last 20 (they were all assigned higher values than all the training data), so perhaps Predict ends up having to choose between very bad options. Bottom line is, to evaluate how good a preditor function is, you should split your input, train with some, and test with the others. In this case, you would probably realise easily that if you trained with imgs=ExampleData/@ExampleData["TestImage"];imgs~Take~20, it makes no sense to expect a good result on imgs~Drop~20 simply because your sets don't share information.