Training a classifier
Let's go through the process of training a classifier for this problem. We start by loading the images:
imgs = Import /@ FileNames[".../dir1/*.png"];
imgs = Join[
imgs,
Import /@ FileNames[".../dir2/*.png"]
];
imgs = Join[
imgs,
Import /@ FileNames[".../dir3/*.png"]
];
imgs = RandomSample[imgs];
Randomizing the order with RandomSample
is important because in the next step we are going to annotate the images one by one, but we're not going to annotate all of them, and we want to annotate images from all three datasets, not just the first one.
We build the following interface for annotating these images:
$i = 1;
$marked = {};
$unmarked = {};
$selected = {};
subimages = ImagePartition[First[imgs], {45, 30}];
Column[{
Dynamic[
Map[Button[#, AppendTo[$selected, #]] &, subimages, {2}] // Grid,
TrackedSymbols :> {subimages}
],
Button[
"Next grid",
$marked = Join[$marked, $selected];
$unmarked = Join[$unmarked, Complement[Flatten[subimages], $selected]];
$selected = {};
subimages = ImagePartition[imgs[[++$i]], {45, 30}];
],
Dynamic@StringJoin[ToString[$i], "/", ToString[Length[imgs]]]
}]

This worked well enough. To use this, click on all the marked images and then hit the button at the bottom to go to the next image. In the end, $marked
is a list of images of marked patches, and $unmarked
is a list of images of unmarked patches.
I annotated one hundred images (834 marked patches) using this interface and exported the patches. You can download the dataset here. If you want to follow along, download and unzip the dataset, and then import the images like this:
unmarked = Import /@ FileNames[".../patches/unmarked/*.png"];
marked = Import /@ FileNames[".../patches/marked/*.png"];
marked = Thread[marked -> 1];
unmarked = Thread[unmarked -> 0];
train = Part[marked, ;; 500]~Join~Part[unmarked, ;; 500];
train = RandomSample[train];
test = Part[marked, 501 ;;]~Join~Part[unmarked, 501 ;; 834];
test = RandomSample[test];
I have more unmarked patches than marked ones, but I make the datasets such that I have the same number from each class. Having a balanced dataset is usually preferable.
Now we train a classifier, Mathematica will automatically select the type of classifier for us:
classifier = Classify[train];
Next, we test it on the test set:
Mean@N@Boole[test /. Rule[in_, out_] :> classifier[in] == out]
0.992515
Awesome, it worked for 99.25 percent of the images. We can look at the images it missed using this code:
Pick[test, test /. Rule[in_, out_] :> classifier[in] != out]
I'm not sure why it missed these because it looks like it shouldn't have, but it will probably be resolved if we annotate more images. In any case, this was just whatever classifier Mathematica decided to use for us, there might be a better one. In fact, setting Method
to SupportVectorMachine
we get a classifier which performs perfectly:
classifier = Classify[train, Method -> "SupportVectorMachine"];
Mean@N@Boole[test /. Rule[in_, out_] :> classifier[in] == out]
1
Locating objects
The classifier works great if we know where to look for the objects. But if we don't know how to extract the subimages to run the classifier on, then how can we do it? The most time-honored method for this is the sliding window technique. We simply take a window and slide it over the image, extracting subimages as we go along. We then run the classifier on each subimage to determine if it is a match or not.
imagePartition[{nrows_, ncols_}, {w_, h_}, {stepx_, stepy_}] := Table[{
Rectangle[{c, nrows - r}, {c + w, nrows - r - h}],
ImageTake[img, {r, r + h}, {c, c + w}]
},
{r, 1, nrows - h, stepy},
{c, 1, ncols - w, stepx}
] // Flatten[#, 1] &
classify[classifier_, subimages_, thresh_] := Map[
#[1] > thresh &,
classifier[subimages, "Probabilities"]
]
{ncols, nrows} = ImageDimensions[img];
{boxes, subimages} = Transpose@imagePartition[{nrows, ncols}, {45, 30}, {10, 10}];
selected = Pick[boxes, classify[classifier, subimages, 0.5]];
HighlightImage[img, selected]

In the previous image, we see the subimages that the classifier thinks are marked patches. But there is a problem: the algorithm has selected several boxes for each patch. A solution to this problem is known as non-maximum suppression, and it works like this:
- Group all boxes together that overlap each other with a given fraction, such as fifty percent.
- Take the mean of the bottom left corners and the top right corners of all the boxes in each group to find a new box that will represent that group.
The following code implements this:
intersectingQ[r1_, r2_, thresh_] := Area@RegionIntersection[r1, r2] > thresh Area[r1]
meanRectangle[rectangles_] := Module[{bottomLeft, topRight},
bottomLeft = rectangles[[All, 1]];
topRight = rectangles[[All, 2]];
Rectangle[Mean[bottomLeft], Mean[topRight]]
]
nonMaximumSuppression[boxes_, thresh_] := Module[{graph, groups},
graph = AdjacencyGraph@Boole@Outer[intersectingQ[#, #2, thresh] &, bboxes, bboxes];
groups = VertexList /@ ConnectedGraphComponents[graph];
meanRectangle@Part[boxes, #] & /@ groups
]
HighlightImage[
img,
nonMaximumSuppression[selected, 0.5]
]

This worked quite well! We didn't miss any marked patches, and the boxes are reasonable. The positions of the boxes close to the border are biased inwards because we're taking the mean of all the proposed boxes, and they are all located on one side of the marked patch. This could probably be solved by padding. Also note that decreasing the step size used when generating the subimages will increase precision as well.
ImageAdjust[ImageFilter[Max, ColorDistance[i, Gray], 3], 0, {0, .3}]
But look at all those parameters. The easiest way to start might beColorDistance[i, Gray]
and then see where you go from there.ImageCorrelate
would also be a good option. $\endgroup$