This is my target img:

I can binarize it and select the mark component:

bin = Opening[ColorNegate[Binarize[img, 0.8]], 4]

Then I can find the mark centroid with ComponentMeasurements:

HighlightImage[img, Values[ComponentMeasurements[bin, "Centroid"]]]

But in this post, I want to request a method that don't binarize the img. The binarition method always has to consider the thick of those lines and the threshold about white or black. So I want to process the gray image directly? It is possible?

There are some series image to process:




As the comment discuss, I have made some test dataset here.

  • 1
    $\begingroup$ What's your goal? To find a parameter-free method? Please explain why binarizing the image is a problem. $\endgroup$ – C. E. Nov 17 '18 at 10:19
  • 1
    $\begingroup$ I also find that Vitaliy's answer was good. I don't understand why you can't get a template. $\endgroup$ – C. E. Nov 17 '18 at 10:42
  • $\begingroup$ @C.E. Yes, I want to find a parameter-free method. If I use binarization method. I have to find a threshold. And I have to consider the thick of those lines... $\endgroup$ – yode Nov 17 '18 at 13:59
  • $\begingroup$ Will the highlighted sections always come in as gray? $\endgroup$ – Carl Lange Nov 17 '18 at 18:39
  • $\begingroup$ At some point you are going to want to use Binarize, for definite. Here's a super gross method to get an image of only the markers. ImageAdjust[ImageFilter[Max, ColorDistance[i, Gray], 3], 0, {0, .3}] But look at all those parameters. The easiest way to start might be ColorDistance[i, Gray] and then see where you go from there. ImageCorrelate would also be a good option. $\endgroup$ – Carl Lange Nov 17 '18 at 19:31

You can detect dark blobs without binarization using a Laplacian of Gaussian filter. First remove the thin lines from the image using dilation:

dilated = Dilation[img, 3]

then use an LoG filter to get high local maxima for dark blobs:

log = ImageAdjust[LaplacianGaussianFilter[dilated, {50, 10}]]

then find dark blobs:

max = MaxDetect[log, 0.15]
HighlightImage[img, max]

enter image description here

This seems to work for all your images:

   dilated = Dilation[img, 3];
   log = ImageAdjust[LaplacianGaussianFilter[dilated, {50, 10}]];
   max = MaxDetect[log, 0.15];
   HighlightImage[img, max, ImageSize -> 450])
  {img, imgs}] // Multicolumn

enter image description here

However, it's not parameter-free. I'm not sure you can do this without any parameters. The best you can do is to get parameters that are as tolerant as possible to e.g. different lighting, different rotation etc.

  • 1
    $\begingroup$ "I'm not sure you can do this without any parameters." - I mean, you could build a neural network and have many thousands of parameters, but you don't have to set any of them manually ;) $\endgroup$ – Carl Lange Nov 19 '18 at 14:52
  • 1
    $\begingroup$ @CarlLange: Not with 4 training images ;) $\endgroup$ – Niki Estner Nov 19 '18 at 14:55
  • $\begingroup$ @CarlLange If you can try it with network. I can provide a lot samples.. :) $\endgroup$ – yode Nov 20 '18 at 7:41
  • $\begingroup$ @yode: neural networks are typically trained with 1.000 to 1.000.000 images. Can you provide that many? ;) $\endgroup$ – Niki Estner Nov 20 '18 at 8:00
  • $\begingroup$ @NikiEstner If just 1000, of course I have. :) $\endgroup$ – yode Nov 20 '18 at 8:04

Training a classifier

Let's go through the process of training a classifier for this problem. We start by loading the images:

imgs = Import /@ FileNames[".../dir1/*.png"];
imgs = Join[
   Import /@ FileNames[".../dir2/*.png"]
imgs = Join[
   Import /@ FileNames[".../dir3/*.png"]
imgs = RandomSample[imgs];

Randomizing the order with RandomSample is important because in the next step we are going to annotate the images one by one, but we're not going to annotate all of them, and we want to annotate images from all three datasets, not just the first one.

We build the following interface for annotating these images:

$i = 1;
$marked = {};
$unmarked = {};
$selected = {};
subimages = ImagePartition[First[imgs], {45, 30}];
   Map[Button[#, AppendTo[$selected, #]] &, subimages, {2}] // Grid,
   TrackedSymbols :> {subimages}
   "Next grid",
   $marked = Join[$marked, $selected];
   $unmarked = Join[$unmarked, Complement[Flatten[subimages], $selected]];
   $selected = {};
   subimages = ImagePartition[imgs[[++$i]], {45, 30}];
  Dynamic@StringJoin[ToString[$i], "/", ToString[Length[imgs]]]

Mathematica graphics

This worked well enough. To use this, click on all the marked images and then hit the button at the bottom to go to the next image. In the end, $marked is a list of images of marked patches, and $unmarked is a list of images of unmarked patches.

I annotated one hundred images (834 marked patches) using this interface and exported the patches. You can download the dataset here. If you want to follow along, download and unzip the dataset, and then import the images like this:

unmarked = Import /@ FileNames[".../patches/unmarked/*.png"];
marked = Import /@ FileNames[".../patches/marked/*.png"];

marked = Thread[marked -> 1];
unmarked = Thread[unmarked -> 0];

train = Part[marked, ;; 500]~Join~Part[unmarked, ;; 500];
train = RandomSample[train];
test = Part[marked, 501 ;;]~Join~Part[unmarked, 501 ;; 834];
test = RandomSample[test];

I have more unmarked patches than marked ones, but I make the datasets such that I have the same number from each class. Having a balanced dataset is usually preferable.

Now we train a classifier, Mathematica will automatically select the type of classifier for us:

classifier = Classify[train];

Next, we test it on the test set:

Mean@N@Boole[test /. Rule[in_, out_] :> classifier[in] == out]


Awesome, it worked for 99.25 percent of the images. We can look at the images it missed using this code:

Pick[test, test /. Rule[in_, out_] :> classifier[in] != out]

I'm not sure why it missed these because it looks like it shouldn't have, but it will probably be resolved if we annotate more images. In any case, this was just whatever classifier Mathematica decided to use for us, there might be a better one. In fact, setting Method to SupportVectorMachine we get a classifier which performs perfectly:

classifier = Classify[train, Method -> "SupportVectorMachine"];
Mean@N@Boole[test /. Rule[in_, out_] :> classifier[in] == out]


Locating objects

The classifier works great if we know where to look for the objects. But if we don't know how to extract the subimages to run the classifier on, then how can we do it? The most time-honored method for this is the sliding window technique. We simply take a window and slide it over the image, extracting subimages as we go along. We then run the classifier on each subimage to determine if it is a match or not.

imagePartition[{nrows_, ncols_}, {w_, h_}, {stepx_, stepy_}] := Table[{
    Rectangle[{c, nrows - r}, {c + w, nrows - r - h}],
    ImageTake[img, {r, r + h}, {c, c + w}]
   {r, 1, nrows - h, stepy},
   {c, 1, ncols - w, stepx}
   ] // Flatten[#, 1] &

classify[classifier_, subimages_, thresh_] := Map[
  #[1] > thresh &,
  classifier[subimages, "Probabilities"]

{ncols, nrows} = ImageDimensions[img];
{boxes, subimages} = Transpose@imagePartition[{nrows, ncols}, {45, 30}, {10, 10}];
selected = Pick[boxes, classify[classifier, subimages, 0.5]];

HighlightImage[img, selected]

Mathematica graphics

In the previous image, we see the subimages that the classifier thinks are marked patches. But there is a problem: the algorithm has selected several boxes for each patch. A solution to this problem is known as non-maximum suppression, and it works like this:

  • Group all boxes together that overlap each other with a given fraction, such as fifty percent.
  • Take the mean of the bottom left corners and the top right corners of all the boxes in each group to find a new box that will represent that group.

The following code implements this:

intersectingQ[r1_, r2_, thresh_] := Area@RegionIntersection[r1, r2] > thresh Area[r1]

meanRectangle[rectangles_] := Module[{bottomLeft, topRight},
  bottomLeft = rectangles[[All, 1]];
  topRight = rectangles[[All, 2]];
  Rectangle[Mean[bottomLeft], Mean[topRight]]

nonMaximumSuppression[boxes_, thresh_] := Module[{graph, groups},
  graph = AdjacencyGraph@Boole@Outer[intersectingQ[#, #2, thresh] &, bboxes, bboxes];
  groups = VertexList /@ ConnectedGraphComponents[graph];
  meanRectangle@Part[boxes, #] & /@ groups

 nonMaximumSuppression[selected, 0.5]

Mathematica graphics

This worked quite well! We didn't miss any marked patches, and the boxes are reasonable. The positions of the boxes close to the border are biased inwards because we're taking the mean of all the proposed boxes, and they are all located on one side of the marked patch. This could probably be solved by padding. Also note that decreasing the step size used when generating the subimages will increase precision as well.

  • 2
    $\begingroup$ Excellent answer. Classify is so powerful. $\endgroup$ – Carl Lange Nov 21 '18 at 9:50
  • 1
    $\begingroup$ @CarlLange Now it's even better :) $\endgroup$ – C. E. Nov 22 '18 at 21:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.