Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

The task is:

enter image description here

Image shadow removal is an important topic in image processing. If you feel curious about it, you may take a look here, or just google for it image shadow removal

There are also a few questions on SE around it. For example:

However, I did not find any Mma implementations out there, and I don't want to reinvent the wheel if it is already rolling.

Does anybody know of some Mma codebase for using it as a starter?

Note: I have to do this for a few images, so I may go the manual way better than spend many hours in developing a full blown algorithm.

share|improve this question
    
Have tried Inpaint ? –  kguler Jun 24 '12 at 18:42
    
@kguler AFAIK, Inpaint[] works well for retouching small selected areas of the image by replacing them with a "mean" of its surroundings. The way of taking the "mean" depends on the Method used. But I am not aware of how to use it for this specific problem –  belisarius Jun 24 '12 at 18:49
    
@kguler Inpainting works when the surrounding area is reasonably homogenous. In this case, you have the gray asphalt and the green grass and the light gray bricks in a curved path. If you inpaint it with a mask of the shadow, you'll find pieces of the bricks placed arbitrarily in the mask. –  rm -rf Jun 24 '12 at 21:16

2 Answers 2

up vote 31 down vote accepted

EDIT: I found a version of the source image with fewer jpeg artifacts.

My idea would be to extract pixels along the border of the shadow, inside and outside of the shadow. Then I have a list of pairs of RGB values, and I can find a suitable transformation from "shadowed" pixels to "non-shadowed" pixels.

But first, I must find the shadow area. I would apply a mean shift filter to remove the texture:

im = Import["http://www.cs.sfu.ca/~mark/ftp/Eccv04/path.jpg"]
ms = ms = MeanShiftFilter[GaussianFilter[im, 2], 10, 0.1, MaxIterations -> 5]

enter image description here

I can simply binarize this with default thresholds:

bin = DeleteSmallComponents[ColorNegate[DeleteSmallComponents[Binarize[ms]]]]

enter image description here

Next step: Extract edges and normal directions:

edges = EdgeDetect[bin];
dx = GaussianFilter[ImageData[bin], 5, {0, 1}];
dy = GaussianFilter[ImageData[bin], 5, {1, 0}];

edgePixels = Position[ImageData[edges], 1];
normalDirection = Transpose[{Extract[dy, edgePixels], Extract[dx, edgePixels]}];

Now edgePixels contains a list of edge pixel indices and normalDirection contains a list of normal vectors for each of these edge pixels.

Next step: for each border pixel, go 40 pixels in the normal direction, and get the RGB value at that point. Go 40 pixels the other way, and pick up that RGB value, too:

pixels = ImageData[GaussianFilter[im, 5]];
colorsInsideShadow = Extract[pixels, Round[edgePixels + normalDirection*40]];
colorsOutsideShadow = Extract[pixels, Round[edgePixels - normalDirection*40]];

Just to illustrate, the colors picked up look like this:

ImageAssemble[{{Image[ConstantArray[colorsInsideShadow, 10]]}, {Image[ConstantArray[colorsOutsideShadow, 10]]}}]

enter image description here

I would simply try a linear model from colors "inside the shadow" to colors "outside the shadow":

rgbToModel[{r_, g_, b_}] := {1, r, g, b}
colorTransform = LeastSquares[rgbToModel/@colorsInsideShadow, colorsOutsideShadow]

Now I can apply this linear model to the shadow area:

applyColorTransform = ImageApply[rgbToModel[#].colorTransform &, im, Masking -> DeleteSmallComponents@ColorNegate[Binarize[ms]]]

enter image description here

As a last step, I'll use inpainting to remove the artifacts at the border of the shadow area:

result = Inpaint[applyColorTransform, Dilation[EdgeDetect[Binarize[ms]], DiskMatrix[1]]]

enter image description here

The whole algorithm is still quite ad-hoc (no camera calibration, I didn't even try to model the illumination invariants described in the linked papers), but considering it's simplicity (less than 15 lines of code), I'd say the result isn't that bad.

share|improve this answer
    
+1 The color interpolation trick is very nice! –  belisarius Jun 25 '12 at 8:56
    
pixels never gets Set. Is it just ImageData[im]? –  ArgentoSapiens Jun 25 '12 at 18:15
    
@ArgentoSapiens: Oops, you're right, forgot that line –  nikie Jun 25 '12 at 18:18

This is a difficult problem, as the referenced literature in the linked SE questions may attest.

I can provide some steps towards your goal, but I don't have the full answer.

im = ImageTake[Import["http://i.stack.imgur.com/rSOow.png"], {10, -10}, {10, 220}]

Mathematica graphics

First, trying to find the shadow mask, ChanVeseBinarize seems to work best here. Note that I used a marker value to indicate the area I'm interested in. Blurring gets rid of very small components first.

ChanVeseBinarize[Blur[im], {{99, 39}}]

Mathematica graphics

Getting rid of another set of additional spurious elements using DeleteSmallComponents:

mask = ColorConvert[
           DeleteSmallComponents[ ChanVeseBinarize[Blur[im], {{99, 39}}]], 
           "GrayScale"
       ]

Mathematica graphics

From here, the plan goes awry. The idea was to separate the image in hue, saturation, and brightness components, assuming that the shadows should principally influence brightness and leave hue and saturation alone.

imHSB = ColorConvert[im, "HSB"];

Interactively increasing the brightness in the shadow area until the shadow disappears:

Manipulate[
   ImageMultiply[imB // Image, ImageAdjust[mask // Blur, 0, {0, 1}, {b, 1}]], 
   {b, 0, 1}
]

Mathematica graphics

Bummer, it seems the impact of the shadow on the lower-left part of the image is different from the upper-right part. I don't seem to be able to remove it everywhere.

Another problem becomes apparent if we look at the hsb components now:

GraphicsRow[Image /@ {imH, imS, imB}]

Mathematica graphics

The effects of the shadow doesn't seem to be restricted to the brightness alone. The non-linearities in the camera have caused the color components to be affected too.

For now, I'm out of options, but at least you have a mask.

share|improve this answer
    
I firstly also thought in "assuming that the shadows should principally influence brightness and leave hue and saturation alone." But after discovering that this is not the case I went away from this path. The ChanVeseBinarize is a nice trick for a start +1. –  belisarius Jun 24 '12 at 21:37
    
"assuming that the shadows should principally influence brightness and leave hue and saturation alone." No, that's not true. The light in the shadow region is diffuse light coming from all directions. On a clear day, because the sky is blue shadows are also blue by comparison to direct light. Further, colored objects in the environment also change the color of the shadow because of reflected light. –  Mr.Wizard Jun 24 '12 at 21:42
    
@Mr.Wizard Yep. I convinced myself that it isn't a "smart method discovery" kind of problem. That is why I ask for already existing references. –  belisarius Jun 24 '12 at 21:57
    
@Mr.Wizard I knew, I was just hoping the effect wouldn't be so big. –  Sjoerd C. de Vries Jun 24 '12 at 22:03
    
@Sjoerd C. de Vries maybe you can use monochrome luminance of colors to filter. lum=0.299 * r + .587 * g + .114 * b; –  s.s.o Jun 24 '12 at 22:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.