# Image Shadow Removal in Mathematica

Image shadow removal is an important topic in image processing. If you feel curious about it, you may take a look here, or just google for it image shadow removal

There are also a few questions on SE around it. For example:

However, I did not find any Mma implementations out there, and I don't want to reinvent the wheel if it is already rolling.

Does anybody know of some Mma codebase for using it as a starter?

Note: I have to do this for a few images, so I may go the manual way better than spend many hours in developing a full blown algorithm.

-
Have tried Inpaint ? –  kguler Jun 24 '12 at 18:42
@kguler AFAIK, Inpaint[] works well for retouching small selected areas of the image by replacing them with a "mean" of its surroundings. The way of taking the "mean" depends on the Method used. But I am not aware of how to use it for this specific problem –  belisarius Jun 24 '12 at 18:49
@kguler Inpainting works when the surrounding area is reasonably homogenous. In this case, you have the gray asphalt and the green grass and the light gray bricks in a curved path. If you inpaint it with a mask of the shadow, you'll find pieces of the bricks placed arbitrarily in the mask. –  The Toad Jun 24 '12 at 21:16

EDIT: I found a version of the source image with fewer jpeg artifacts.

My idea would be to extract pixels along the border of the shadow, inside and outside of the shadow. Then I have a list of pairs of RGB values, and I can find a suitable transformation from "shadowed" pixels to "non-shadowed" pixels.

But first, I must find the shadow area. I would apply a mean shift filter to remove the texture:

im = Import["http://www.cs.sfu.ca/~mark/ftp/Eccv04/path.jpg"]
ms = ms = MeanShiftFilter[GaussianFilter[im, 2], 10, 0.1, MaxIterations -> 5]


I can simply binarize this with default thresholds:

bin = DeleteSmallComponents[ColorNegate[DeleteSmallComponents[Binarize[ms]]]]


Next step: Extract edges and normal directions:

edges = EdgeDetect[bin];
dx = GaussianFilter[ImageData[bin], 5, {0, 1}];
dy = GaussianFilter[ImageData[bin], 5, {1, 0}];

edgePixels = Position[ImageData[edges], 1];
normalDirection = Transpose[{Extract[dy, edgePixels], Extract[dx, edgePixels]}];


Now edgePixels contains a list of edge pixel indices and normalDirection contains a list of normal vectors for each of these edge pixels.

Next step: for each border pixel, go 40 pixels in the normal direction, and get the RGB value at that point. Go 40 pixels the other way, and pick up that RGB value, too:

pixels = ImageData[GaussianFilter[im, 5]];
colorsInsideShadow = Extract[pixels, Round[edgePixels + normalDirection*40]];
colorsOutsideShadow = Extract[pixels, Round[edgePixels - normalDirection*40]];


Just to illustrate, the colors picked up look like this:

ImageAssemble[{{Image[ConstantArray[colorsInsideShadow, 10]]}, {Image[ConstantArray[colorsOutsideShadow, 10]]}}]


I would simply try a linear model from colors "inside the shadow" to colors "outside the shadow":

rgbToModel[{r_, g_, b_}] := {1, r, g, b}


Now I can apply this linear model to the shadow area:

applyColorTransform = ImageApply[rgbToModel[#].colorTransform &, im, Masking -> DeleteSmallComponents@ColorNegate[Binarize[ms]]]


As a last step, I'll use inpainting to remove the artifacts at the border of the shadow area:

result = Inpaint[applyColorTransform, Dilation[EdgeDetect[Binarize[ms]], DiskMatrix[1]]]


The whole algorithm is still quite ad-hoc (no camera calibration, I didn't even try to model the illumination invariants described in the linked papers), but considering it's simplicity (less than 15 lines of code), I'd say the result isn't that bad.

-
+1 The color interpolation trick is very nice! –  belisarius Jun 25 '12 at 8:56
pixels never gets Set. Is it just ImageData[im]? –  ArgentoSapiens Jun 25 '12 at 18:15
@ArgentoSapiens: Oops, you're right, forgot that line –  nikie Jun 25 '12 at 18:18

This is a difficult problem, as the referenced literature in the linked SE questions may attest.

I can provide some steps towards your goal, but I don't have the full answer.

im = ImageTake[Import["http://i.stack.imgur.com/rSOow.png"], {10, -10}, {10, 220}]


First, trying to find the shadow mask, ChanVeseBinarize seems to work best here. Note that I used a marker value to indicate the area I'm interested in. Blurring gets rid of very small components first.

ChanVeseBinarize[Blur[im], {{99, 39}}]


Getting rid of another set of additional spurious elements using DeleteSmallComponents:

mask = ColorConvert[
DeleteSmallComponents[ ChanVeseBinarize[Blur[im], {{99, 39}}]],
"GrayScale"
]


From here, the plan goes awry. The idea was to separate the image in hue, saturation, and brightness components, assuming that the shadows should principally influence brightness and leave hue and saturation alone.

imHSB = ColorConvert[im, "HSB"];


Interactively increasing the brightness in the shadow area until the shadow disappears:

Manipulate[
{b, 0, 1}
]


Bummer, it seems the impact of the shadow on the lower-left part of the image is different from the upper-right part. I don't seem to be able to remove it everywhere.

Another problem becomes apparent if we look at the hsb components now:

GraphicsRow[Image /@ {imH, imS, imB}]


The effects of the shadow doesn't seem to be restricted to the brightness alone. The non-linearities in the camera have caused the color components to be affected too.

For now, I'm out of options, but at least you have a mask.

-
I firstly also thought in "assuming that the shadows should principally influence brightness and leave hue and saturation alone." But after discovering that this is not the case I went away from this path. The ChanVeseBinarize is a nice trick for a start +1. –  belisarius Jun 24 '12 at 21:37
"assuming that the shadows should principally influence brightness and leave hue and saturation alone." No, that's not true. The light in the shadow region is diffuse light coming from all directions. On a clear day, because the sky is blue shadows are also blue by comparison to direct light. Further, colored objects in the environment also change the color of the shadow because of reflected light. –  Mr.Wizard Jun 24 '12 at 21:42
@Mr.Wizard Yep. I convinced myself that it isn't a "smart method discovery" kind of problem. That is why I ask for already existing references. –  belisarius Jun 24 '12 at 21:57
@Mr.Wizard I knew, I was just hoping the effect wouldn't be so big. –  Sjoerd C. de Vries Jun 24 '12 at 22:03
@Sjoerd C. de Vries maybe you can use monochrome luminance of colors to filter. lum=0.299 * r + .587 * g + .114 * b; –  s.s.o Jun 24 '12 at 22:15