9
$\begingroup$

I want to use TrainImageContentDetector on a directory of ~2k unlabeled images. I'm willing to put in the effort to label them, but I was never good at using "Drawing Tools" or the "Image Ribbon".

Can anyone suggest an ergonomic workflow to draw the rectangles/masks over the image to specify different objects, in a way that is simple and scales?

Perhaps there's an internal tool that developers used to make the examples by hand:

enter image description here

Here's an example image and a very bad attempt at a workflow to get started:

img = CloudGet @ "https://www.wolframcloud.com/obj/f82a9e7e-81b7-419d-84e4-73de383f8d62";
Canvas @ Graphics @ img
(* add rectangles and shift enter *)
polys = DeleteDuplicates /@ 
   Cases[%, _BezierCurve | _BezierCurveBox, \[Infinity]][[All, 1]];
Graphics[{img, EdgeForm[{Thick, Blue}], FaceForm[None], Polygon /@ polys}] 

enter image description here


Update

Here is a modification of @Batracos's answer that addresses point #1 and half of point #4. I'm still unsure how to make the rectangles editable and the carousel interactive. The most crucial missing piece is how to undo/edit or simply delete individual boxes, because it's very easy to get them wrong!

enter image description here

pointsToRec[{x1_, y1_}, {x2_, y2_}] := Rectangle[{Min[x1, x2], Min[y1, y2]}, {Max[x1, x2], Max[y1, y2]}]
clipCoords[{x_,y_},{w_,h_}]:={Clip[x,{0,w}],Clip[y,{0,h}]}
scaleBox[{w_,h_}][Rectangle[{x1_,y1_},{x2_,y2_}]]:=Rectangle[{x1 w,y1 h},{x2 w,y2 h}]
scaleBox[{w_,h_}][list:{__Rectangle}]:=scaleBox[{w,h}]/@list

ClearAll[boxCreator];
boxCreator[classList_, ifns_, labels_:<||>] := 
Panel @ DynamicModule[
    {
        (* additional stuff *)
        idx = 1, i, allboxes=labels, box={}, class = classList[[1]], nav, classColors = AssociationThread[classList->ResourceFunction["SampleColors"][Length@classList]], 
        thumbs, carousel, len, ithumb, window=Min[8, Length@ifns], cf, 
        (* states *)
        first = True, mouseDown = False, left = {0,0}, current = {0,0}, draggingQ = False, pos = None,
        (* settings*)
        clipQ = True, scaledQ = True, guidesQ = True,
        (* variables *)
        dims, color, margin = 5, size = 400, sizeC, locator
    },
    
    color = classColors[class];
    i = Import@ifns[[idx]];
    len = Length@ifns;
    box = Lookup[allboxes, ifns[[idx]], {}];
    
    dims = ImageDimensions[i];
    size = 500;
    sizeC = N @ Log[10, size];
    color = If[ImageMeasurements[i, "MeanIntensity"] < .4, White, Black];
    locator = Graphics @ {
                {
                    Opacity[Dynamic[.4 Boole[MousePosition["Graphics"] =!= None]]], ColorNegate[color],
                    AbsoluteThickness[3], Circle[{0,0}, 10{1,1}], 
                    Line[{{{-15, 0}, {-5, 0}}, {{5, 0}, {15, 0}}, {{0, -15}, {0, -5}}, {{0, 5}, {0, 15}}}]
                },
                Opacity[Dynamic[Boole[MousePosition["Graphics"] =!= None]]], color,
                Circle[{0,0}, 10{1,1}],
                Line[{{{-15, 0}, {-5, 0}}, {{5, 0}, {15, 0}}, {{0, -15}, {0, -5}}, {{0, 5}, {0, 15}}}]
            };
    
    (* additional functions *)
                              (*save state*)         (*move*)                    (*load boxes*)                          (*load image*)
    nav[n_Integer] := (allboxes[ifns[[idx]]]=box; idx=Clip[n, {1,Length[ifns]}]; box=Lookup[allboxes, ifns[[idx]], {}]; i=Import[ifns[[idx]]]);
    
    ithumb[fn_] := ithumb[fn] = ImageResize[Import[fn], {230,140}, Padding->Black (*, ConformationMethod->"Fit"... how to maintain aspect ?*)];
    carousel[idx_, slots_:window] := Module[{cs, ce},
        cs = If[idx < slots, 1, Min[len-slots+1, idx-slots+1]]; ce = If[idx < slots, slots, idx];
        Grid[{MapAt[Framed[#,FrameStyle->{Blend[{Yellow,Orange}],Thickness[2]}, FrameMargins->{0,0}]&, ithumb /@ Part[ifns, cs;;ce], Min[idx, window]]},
         Spacings->{1,0}, ItemSize->All, Background->Gray]
    ];
    cf = If[#==={}, #, {FaceForm[Opacity[.2], Lookup[classColors, #[[2]], color]], EdgeForm[Lookup[classColors, #[[2]], color]], #[[1]]}] &;
    
    Deploy @ Column[{
    Framed[#, FrameMargins->0, FrameStyle->LightGray]& @ 
    EventHandler[
        MouseAppearance[Graphics[{
            (* image *)
            Inset[Dynamic@i, Center, Center, Scaled[1]],
            FaceForm[Opacity[.2, color]], EdgeForm[color],
            (* boxes present in list *)
            Dynamic[cf /@ box],
            (* box being currently drawn *)
            {
                Opacity[Dynamic[Boole[!first]]], EdgeForm[Opacity[Dynamic[Boole[!first]]]],
                FaceForm[], Rectangle[Dynamic@left, Dynamic@current]
            },
            (* guide lines *)
            {Opacity[Dynamic[.5 Boole[guidesQ && first && MousePosition["Graphics"] =!= None]], color], InfiniteLine[Dynamic@current, {1,0}], InfiniteLine[Dynamic@current, {0,1}]},
            (* locator *)
            Inset[locator, Dynamic[current], Center, Offset[{25, 25}]]
        },
            ImageSize -> Dynamic[size], 
            PlotRange -> Dynamic@If[scaledQ, {{0,1}, {0,1}}, Tuples[{{0}, dims}]],
            AspectRatio -> ImageAspectRatio[i],
            Background -> GrayLevel[.1],
            ImageMargins -> margin
        ], Graphics[]],
        {
        {"MouseDown", 1} :> (If[first,
            (* record first box corner *)
            left = current;
            first = False,
            (* record second box corner and save box *)
            If[
                EuclideanDistance[left, current] > If[scaledQ, .005, 2],
                AppendTo[box, pointsToRec[left, current] -> class];
                first = True
            ]
        ]),
        {"MouseUp", 1} :> (
            If[!first && draggingQ,
                (* record second box corner and save box *)
                If[
                    EuclideanDistance[left, current] > If[scaledQ, .005, 2],
                    AppendTo[box, pointsToRec[left, current] -> class];
                    first = True
                ]
            ];
            draggingQ = False
        ),
        {"MouseClicked", 2} :> (If[!first,
            (* cancel current selection *)
            first = True; draggingQ = False,
            (* cancel last selection *)
            If[Length[box] > 0, box = Most[box]]; draggingQ = False
        ]),
        "MouseMoved" :> (
            (* store current mouse position *)
            pos = MousePosition["Graphics"];
            If[VectorQ[pos], current = If[clipQ, clipCoords[pos, If[scaledQ, {1, 1}, dims]], pos]];
        ),
        "MouseDragged" :> (
            draggingQ = True;
        ),
        "RightArrowKeyDown" :> (   (*doesn't work*)
           nav[idx+1];
        ),
        "LeftArrowKeyDown" :> ((*doesn't work, idk why*)
            nav[idx-1];
        )
        }
    ],
    Row[{
        Framed[Pane[Dynamic @ carousel[idx], Dynamic @ size, ImageSizeAction->"ShrinkToFit"], Background->Gray, FrameMargins->None], "\n",
        
        (*"Index: " Slider[Dynamic[idx, {nav[idx]&, Automatic, nav[#1]&}], {1, Length[ifns], 1}], Dynamic[idx], "\n\n", *) (*doesn't work, idk why*)
        RadioButtonBar[Dynamic[class], classList], "\n",
        Button["Prev", nav[idx-1]],
        Button["Next", nav[idx+1]], "\t", 
        Button["Save", allboxes[ifns[[idx]]]=box; Print[Iconize[allboxes, "Labels"]]],"\n\n",
        
        
        "Clip", Checkbox[Dynamic[clipQ]],
        "Scale", Checkbox[Dynamic[scaledQ, 
            (scaledQ = #; If[# && Length[box] > 0, box[[All,1]] = scaleBox[1/dims][box[[All,1]]], box[[All,1]] = scaleBox[dims][box[[All,1]]]])&]],
        "Guides", Checkbox[Dynamic[guidesQ]],
        Button["Clear", box = {}; first = True; draggingQ = False], "\n",
        "Scale: ", Slider[Dynamic[sizeC, (sizeC = #; size = 10^#)&], {2.5, 3.1}], Dynamic[sizeC], "\n\n" (*shouldn't delimiters work in rows/cols?*)
        
        }]
    }]
]

Then to test it:

dir=CreateDirectory[]
Do[Export[dir<>"/"<>ToString[i]<>".jpeg",ResourceFunction["RandomPhoto"][500,300]],{i,15}];

classes={"face", "tree", "road"};
fns=FileNames["*.jpeg",dir];
previousWork=<||> (*replace with iconized after clicking save*);

boxCreator[classes, fns, previousWork]
$\endgroup$
4
  • 1
    $\begingroup$ This should be built in to the system, and image masks have never been easy enough $\endgroup$
    – M.R.
    Jul 19, 2022 at 4:16
  • $\begingroup$ Yes thanks. There are many front end gui building experts here, please try your hand at this! $\endgroup$
    – user5601
    Jul 21, 2022 at 4:33
  • 1
    $\begingroup$ @Batracos changing the box type in the radio button bar makes my mouse disappear, do you see that too? $\endgroup$
    – user5601
    Jul 26, 2022 at 5:48
  • 1
    $\begingroup$ Thanks @user5601 this is pretty good but needs the ability to edit labels individually, also the frame rate is too slow. $\endgroup$
    – M.R.
    Jul 27, 2022 at 4:05

1 Answer 1

13
+300
$\begingroup$

Updated answer^2

(the old answer is below)

I have updated the code to manage image selection, class selection and editing. The main annoyance is that event handler needs some box structure to be selected to work and the only way I manage to do it is by keeping the curson in the input field. Such is life ¯_(ツ)_/¯

Animation in WebM

Helpers

Some small updates to the class input field to make it less aggressive in selecting the first proposal

Clear[suggestionField]
suggestionField[Dynamic[x_, f_:Null], Dynamic[list_] | list_, hint_ : ""] := 
    InputField[
        Dynamic[x, (x = #; f[x])&],
        String,
        FieldHint -> hint,
        FieldCompletionFunction :> getSuggestions[list],
        ContinuousAction -> True
    ]

This code can be use to select up to 10 suggestion from a list sorted by EditDistance and with a simple fuzzy matching (not a cool as the one in sublime text but that is more complex to implement here)

getSuggestions[list_][string_?StringQ] :=
    list // 
        DeleteDuplicates //
        Select[StringLength[#] > 0 &] // 
        Select[StringContainsQ[RegularExpression[StringRiffle[Characters[string], ".*"]], IgnoreCase -> True]] // 
        SortBy[Function[x, EditDistance[x, string]]] //
        Take[#, UpTo[10]]&
getSuggestions[list_][_] := {}

A separate deginition for the locator, because I just don't like the system one. (Probably more code could be factored out)

Clear[locator]
locator[Dynamic[color_] | color_, opts___?OptionQ] := 
    Graphics[{
        {
            Opacity[.4], ColorNegate[color],
            AbsoluteThickness[3], Circle[{0,0}, 10{1,1}], 
            Line[{{{-15, 0}, {-5, 0}}, {{5, 0}, {15, 0}}, {{0, -15}, {0, -5}}, {{0, 5}, {0, 15}}}]
        },
        color,
        Circle[{0,0}, 10{1,1}],
        Line[{{{-15, 0}, {-5, 0}}, {{5, 0}, {15, 0}}, {{0, -15}, {0, -5}}, {{0, 5}, {0, 15}}}],
    },
        opts, ImageSize -> {25, 25}
    ]

Some formatting for the bounding boxes to take a little less space. Used in the new box list.

formatBox[Rectangle[{x1_,x2_},{x3_,x4_}]] := Sequence[Round[x1],Round[x2],"|", Round[x3],Round[x4],"\[Rule]"]

A utility to get a fixed window around a point. Used in the carousel

slidingWindow[i_, max_, size_] := Take[
    DeleteCases[0] @ DeleteDuplicates @ Flatten @ Thread[{
        PadRight[Range[i, Max[1,i - size], -1],    size, 0],
        PadRight[Range[i, Min[max, i + size], +1], size, 0]
    }], UpTo[size]] // Sort

This is to generate a consistent (and hopefully decent) color given a label

exprColor[expr_] :=
    Lighter[ColorData["BalancedHue", Mod[Hash[Unevaluated[expr]], 360]/360], .2]

A helper to load in-core and out-of-core images

Clear[getImage]
getImage[list_List, size_ : Automatic] := Map[getImage[#, size]&, list]
getImage[file_String?FileExistsQ, size_ : Automatic] := getImage[fastImageImport[file], size] (* fixed this line *)
getImage[image_Image?ImageQ, size_: Automatic] := ImageResize[Image[image, ImageSize -> 500, ImageResolution -> 72], Replace[size, Automatic -> Scaled[1]]]
getImage[e_, _] := Graphics[Text[Row[{"Failed to load image: ", e}]], ImageSize -> 500, AspectRatio -> 1/GoldenRatio]

Some internal function to skip the Import framework on common filenames

fastImageImport[file_String] := Module[
    {ext, imported},
    ext = FileExtension[file];
    Switch[ext,
        "jpg" | "jpeg",
            imported = First @ Image`ImportExportDump`ImageReadJPEG[file],
        "png",
            imported = First @ Image`ImportExportDump`ImageReadPNG[file];
            Image`ImportExportDump`DeleteCachePNG[],
        "tif" | "tiff",
            imported = First @ Image`ImportExportDump`ImageReadTIFF[file]
    ];

    (* If not an image yet, try Import *)
    If[!ImageQ[imported], imported = Import[file]];
    If[!ImageQ[imported], Throw[$Failed]];
    imported
]
fastImageImport[File[string_]] := fastImageImport[string]

Code

Now the gory part. It's still not out-of-core but that is a small adaptation (I might edit again if I have time).

The main changes are

  • out-of-core support
  • the support for a list of images (and annotations)
  • thumbnail carousel
  • arrow movement
  • the edit button which enables
    • box selection by clicking
    • box resizing and moving using locators

The latter is implemented via an parallel GraphicsBox with locator to move the box center and adjust the corners. The box id can be picked from the list of current boxes that are drown on the side. While in edit mode, writing in the class field with edit the class name for the current box.

I have also tweak the code to make more use of MousePosition because setting it via EventHandler was sending a lot of unnecessary traffic between the kernel and the frontend.

That said, is it probably still full of bugs so bear with me... =)

ClearAll[boxCreatorList]
SetAttributes[boxCreatorList, HoldAll]
boxCreatorList[imageList_, previousResults_ : Automatic] := 
DynamicModule[
    {
        max, classes,
        (* to keep only one serialized copy of the input data *)
        images = imageList,
        data = previousResults,
        (* states *)
        image, class, dims, range, color,
        id = 1, (*box id*)
        i = 1, (* image id*)
        j = 1, (* class id*)
        thumbID = 1, (* image thumbnail id*)
        first = True, mouseDown = False, left = {0,0}, current = {0,0},
        mouse = Dynamic@MousePosition[{"Graphics", PaneSelector}, {-5, -5}],
        (* settings*)
        clipQ = True, guidesQ = True, draggingQ = False, editQ = False, editClassQ = False,
        (* variables *)
        margin = 20, size, sizeC, minMove = 2, boxcenter, boxsize, thumbnail,
        (* herlpers *)
        safePos, getClasses, loadImage, addBox, deleteBox, editBox, checkCoord, carousel, styled
    },
    
    (* used to load a image from a list, can be modified to work out of core *)
    loadImage[k_] := (
        image = getImage[images[[k]]];
        dims = ImageDimensions[image];
        range = Tuples[{{0}, dims}];
        color = If[ImageMeasurements[image, "MeanIntensity"] < .4, White, Black];
        editQ = False; (* new image might not have associated boxes *)
    );
    (* used to get an updated class list *)
    getClasses[] := Union[Catenate[data[[All, All, 2]]]];
    (* to avoid wrting garbage to the box *)
    safePos[] := If[VectorQ[MousePosition["Graphics"]],
        current = checkCoord@MousePosition["Graphics"],
        current
    ];
    (* code to add a new box,
        it does not trigger if the box is too small to prevent accidentaly adding one
        the box id will be set to the new box
        the class list is updated to pick up new classes *)
    addBox[] := If[
        EuclideanDistance[left, safePos[]] > minMove,
        AppendTo[data[[i]], pointsToRec[left, safePos[]] -> class];
        classes = getClasses[];
        id = Length[data[[i]]];
        first = True
    ];
    (* code to delete a box,
        the box id will be set to the next available one
        the class list is updated to remove unused classes *)
    deleteBox[] := If[Length[data[[i]]] > 0,
        data[[i]] = Delete[data[[i]], id];
        classes = getClasses[];
        id = Min[id, Length[data[[i]]]]
    ];
    (* code to initialize the edited box info *)
    editBox[] := (
        boxcenter = Mean[List @@ data[[i]][[id, 1]]];
        boxsize = Abs[Subtract @@ data[[i]][[id, 1]]];
    );
    checkCoord[new_] := If[clipQ, clipCoords[new, dims], new];
    
    styled[data_] := MapThread[
        {#2, EdgeForm[#2], Button[#1, id = #3; editBox[];, Appearance -> "Frameless"]}&,
        {data[[All, 1]], exprColor /@ data[[All, 2]], Range[Length[data]]}
    ];
    
    (* memoized to reduce overhead, these are small images*)
    thumbnail[pos_] :=  thumbnail[pos] = getImage[images[[pos]], 50];
    carousel[] = Dynamic[With[{window = slidingWindow[thumbID, Length[images], 9]},
        Row[MapIndexed[
            Button[#, (i = thumbID = window[[#2[[1]]]]; loadImage[i]), Appearance -> "Frameless"]&,
            MapAt[Framed, Position[window, i]]@
            MapAt[Framed[#, FrameStyle->Transparent]&, Position[window, Except[i, _Integer]]]@
                ConformImages[thumbnail /@ window, 60, "Fit", Padding -> GrayLevel[.8]]
        ]]
    ], TrackedSymbols :> {id, thumbID}];
    
    (* variable initialization *)
    loadImage[1];
    data = PadRight[Replace[data, Automatic -> {}], Length[images], {{}}];
    classes = Replace[getClasses[], {} -> {"None"}];
    max = Length[images];
    size = {700, 500};
    sizeC = N @ Log[size[[1]]/200];
    class = First[classes];
    
    (* main interface *)
    EventHandler[
    Deploy @
    Framed[#, FrameMargins -> 10, FrameStyle -> LightGray]& @ 
    Grid[{
        (* image carousel *)    
        {Row[{
                Button["\[LeftAngleBracket]\[LeftAngleBracket]", thumbID = Max[1, thumbID - 5 9]], Button["\[LeftAngleBracket]", thumbID = Max[1, thumbID - 9]],
                Column[{carousel[], 
                    Dynamic[Style[StringForm["Showing `` to `` of ``", Sequence@@MinMax[slidingWindow[thumbID, Length[images], 9]], Length[images]], "Text"]]}],
                Button["\[RightAngleBracket]", thumbID = Min[Length[images], thumbID + 9]], Button["\[RightAngleBracket]\[RightAngleBracket]", thumbID = Min[Length[images], thumbID + 5 9]]
            }], SpanFromLeft},
    {
    (* info and controls *)
    Column[{
        (* controls *)
        Grid[{
            {"Clip", Checkbox[Dynamic[clipQ]]},
            {"Guides", Checkbox[Dynamic[guidesQ]]},
            {"Edit", Checkbox[Dynamic[editQ, (editQ = #; editBox[])&], Enabled -> Dynamic[Length[data[[i]]] > 0]]},
            {Slider[Dynamic[sizeC, (sizeC = #; size = 200{1, 5/7}*E^#)&], {0, 3}], SpanFromLeft},
            {Button["Clear", data[[i]] = {}; first = True; draggingQ = False; editQ = False], SpanFromLeft},
            {Button["Save", Print[Iconize[data]]; $LastAnnotation = data], SpanFromLeft},
            {suggestionField[Dynamic[class, (class = #; If[editQ, data[[i]][[id, 2]] = #]; classes = Append[getClasses[], class])&], Dynamic[classes], "label"],
                SpanFromLeft}
        }, Alignment -> Left, BaseStyle -> "Text"],
        (* box list *)
        Pane[
            Dynamic[If[Length[data[[i]]] > 0, Grid[
                MapIndexed[
                    Prepend[#, RadioButton[Dynamic[id, (id = #; editBox[])&], #2[[1]]]]&,
                    MapAt[formatBox, List @@@ data[[i]], {All, 1}]
                ],
                Alignment -> Left], ""]],
            250, ImageSizeAction->"ShrinkToFit", BaseStyle -> {"Text", FontSize -> 14}]
    }],
    (* image panel *)
    Framed[#, FrameMargins -> 0, FrameStyle -> LightGray, Background -> GrayLevel[.98]]& @ 
    PaneSelector[{
        True -> Graphics[{
            (* image *)
            Dynamic[Image[image, ImageSize -> size]],
            Dynamic@FaceForm[Opacity[.2, color]], Dynamic@EdgeForm[color],
            (* drawn boxes *)
            Dynamic[styled[data[[i]]]],
            (* controls to edit the current box *)
            Locator[Dynamic[boxcenter, (
                    boxcenter = #;
                    data[[i]][[id, 1, 1]] = checkCoord[boxcenter + boxsize/2];
                    data[[i]][[id, 1, 2]] = checkCoord[boxcenter - boxsize/2];
                )&
            ], locator[Dynamic[color], ImageSize -> 15]],
            Locator[Dynamic[data[[i]][[id, 1, 1]], (data[[i]][[id, 1, 1]] = checkCoord[#]; editBox[])&], locator[Dynamic[color]]],
            Locator[Dynamic[data[[i]][[id, 1, 2]], (data[[i]][[id, 1, 2]] = checkCoord[#]; editBox[])&], locator[Dynamic[color]]]

        },
            ImageSize -> Dynamic[size], PlotRange -> Dynamic[range], ImageMargins -> margin
        ],
        False ->
        EventHandler[
        MouseAppearance[Graphics[{
            (* image *)
            Dynamic[Image[image, ImageSize -> size]],
            Dynamic[color], (*Dynamic@FaceForm[Opacity[.2, color]], Dynamic@EdgeForm[color],*) Opacity[.2],
            (* boxes present in list *)
            {Dynamic[styled[data[[i]]]]},
            Dynamic@If[TrueQ @ first, 
                (* box being currently drawn *)
                {Opacity[Dynamic[If[guidesQ, .5, 0]]], InfiniteLine[mouse, {1,0}], InfiniteLine[mouse, {0,1}]}, 
                (* guide lines *)
                {FaceForm[], EdgeForm[Opacity[1]], Rectangle[Dynamic@left, mouse]}
            ],
            (* locator *)
            Inset[locator[Dynamic[color]], mouse, Center, Offset[{25, 25}]]
        },
            ImageSize -> Dynamic[size], PlotRange -> Dynamic[range], ImageMargins -> margin
        ], Graphics[]],
        {
        {"MouseDown", 1} :> If[first,
            (* record first box corner *)
            left = safePos[];
            first = False,
            (* record second box corner and save box *)
            addBox[]
        ],
        {"MouseUp", 1} :> (
            If[!first && draggingQ,
                (* record second box corner and save box *)
                addBox[]
            ];
            draggingQ = False
        ),
        {"MouseClicked", 2} :> If[!first,
            (* cancel current selection *)
            first = True; draggingQ = False,
            (* cancel last selection *)
            deleteBox[]; draggingQ = False
        ],
        "MouseDragged" :> (
            draggingQ = True;
        )
        }
    ]},
        Dynamic[editQ]
    ]
    }}, Alignment->Top],
        {
            "LeftArrowKeyDown" :> (thumbID = i = Max[1, i - 1]; loadImage[i];),
            "RightArrowKeyDown" :> (thumbID = i = Min[max, i + 1]; loadImage[i];),
            "UpArrowKeyDown" :> (j = Min[Length[classes], j + 1]; class = classes[[j]]),
            "DownArrowKeyDown" :> (j = Max[1, j - 1]; class = classes[[j]])
        }
    ]
]

Old answer

I used to have an old prototype for this that was in need of a revamp. This gives me the occasion to do it. The goal is to have an interface which makes the drawing of the boxes as easy as possible an eventually returns something that can be used for training.

This version was built to support the following features:

  • interactions

    1. boxes can be drawn by clicking on a corner and then dragging to the other corner
    2. boxes can be drawn by clicking on a corner and then clicking on the other corner
    3. right click while drawing should cancel the current box
    4. right click should cancel the last box drawn
  • options

    1. the coordinates can be clipped to the image range
    2. the range can be scaled or the image size
    3. the UI should offer information about the box edges
    4. the UI should be scalable

image annotation animation


With the goals in mind let's start by putting together each element

Helpers

  • convert coordinates to a region
pointsToRec[{x1_, y1_}, {x2_, y2_}] := Rectangle[{Min[x1, x2], Min[y1, y2]}, {Max[x1, x2], Max[y1, y2]}]
  • clip coordinates to a max range
clipCoords[{x_, y_}, {w_, h_}] := {Clip[x, {0, w}], Clip[y, {0, h}]}
  • scale coordinates to a new range
scaleBox[{w_, h_}][Rectangle[{x1_, y1_}, {x2_, y2_}]] := Rectangle[{x1 w, y1 h}, {x2 w, y2 h}]
scaleBox[{w_, h_}][list: {__Rectangle}] := scaleBox[{w,h}] /@ list

Actions

  • getting the position
"MouseMoved" :> (
            (* store current mouse position *)
            pos = MousePosition["Graphics"];
            If[VectorQ[pos], current = If[clipQ, clipCoords[pos, If[scaledQ, {1, 1}, dims]], pos]];
        )
  • checking for dragging motion
"MouseDragged" :> (
            draggingQ = True;
        )
  • first clik or press
{"MouseDown", 1} :> If[first,
            (* record first box corner *)
            left = current;
            first = False,
        ]
  • second clik or "unpress" (here the boxes are saved as Rectangle[] -> class)
{"MouseDown", 1} :> If[!first,
            (* record second box corner and save box *)
            If[
                EuclideanDistance[left, current] > If[scaledQ, .005, 2],
                AppendTo[box, pointsToRec[left, current] -> class];
                first = True
            ]
        ]
{"MouseUp", 1} :> (
            If[!first && draggingQ,
                (* record second box corner and save box *)
                If[
                    EuclideanDistance[left, current] > If[scaledQ, .005, 2],
                    AppendTo[box, pointsToRec[left, current] -> class];
                    first = True
                ]
            ];
            draggingQ = False
        )
  • right click while dragging or moving
{"MouseClicked", 2} :> If[!first,
            (* cancel current selection *)
            first = True; draggingQ = False,
            (* cancel last selection *)
            If[Length[box] > 0, box = Most[box]]; draggingQ = False
        ],

Controls

  • clip and guides
Checkbox[Dynamic[clipQ]]
Checkbox[Dynamic[guidesQ]]
  • scale
Checkbox[Dynamic[scaledQ, 
    (scaledQ = #;
    If[# && Length[box] > 0,
        box[[All,1]] = scaleBox[1/dims][box[[All,1]]], 
        box[[All,1]] = scaleBox[dims][box[[All,1]]]
    ])&
]]
  • size
Slider[Dynamic[sizeC, (sizeC = #; size = 10^#)&], {2, 4}]

Now we need to put everthing together in a decent UI. My main structure is giong to be

Column[{
    controls,
    EventHandler[
        Graphics[],
        actions
    ]
}]

And here is the UI in all its gory details

ClearAll[boxCreator]
SetAttributes[boxCreator, HoldAll]
boxCreator[Dynamic[box_], Dynamic[class_], Dynamic[i_]] := boxCreator[box, class, i]
boxCreator[box_, class_, i_] := 
DynamicModule[
    {
        (* states *)
        first = True, mouseDown = False, left = {0,0}, current = {0,0}, draggingQ = False, pos = None,
        (* settings*)
        clipQ = True, scaledQ = True, guidesQ = True,
        (* variables *)
        dims, color, margin = 20, size = 400, sizeC, locator
    },
    dims = ImageDimensions[i];
    size = 700;
    sizeC = N @ Log[10, size];
    color = If[ImageMeasurements[i, "MeanIntensity"] < .4, White, Black];
    locator = Graphics@{
                
                {
                    Opacity[Dynamic[.4 Boole[MousePosition["Graphics"] =!= None]]], ColorNegate[color],
                    AbsoluteThickness[3], Circle[{0,0}, 10{1,1}], 
                    Line[{{{-15, 0}, {-5, 0}}, {{5, 0}, {15, 0}}, {{0, -15}, {0, -5}}, {{0, 5}, {0, 15}}}]
                },
                Opacity[Dynamic[Boole[MousePosition["Graphics"] =!= None]]], color,
                Circle[{0,0}, 10{1,1}],
                Line[{{{-15, 0}, {-5, 0}}, {{5, 0}, {15, 0}}, {{0, -15}, {0, -5}}, {{0, 5}, {0, 15}}}]
            };
    Deploy@Column[{
    Row[{
        "Clip", Checkbox[Dynamic[clipQ]],
        "Scale", Checkbox[Dynamic[scaledQ, 
            (scaledQ = #; If[# && Length[box] > 0, box[[All,1]] = scaleBox[1/dims][box[[All,1]]], box[[All,1]] = scaleBox[dims][box[[All,1]]]])&]],
        "Guides", Checkbox[Dynamic[guidesQ]],
        Button["Clear", box = {}; first = True; draggingQ = False]
    }],
    Slider[Dynamic[sizeC, (sizeC = #; size = 10^#)&], {2, 4}],
    Framed[#, FrameMargins->0, FrameStyle->LightGray]& @ 
    EventHandler[
        MouseAppearance[Graphics[{
            (* image *)
            Inset[i, Center, Center, Scaled[1]],
            FaceForm[Opacity[.2, color]], EdgeForm[color],
            (* boxes present in list *)
            {Dynamic[box[[All, 1]]]},
            (* box being currently drawn *)
            {
                Opacity[Dynamic[Boole[!first]]], EdgeForm[Opacity[Dynamic[Boole[!first]]]],
                FaceForm[], Rectangle[Dynamic@left, Dynamic@current]
            },
            (* guide lines *)
            {Opacity[Dynamic[.5 Boole[guidesQ && first && MousePosition["Graphics"] =!= None]], color], InfiniteLine[Dynamic@current, {1,0}], InfiniteLine[Dynamic@current, {0,1}]},
            (* locator *)
            Inset[locator, Dynamic[current], Center, Offset[{25, 25}]]
        },
            ImageSize -> Dynamic[size], PlotRange -> Dynamic@If[scaledQ, {{0,1}, {0,1}}, Tuples[{{0}, dims}]],
            AspectRatio -> ImageAspectRatio[i],
            ImageMargins -> margin
        ], Graphics[]],
        {
        {"MouseDown", 1} :> If[first,
            (* record first box corner *)
            left = current;
            first = False,
            (* record second box corner and save box *)
            If[
                EuclideanDistance[left, current] > If[scaledQ, .005, 2],
                AppendTo[box, pointsToRec[left, current] -> class];
                first = True
            ]
        ],
        {"MouseUp", 1} :> (
            If[!first && draggingQ,
                (* record second box corner and save box *)
                If[
                    EuclideanDistance[left, current] > If[scaledQ, .005, 2],
                    AppendTo[box, pointsToRec[left, current] -> class];
                    first = True
                ]
            ];
            draggingQ = False
        ),
        {"MouseClicked", 2} :> If[!first,
            (* cancel current selection *)
            first = True; draggingQ = False,
            (* cancel last selection *)
            If[Length[box] > 0, box = Most[box]]; draggingQ = False
        ],
        "MouseMoved" :> (
            (* store current mouse position *)
            pos = MousePosition["Graphics"];
            If[VectorQ[pos], current = If[clipQ, clipCoords[pos, If[scaledQ, {1, 1}, dims]], pos]];
        ),
        "MouseDragged" :> (
            draggingQ = True;
        )
        }
    ]
    }]
]

Feedback from OP:

This is great work, thank you!

Your function could easily be a system function with a only 3 tweaks:

Missing essential features:

  1. Needs to work for multiple images (e.g. out-of-core directory of files) and store the labels per image, which requires navigation controls (right and left arrow keys) to cycle through the images as well as a saving/loading to resume work.

  2. Users need to be able to click on a box to edit it after it is drawn, e.g. to move/pan it, resize, or delete it.

  3. Users need to be able to change the class (each should have it's own color) with up and down arrow keys.

Missing nice-to-haves:

  1. Display a scrollable carousel of thumbnails of all the images to visually navigate the directory.

  2. Ability to edit "a master list" of classes and toggle them per box or even add new classes.

  3. This one is very cool: Accelerate the labeling process by continually retraining the detector (in the background on the labels made thus far) to pre-populate high scoring detections (add a slider for users to set thresholds). Then users can simply accept or edit the proposals on unlabeled images instead of dragging them.

$\endgroup$
24
  • 1
    $\begingroup$ I can try and expand the code to accommodate suggestions. For once, I would like to have dynamic class selection and box (fine) editing. $\endgroup$
    – Batracos
    Jul 22, 2022 at 12:40
  • 1
    $\begingroup$ So here's my feedback in a few badly needed and some nice-to-have features. $\endgroup$
    – user5601
    Jul 25, 2022 at 18:52
  • 1
    $\begingroup$ Can we somehow reuse the code for rectangle drawing and editing from Canvas[]? That drawing tools thing gives users the ability to click on already-drawn rectangles to select them and then move, resize, or edit their styles. $\endgroup$
    – user5601
    Jul 25, 2022 at 19:11
  • 1
    $\begingroup$ @Batracos Do you think you could figure out #2 or #3? I'm stuck on implementing selecting/editing/deleting boxes after they are drawn. Neither can I get the arrow keys to work :( $\endgroup$
    – user5601
    Jul 26, 2022 at 6:10
  • 2
    $\begingroup$ @user5601 I tried to address most of your suggestions, Let me know what you think! $\endgroup$
    – Batracos
    Jul 30, 2022 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.