Different actions with dragging and clicking points in same graphic

Question

On a graphic (in this example a simple plot), I need two points to move when I drag with the mouse, and these points must move to location {1,1} when I click on them.

This is how I got it to work, but I don't think it's a good way.

 DynamicModule[{pt = {{1, 2}, {2, 3}}},
  EventHandler[  LocatorPane[Dynamic[pt], Plot[t, {t, -3, 3}], 
  Appearance -> Graphics[Circle[], ImageSize -> 10]], 
 "MouseClicked" :> (pt = 
 ReplacePart[pt, 
  First@Flatten@
     Sort[MapIndexed[{#2, Norm[MousePosition["Graphics"] - #]} &, 
       pt], #1[[2]] < #2[[2]] &] -> {-1, -1}]), 
 "MouseDragged" :> (pt = 
 ReplacePart[pt, 
  First@Flatten@
     Sort[MapIndexed[{#2, Norm[MousePosition["Graphics"] - #]} &, 
       pt], #1[[2]] < #2[[2]] &] -> MousePosition["Graphics"]])]]

Any suggestions for improvement?