TBH I didn't really read the question, but here's a way to make a custom object with a constructor GUI that I think follows what you want.
First we define a validator function to make sure the data is good and a constructor to get the data into the canonical format
ClearAll[BinaryMatrix, validateMatrix, constructMatrix];
validateMatrix[vals_List] :=
Dimensions[vals] === {3, 3} &&
Developer`PackedArrayQ[vals] &&
Head[vals[[1, 1]]] == Integer;
constructMatrix[vals_List] :=
If[Not@Developer`PackedArrayQ[vals],
{Developer`ToPackedArray[vals], True},
{vals, False}
]
In this case I'm forcing a 3x3 packed array of integers (so not really binary). The second return value from constructMatrix
simply tells us if the object needs to be rebuilt.
Next we define a constructor that sets the Valid
flag and a nice format form
b : BinaryMatrix[vals_List]?System`Private`HoldNotValidQ :=
Block[{data, rebuild, isValid},
{data, rebuild} = constructMatrix[vals];
If[rebuild, isValid = True, isValid = validateMatrix[data]];
If[rebuild,
BinaryMatrix[data],
System`Private`HoldSetValid[b]
] /; isValid
];
Format[b_BinaryMatrix?System`Private`ValidQ] :=
Interpretation[
BinaryMatrix[MatrixForm[b[[1]]]],
b
];
We can make one of these matrices like this, then

Note that the mis-shapen one doesn't format because it failed to validate.
Finally, we can make our constructor GUI. In this case I'll use Checkbox
to make a table of 0 and 1. The key is that we provide a BoxID
so we can replace the GUI using BoxReferenceReplace
once we're done. I set this up to be very modular so that it can be easily adapted for other objects.
ClearAll[initializeMatrixInputGUIData, createMatrixInputGUI,
constructMatrixFromGUIData, overwriteGUIWithMatrix];
initializeMatrixInputGUIData[] := ConstantArray[False, {3, 3}];
createMatrixInputGUI[Dynamic[data_]] :=
Panel@Grid@
Table[
With[{i = i, j = j}, Checkbox[Dynamic[data[[i, j]]]]],
{i, 3},
{j, 3}
];
constructMatrixFromGUIData[data_] := BinaryMatrix[Boole@data]
overwriteGUIWithMatrix[mat_, id_] :=
MathLink`CallFrontEnd@
FrontEnd`BoxReferenceReplace[
FE`BoxReference[
EvaluationNotebook[],
{{id}},
FE`SearchStart -> "StartFromBeginning"
],
ToBoxes@mat
]
BinaryMatrix[] :=
With[{id = CreateUUID[]},
DynamicModule[
{data = initializeMatrixInputGUIData[]},
Column[{
createMatrixInputGUI[Dynamic[data]],
Button["Done",
overwriteGUIWithMatrix[constructMatrixFromGUIData[data], id]
]
},
Alignment -> Center
],
BoxID -> id
]
]
This looks like

and after pressing Done

I have no idea if this solves your problem, but realistically it gives you the toolset you need to get the job done.
This is also easy to work into the Dynamic[outputVar]
paradigm like
BinaryMatrix[Dynamic[var_]] :=
With[{id = CreateUUID[]},
DynamicModule[
{data = initializeMatrixInputGUIData[]},
Column[{
createMatrixInputGUI[Dynamic[data]],
Button["Done",
var = constructMatrixFromGUIData[data];
overwriteGUIWithMatrix[Defer[var], id]]
},
Alignment -> Center
],
BoxID -> id]
]
where now Done
will replace the object with the variable you specified for the output
Or if you want to allow for initializing the object/more advanced Dynamic
controls
initializeMatrixInputGUIData[BinaryMatrix[v_]?System`Private`ValidQ] :=
Map[(# === True || # != 0) &, v, {2}];
initializeMatrixInputGUIData[l_List] :=
Replace[constructMatrix[l],
{
{a_List?validateMatrix, _} :> Map[
If[Head[#] === Integer, # != 0, TrueQ[#]] &, a, {2}
],
{a_List?(Dimensions[#] === {3, 3} && MatrixQ[#, BooleanQ] &), _} :> Map[
If[Head[#] === Integer, # != 0, TrueQ[#]] &, a, {2}
],
e_ :> initializeMatrixInputGUIData[]
}
];
initializeMatrixInputGUIData[e_] := initializeMatrixInputGUIData[];
BinaryMatrix[Dynamic[var_]] :=
With[{id = CreateUUID[]},
DynamicModule[
{},
var = initializeMatrixInputGUIData[var];
Column[{
createMatrixInputGUI[Dynamic[var]],
Button["Done",
var = constructMatrixFromGUIData[var];
overwriteGUIWithMatrix[Defer[var], id]]
},
Alignment -> Center
],
BoxID -> id]
]