How to define floor/ceiling like TraditionalForm?

Question

We have

Floor[x]//TraditionalForm

with output

⌊x⌋

If we define

FracPart[x_]:=x-Floor[x]

How can we get

FracPart[x]//TraditionalForm

to output something like

{x}

Answer 1

I would do this by giving FracPart a TemplateBox formatting rule:

MakeBoxes[FracPart[x_], TraditionalForm]:=TemplateBox[{MakeBoxes[x,TraditionalForm]},
    "FracPart",
    DisplayFunction->(RowBox[{"{",#, "}"}]&),
    Tooltip->"Fractional part"
]

Then:

FracPart[x^2-1] //TraditionalForm

Using a TemplateBox ensures that copy/paste produces a FracPart object instead of a List object, and the Tooltip is a useful reminder that the braces are used to represent the FracPart function instead of a list.

Addendum

The OP asks how this format can be extended to Inactive[FracPart]. Since FracPart would be too deep for UpValues to work, one needs to modify the Inactive symbol instead. However, Inactive already has many FormatValues defined, and one needs to define the new FormatValues so that it takes precedence over the already existing ones. One way to do this is with my Initial function, which I reproduce below:

Initial /: Verbatim[TagSetDelayed][Initial[sym_], lhs_, rhs_] := With[
    {
    new=Block[{sym},
        TagSetDelayed[sym,lhs,rhs];
        First @ Language`ExtendedDefinition[sym]
    ],
    protect=Unprotect[sym]
    },

    sym;
    Quiet@MakeBoxes[sym[],TraditionalForm];

    Unprotect[sym];
    Replace[
        new,
        Rule[values_,n:Except[{}]] :> (
            values[sym] = DeleteDuplicates @ Join[n, values[sym]]
        ),
        {2}
    ];
    Protect@protect;
]

Then, use the Initial wrapper:

Unprotect[Inactive];
Initial[Inactive] /: MakeBoxes[Inactive[FracPart][x_], TraditionalForm] := TemplateBox[
    {MakeBoxes[x,TraditionalForm]},
    "FracPart",
    DisplayFunction->(RowBox[{"{",#,"}"}]&),
    Tooltip->"Fractional part"
]
Protect[Inactive];

Here is the formatting in action:

Inactive[FracPart][2.3] //TraditionalForm

{2.3}

Answer 2

As noted in Stork's comment, your requested notation is ill advised. If you still want to shoot yourself in the foot, try the following.

Clear[FracPart];

FracPart[x_?NumericQ] := x - Floor[x];

FracPart /: Format[FracPart[x_], TraditionalForm] := 
 Interpretation[
  RawBoxes[RowBox[{"{", ToBoxes[x, TraditionalForm], "}"}]], 
  FracPart[x]]

The condition in the definition is so that only numerical values expand.

{FracPart[5/4], FracPart[x], FracPart[x] // TraditionalForm}
(*{1/4, FracPart[x], {x}}*)

The downside of using TraditionalForm is that the wrap prevents further calculations.

TraditionalForm[1] - 1
(* -1 + 1 *)