This is more of an extended comment than an answer.
If one allows for
f = Plus,
g = Times and
h = Conjugate then the rhs of the assignments in the question can be written as
f[g[a[x], h[a[x]]], g[b[x], h[b[x]]]].
Now, forgetting for a moment the attempted assignments and focusing only on the derived expression ie
f[g[a[x], h[a[x]]], g[b[x], h[b[x]]]], it is evident that one cannot assign
UpValues to symbols
a, b, x because they appear too deep in the expression.
If the intended use of the code is to make algebraic manipulations of expressions involving symbols
a, b and their
Conjugate's then perhaps doing something like the following might be helpful:
UpValues for symbols
a, b relating their use wrt to
a /: a[x_] Conjugate[a[x_]] := timesConjugate[a[x]]
b /: b[x_] Conjugate[b[x_]] := timesConjugate[b[x]]
Please note that if the code intends to tackle more symbols like
a, b eg like
c then the list above should be extended in order to cover the use of those symbols, in a similar manner.
UpValue for the auxiliary symbol
timeConjugate wrt to
Plus, when the arguments are
timesConjugate /: timesConjugate[b[x_]] + timesConjugate[a[x_]] = 1
Please note that if more symbols like
a, b are used, then respective definitions of
timeConjugate will have to be added to the list above in order to account for their use.
Now, hopefully after making the above definitions, expressions that involve
a, b in the context of the expressions in the question, should evaluate to unity, as desired.
Hope this helps.