###Edit
(Corrected error in algorithm)
You can reverse the encryption with Solve
, which can handle modular equations.
With[{alphabet = Alphabet[]},
With[{n = Length[alphabet]},
EncryptChar[char_, a_, b_] :=
Module[{position, shift, mod},
position = Position[alphabet, char][[1, 1]] - 1;
shift = Mod[a position + b, n];
alphabet[[shift + 1]]];
DecryptChar[char_, a_, b_] :=
Module[{position, shift, mod},
position = Position[alphabet, char][[1, 1]] - 1;
shift = Solve[a x + b == position, x, Modulus -> n][[1, 1, 2]];
alphabet[[shift + 1]]]]]
(DecryptChar[EncryptChar[#, 3, 5], 3, 5] & /@ Alphabet[]) == Alphabet[]
True
Note: I also cleaned up you code a bit.