###Edit
Edit
(Corrected error in algorithm)
You can reverse the encryption with Solve
, which can handle modular equations.
With[{alphabet = Alphabet[]},
With[{n = Length[alphabet]},
EncryptChar[char_, a_, b_] :=
Module[{position, shift},
position = Position[alphabet, char][[1, 1]] - 1;
shift = Mod[a position + b, n];
alphabet[[shift + 1]]];
DecryptChar[char_, a_, b_] :=
Module[{position, shift},
position = Position[alphabet, char][[1, 1]] - 1;
shift = Solve[a x + b == position, x, Modulus -> n][[1, 1, 2]];
alphabet[[shift + 1]]]]]
Confirm that the code now works:
(DecryptChar[EncryptChar[#, 3, 5], 3, 5] & /@ Alphabet[]) == Alphabet[]
True
Note: I also cleaned up your code a bit.
###Update
Update
I thought about this some more and decided that the problem would be better solved with associations (hash tables). An advantage of this approach is that the coder doesn't to worry about the indexing issues that trouble the OP's code. It is also is concise and efficient.
With[{chars = Alphabet[]},
With[{n = Length[chars]},
With[{
fwdHash = AssociationThread[chars, Range[0, n - 1]],
bkwHash = AssociationThread[Range[0, n - 1], chars]},
encryptChar[char_, a_, b_] := bkwHash @ Mod[a fwdHash[char] + b, n];
decryptChar[char_, a_, b_] :=
bkwHash @ Solve[a x + b == fwdHash[char], x, Modulus -> n][[1, 1, 2]]]]]
(decryptChar[encryptChar[#, 3, 5], 3, 5] & /@ Alphabet[]) == Alphabet[]
True