privKey = PrivateKey[<|"Cipher" -> "RSA", "Padding" -> None,
"PublicExponent" -> 80239, "PublicModulus" -> 16678327,
"PrivateExponent" -> 8124550|>];
(*private exponent was calculated as
PowerMod[PublicExponent,-1,PublicModulus]*)
pubKey = PublicKey[<|"Cipher" -> "RSA", "Padding" -> None,
"PublicExponent" -> 80239, "PublicModulus" -> 16678327|>];

Decrypt[privKey, ByteArray[{1, 0, 0}]] // Normal // BaseForm[#, 16] &
(*unpredictable result*)
Decrypt[privKey, ByteArray[{1, 0, 0}]] // Normal // BaseForm[#, 16] &
(*unpredictable result*)

Decrypt[pubKey, ByteArray[{1, 0, 0}]] // Normal // BaseForm[#, 16] &
(*nice, repeatable result*)
Decrypt[pubKey, ByteArray[{1, 0, 0}]] // Normal // BaseForm[#, 16] &
(*nice, repeatable result*)


The above is a simple RSA decryption; I just started playing with the cryptography support in Mma 10.1. I am not concerned with the actual values returned or even whether the PrivateExponent is correct, but I am somewhat surprised that the Decrypt[privKey, …] lines would return non-repeatable and ever-changing results. A kernel restart produces a new series of results instead of repeating the most recent one.

What am I doing wrong?

• Cautiously and carefully look in Wolfram Research/Mathematica/10.1/SystemFiles/Components/Cryptography/EncryptDecrypt.m for the Mathematica source code for Encrypt and Decrypt to attempt to understand exactly why it does what it does. Be very careful that you don't break anything. – Bill Jul 8 '15 at 3:01
• Thanks for the suggestion, Bill. The trace ends at SystemFiles\Components\Cryptography\Resources\Libraries\OpenSSLLink64.dll, which is loaded by EncryptDecrypt.m. Regrettably, there seems to be no documentation on the contents of the RSA structure built by the glue code in this DLL before it calls into OpenSSL (or libeay32). This notwithstanding, I still think that in the absence of padding, randomly generated IVs and such, a simple decryption should produce a repeatable result. Perhaps nobody ever tested with "Padding" -> "None"? – Felix Kasza Jul 8 '15 at 11:48
• Under Details in the documentation for Decrypt it says the second argument is to be a valid encrypted message. Using FullForm on a few example encrypted messages shows these appear to be very different from an arbitrary three byte array that I might make up. I'm not trying to defend them, but it seems that you are passing bad input to a function and puzzled at the output you get. If you could show a similar error when decrypting a valid encrypted message then that would be more interesting. – Bill Jul 8 '15 at 14:03
• The thing is, a ByteArray[] is valid (as long as either it conforms to the padding spec, or "Padding" -> "None", in which case it is the raw encrypted message with length <= length of modulus). The docs for Decrypt have an example, the last one in section "Scope": Decrypt can be passed a ByteArray: The code to which you pointed me, EncryptDecrypt.m, also deals specifically with ByteArray[]. – Felix Kasza Jul 8 '15 at 14:20

Your privKey is not a valid RSA key, so I wouldn't expect Decrypt to work properly with it. The formula you've used PrivateExponent = PowerMod[PublicExponent,-1,PublicModulus] makes no sense and contradicts the purpose of private key cryptography. Using your formula means anyone who has the public key can easily calculate the private key, then it's not private anymore. The privKey you feed into Decrypt is now basically a valid PublicExponent, a valid PublicModulus, and a random PrivateExponent. Moreover, the above privKey and pubKey are not a pair. Decrypt here could throw an error about the key being invalid, but its definitely not supposed to work properly.
• 2/ @Dariia Porechna More to the point, I am of your opinion that bad input need not produce sensible output, although some error checking would be nice. An earlier commenter, @Bill, mentioned that my ByteArray[{1,0,0}] was not a valid RSA message; actually, it is valid both as plaintext and as ciphertext, because I specified Padding->None and because i made veru sure to remain below the size limit for a single RSA message given my key. – Felix Kasza Jan 8 '20 at 9:29