Skip to main content
Added some more examples / structure / arguments
Source Link
Leonid Shifrin
  • 114.9k
  • 16
  • 333
  • 424

The real problem here seems somewhat deeper than what the (correctmostly correct) observations in comments indicate. In Mathematica, a number of objects, which are so-called raw objects (including in particular Association, SparseArray, and Rational) require a non-trivial construction stage, which usually happens during evaluation.

In a sense, this to a degree destroys the consistency of the "everything is an expression" principle. Here is where the previous discussion about the lack of the explicit constructors comes tointo play. Had there been explicit constructors for such objects, and there would've been a structural difference between fully formed and not fully formed objects, in which case everything would remain fully consistent with the principle. That said, things are unlikely to change. From the pragmatic viewpoint, one should simply be aware of this, because in the vast majority of cases (unlike the one under discussion) this issue doesn't show up.

The real problem here seems somewhat deeper than what the (correct) observations in comments indicate. In Mathematica, a number of objects, which are so-called raw objects (including in particular Association, SparseArray, and Rational) require a non-trivial construction stage, which usually happens during evaluation.

In a sense, this to a degree destroys the consistency of the "everything is an expression" principle. Here is where the previous discussion about the lack of the explicit constructors comes to play. Had there been explicit constructors for such objects, and there would've been a structural difference between fully formed and not fully formed objects, in which case everything would remain fully consistent with the principle. That said, things are unlikely to change. From the pragmatic viewpoint, one should simply be aware of this, because in the vast majority of cases (unlike the one under discussion) this issue doesn't show up.

The real problem here seems somewhat deeper than what the (mostly correct) observations in comments indicate. In Mathematica, a number of objects, which are so-called raw objects (including in particular Association, SparseArray, and Rational) require a non-trivial construction stage, which usually happens during evaluation.

In a sense, this to a degree destroys the consistency of the "everything is an expression" principle. Here is where the previous discussion about the lack of the explicit constructors comes into play. Had there been explicit constructors for such objects, and there would've been a structural difference between fully formed and not fully formed objects, in which case everything would remain fully consistent with the principle. That said, things are unlikely to change. From the pragmatic viewpoint, one should simply be aware of this, because in the vast majority of cases (unlike the one under discussion) this issue doesn't show up.

Added some more examples / structure / arguments
Source Link
Leonid Shifrin
  • 114.9k
  • 16
  • 333
  • 424

The real problemWhat really happens

Now, here is the crux of the matter. Most such (raw) objects are turned into their InputForm before the WXF serialization. But since there is no way to tell from the InputForm, whether we deal with a fully formed (evaluated) expressions or not, this information, or distinction, is lost. As a result, such objects don't always round-trip. Instead, they are deserialized as their InputForm expression, which needs explicit evaluation to be converted back to the original object. And if they end up inside held (sub)expressions, as parts of a larger expression being deserialized, then such evaluation never happens naturally, and the user would have to perform some post-processing of the deserialized expression, to enable it.   

Associations are an exceptionsome of the exceptions, since WXF format has a special token A to represent fully formed onesassociations. Looking at the documentation, one can see some other types which have special tokens.

  The type Rational does not have a separate token, which is probably the technical reason behind the behavior in question. My familiarity with the WXF is not deep enough to tell whether this is an oversight, or there are good reasons to keep things this way.

So, the Rational[...] object gets transformed into an expression Rational[5, 7] during serialization. When it is deserialized, it remains a normal expression: WXF does not convert it back to the object, because it has no special token, and instead a generic InputForm was used to serialize it. And since it is inside association (which does get fully constructed by WXF - exactly because it is an exception with a special token), it is not re-evaluated by the main evaluator after the deserialization - since fully constructed association does not re-evaluate its parts (formally it is HoldAllComplete).

The real problem

Now, here is the crux of the matter. Most such (raw) objects are turned into their InputForm before the WXF serialization. But since there is no way to tell from the InputForm, whether we deal with a fully formed (evaluated) expressions or not, this information, or distinction, is lost. As a result, such objects don't always round-trip. Instead, they are deserialized as their InputForm expression, which needs explicit evaluation to be converted back to the original object. And if they end up inside held (sub)expressions, such evaluation never happens.  Associations are an exception, WXF format has a special token A to represent fully formed ones. Looking at the documentation, one can see some other types which have special tokens.

  The type Rational does not have a separate token, which is probably the technical reason behind the behavior in question. My familiarity with the WXF is not deep enough to tell whether this is an oversight, or there are good reasons to keep things this way.

So, the Rational[...] object gets transformed into an expression Rational[5, 7] during serialization. When it is deserialized, it remains a normal expression: WXF does not convert it back to the object, because it has no special token, and instead a generic InputForm was used to serialize it. And since it is inside association (which does get fully constructed by WXF - exactly because it is an exception with a special token), it is not re-evaluated by the main evaluator after the deserialization - since fully constructed association does not re-evaluate its parts.

What really happens

Now, here is the crux of the matter. Most such (raw) objects are turned into their InputForm before the WXF serialization. But since there is no way to tell from the InputForm, whether we deal with a fully formed (evaluated) expressions or not, this information, or distinction, is lost. As a result, such objects don't always round-trip. Instead, they are deserialized as their InputForm expression, which needs explicit evaluation to be converted back to the original object. And if they end up inside held (sub)expressions, as parts of a larger expression being deserialized, then such evaluation never happens naturally, and the user would have to perform some post-processing of the deserialized expression, to enable it. 

Associations are some of the exceptions, since WXF format has a special token A to represent fully formed associations. Looking at the documentation, one can see some other types which have special tokens. The type Rational does not have a separate token, which is probably the technical reason behind the behavior in question. My familiarity with the WXF is not deep enough to tell whether this is an oversight, or there are good reasons to keep things this way.

So, the Rational[...] object gets transformed into an expression Rational[5, 7] during serialization. When it is deserialized, it remains a normal expression: WXF does not convert it back to the object, because it has no special token, and instead a generic InputForm was used to serialize it. And since it is inside association (which does get fully constructed by WXF - exactly because it is an exception with a special token), it is not re-evaluated by the main evaluator after the deserialization - since fully constructed association does not re-evaluate its parts (formally it is HoldAllComplete).

Added some more examples / structure / arguments
Source Link
Leonid Shifrin
  • 114.9k
  • 16
  • 333
  • 424

Now, here is the crux of the matter. Most such (raw) objects are turned into their InputForm before the WXF serialization. But since there is no way to tell from the InputForm, whether we deal with a fully formed (evaluated) expressions or not, this information, or distinction, is lost. As a result, such objects don't always round-trip. Instead, they are deserialized as their InputForm expression, which needs explicit evaluation to be converted back to the original object. And if they end up inside held (sub)expressions, such evaluation never happens. Associations are an exception, WXF format has a special token A to represent fully formed ones. Looking at the documentation, one can see some other types which have special tokens.

So, the Rational[...] object gets transformed into an expression Rational[5, 7] during serialization. When it is deserialized, it remains a normal expression: WXF does not convert it back to the object, because it has no special token, and instead a generic InputForm was used to serialize it. And since it is inside association (which does get fully constructed by WXF - exactly because it is an exception with a special token), it is not re-evaluated by the main evaluator after the deserialization - since fully constructed association does not re-evaluate its parts.

This is basically it, so those not interested in more details, can skip the next section and go straight to the summary.

It is important to realize that WXF deserializer can not and should not, by itself, implement any non-trivial evaluation semantics. Its role is only to reconstruct WL expressions based on its spec and the binary serialized form of an expression. It does not call the main evaluator at any stage, and it creates WL expressions based purely on the information it finds in the serialized form. 

For certain specific types of objects, WXF format has special tokens to indicate objects that can be constructed directly in memory. For all the others, they will be constructed as normal WL expressions. If those actually represent raw objects, then they will end up in a not fully constructed form of object-defining expressions, rather than fully formed objects - just as it was the case for Rational in the original example. SomeAnd if they happen to be inside some held (sub)expressions, then the main evaluation procedure won't reach them either, during the normal evaluation of expression after it has been deserialized. So, they will remain expressions. In such cases, some special postprocessing will then be needed to evaluate those in-place and convert them to fully formed objects, already after the expression has been deserialized.


I'd like to stress that this has nothing to do specifically with Association. Here is an example involving just Hold instead:

There is no other way that would make sense for WXF, given that it aims at being language-agnostic and independent of any details of the WL kernel. But even if somehow WXF would be able to call the main evaluator, it would likely be very hard to deserialize arbitrary expressions correctly and efficiently, but also respect all the evaluation semantics of all held (sub)expressions involved in all cases, and avoid all kinds of evaluation leaks - since some parts of the serialized expressions could be held on purpose, and not at all intended for evaluation by the deserializer.

held = Hold @ Evaluate [5/7]
heldProcessed = BinaryDeserialize @ BinarySerialize @held
held === heldProcessed
ByteCount /@ {held, heldProcessed}
Map[f, {held, heldProcessed}, {3}]

(*

  Hold[5/7]

  Hold[Rational[5, 7]]

  True

  {104, 136}

  {Hold[5/7], Hold[Rational[f[5], f[7]]]}

*)

where we observe the same behavior - the round-tripped object became an expression, which shows in how Map and ByteCount produce different results. That despite the fact that SameQ returns True - which can be explained by recalling that SameQ is based on structural comparison. You could also use Complex in place of Rational, and observe the same thing.

OTOH, this shows once again, that there are two semantically inequivalent forms of expressions for many types of objects, which however are treated as syntactically equivalent by Mathematica. Which is at best confusing, and at worst inconsistent, but this structural / semantic impedance mismatch can't be totally removed, since one can't hope to have all objects in the language to behave according to their purely structural form, and be able to do anything useful at the same time. So it must show up somewhere, and this is one of such places.

In a sense, this to a degree destroys the consistency of the "everything is an expression" principle. Here is where the previous discussion about the lack of the explicit constructors comes to play. Had there been explicit constructors for such objects, and there would've been a structural difference between fully formed and not fully formed objects, in which case everything would remain fully consistent with the principle. That said, things are unlikely to change. From the pragmatic viewpoint, one should simply be aware of this, because in the vast majority of cases (unlike the one under discussion) this issue doesn't show up.


Returning back to WXF, there seems to be no other way to operate, that would make sense for it, given that it aims at being language-agnostic and independent of any details of the WL kernel. But even if somehow WXF would be able to call the main evaluator, it would likely be very hard to deserialize arbitrary expressions correctly and efficiently, but also respect all the evaluation semantics of all held (sub)expressions involved in all cases, and avoid all kinds of evaluation leaks - since some parts of the serialized expressions could be held on purpose, and not at all intended for evaluation by the deserializer.

So what is reconstructed back, is an expression representing a fully formed association, not a constructor for one - which is why in this case Rational has not been evaluated (since fully constructed association is inert and does not evaluate its elements). So the Rational inside remains a normal expression, not a formed Rational object. As was suggested in comments, in this particular case a relatively simple evaluation fixes the problem.

In general, the structure of the deserialized expression can be quite complex, and we we may end up with a bunch of raw objects being held unevaluated as some deeply nested parts. What this means is that in general, at present, expressions involving raw objects can't be expected to round-trip properly. The best one can do is to perform certain particular post-processing, like the evaluation that was suggested to fix the original example.

Now, here is the crux of the matter. Most such (raw) objects are turned into their InputForm before the WXF serialization. But since there is no way to tell from the InputForm, whether we deal with a fully formed (evaluated) expressions or not, this information, or distinction, is lost. As a result, such objects don't always round-trip. Associations are an exception, WXF format has a special token A to represent fully formed ones. Looking at the documentation, one can see some other types which have special tokens.

It is important to realize that WXF deserializer can not and should not, by itself, implement any non-trivial evaluation semantics. Its role is only to reconstruct WL expressions based on its spec and the binary serialized form of an expression. It does not call the main evaluator at any stage, and it creates WL expressions based purely on the information it finds in the serialized form. For certain specific types of objects, WXF format has special tokens to indicate objects that can be constructed directly in memory. For all the others, they will be constructed as normal WL expressions. If those actually represent raw objects, then they will end up in a not fully constructed form of object-defining expressions, rather than fully formed objects - just as it was the case for Rational in the original example. Some special postprocessing will then be needed to evaluate those in-place, already after the expression has been deserialized.

There is no other way that would make sense for WXF, given that it aims at being language-agnostic and independent of any details of the WL kernel. But even if somehow WXF would be able to call the main evaluator, it would likely be very hard to deserialize arbitrary expressions correctly and efficiently, but also respect all the evaluation semantics of all held (sub)expressions involved in all cases, and avoid all kinds of evaluation leaks - since some parts of the serialized expressions could be held on purpose, and not at all intended for evaluation by the deserializer.

So what is reconstructed back, is an expression representing a fully formed association, not a constructor for one - which is why in this case Rational has not been evaluated (since fully constructed association is inert and does not evaluate its elements). As was suggested in comments, in this particular case a relatively simple evaluation fixes the problem.

In general, the structure of the deserialized expression can be quite complex, and we may end up with a bunch of raw objects being held unevaluated as some deeply nested parts. What this means is that in general, at present, expressions involving raw objects can't be expected to round-trip properly. The best one can do is to perform certain particular post-processing, like the evaluation that was suggested to fix the original example.

Now, here is the crux of the matter. Most such (raw) objects are turned into their InputForm before the WXF serialization. But since there is no way to tell from the InputForm, whether we deal with a fully formed (evaluated) expressions or not, this information, or distinction, is lost. As a result, such objects don't always round-trip. Instead, they are deserialized as their InputForm expression, which needs explicit evaluation to be converted back to the original object. And if they end up inside held (sub)expressions, such evaluation never happens. Associations are an exception, WXF format has a special token A to represent fully formed ones. Looking at the documentation, one can see some other types which have special tokens.

So, the Rational[...] object gets transformed into an expression Rational[5, 7] during serialization. When it is deserialized, it remains a normal expression: WXF does not convert it back to the object, because it has no special token, and instead a generic InputForm was used to serialize it. And since it is inside association (which does get fully constructed by WXF - exactly because it is an exception with a special token), it is not re-evaluated by the main evaluator after the deserialization - since fully constructed association does not re-evaluate its parts.

This is basically it, so those not interested in more details, can skip the next section and go straight to the summary.

It is important to realize that WXF deserializer can not and should not, by itself, implement any non-trivial evaluation semantics. Its role is only to reconstruct WL expressions based on its spec and the binary serialized form of an expression. It does not call the main evaluator at any stage, and it creates WL expressions based purely on the information it finds in the serialized form. 

For certain specific types of objects, WXF format has special tokens to indicate objects that can be constructed directly in memory. For all the others, they will be constructed as normal WL expressions. If those actually represent raw objects, then they will end up in a not fully constructed form of object-defining expressions, rather than fully formed objects - just as it was the case for Rational in the original example. And if they happen to be inside some held (sub)expressions, then the main evaluation procedure won't reach them either, during the normal evaluation of expression after it has been deserialized. So, they will remain expressions. In such cases, some special postprocessing will then be needed to evaluate those in-place and convert them to fully formed objects, already after the expression has been deserialized.


I'd like to stress that this has nothing to do specifically with Association. Here is an example involving just Hold instead:

held = Hold @ Evaluate [5/7]
heldProcessed = BinaryDeserialize @ BinarySerialize @held
held === heldProcessed
ByteCount /@ {held, heldProcessed}
Map[f, {held, heldProcessed}, {3}]

(*

  Hold[5/7]

  Hold[Rational[5, 7]]

  True

  {104, 136}

  {Hold[5/7], Hold[Rational[f[5], f[7]]]}

*)

where we observe the same behavior - the round-tripped object became an expression, which shows in how Map and ByteCount produce different results. That despite the fact that SameQ returns True - which can be explained by recalling that SameQ is based on structural comparison. You could also use Complex in place of Rational, and observe the same thing.

OTOH, this shows once again, that there are two semantically inequivalent forms of expressions for many types of objects, which however are treated as syntactically equivalent by Mathematica. Which is at best confusing, and at worst inconsistent, but this structural / semantic impedance mismatch can't be totally removed, since one can't hope to have all objects in the language to behave according to their purely structural form, and be able to do anything useful at the same time. So it must show up somewhere, and this is one of such places.

In a sense, this to a degree destroys the consistency of the "everything is an expression" principle. Here is where the previous discussion about the lack of the explicit constructors comes to play. Had there been explicit constructors for such objects, and there would've been a structural difference between fully formed and not fully formed objects, in which case everything would remain fully consistent with the principle. That said, things are unlikely to change. From the pragmatic viewpoint, one should simply be aware of this, because in the vast majority of cases (unlike the one under discussion) this issue doesn't show up.


Returning back to WXF, there seems to be no other way to operate, that would make sense for it, given that it aims at being language-agnostic and independent of any details of the WL kernel. But even if somehow WXF would be able to call the main evaluator, it would likely be very hard to deserialize arbitrary expressions correctly and efficiently, but also respect all the evaluation semantics of all held (sub)expressions involved in all cases, and avoid all kinds of evaluation leaks - since some parts of the serialized expressions could be held on purpose, and not at all intended for evaluation by the deserializer.

So what is reconstructed back, is an expression representing a fully formed association, not a constructor for one - which is why in this case Rational has not been evaluated (since fully constructed association is inert and does not evaluate its elements). So the Rational inside remains a normal expression, not a formed Rational object. As was suggested in comments, in this particular case a relatively simple evaluation fixes the problem.

In general, the structure of the deserialized expression can be quite complex, and we may end up with a bunch of raw objects being held unevaluated as some deeply nested parts. What this means is that in general, at present, expressions involving raw objects can't be expected to round-trip properly. The best one can do is to perform certain particular post-processing, like the evaluation that was suggested to fix the original example.

Added a paragraph on WXF vs evaluation
Source Link
Leonid Shifrin
  • 114.9k
  • 16
  • 333
  • 424
Loading
added 2323 characters in body
Source Link
Leonid Shifrin
  • 114.9k
  • 16
  • 333
  • 424
Loading
Source Link
Leonid Shifrin
  • 114.9k
  • 16
  • 333
  • 424
Loading