Handle Format when importing package file content unevaluated

Question

This question and its answers explains two methods which allow to import the content of a package file as held expressions. That is very useful as one then can manipulate that imported held code as expressions before evaluating it or writing it back to a new package file.

While making use of that functionality I discovered that something like:

codestr="Format[bin[x_,y_],StandardForm] := MatrixForm[{{x},{y}}]"

(example is from the Format documentation page) will be imported as:

{HoldComplete[RowBox[{"bin", "(", RowBox[{"x_", ",", "y_"}], ")"}] := 
  MatrixForm[{{x}, {y}}]; ]}

with both ImportString[codestr,{"Package","HeldExpressions"}] or ToExpression[codestr,InputForm,HoldComplete]. The problem seems to be that neither of them will really hold Format which seems to be evaluated at parse time (is that maybe unavoidable?).

Altogether that breaks the intended workflow to read, convert to held expression, manipulate and write the result back to a package file like shown at the end of this question.

I know the following workarounds, but none of it looks very satisfactory:

1. rewrite the Format expression to use e.g. MakeBoxes
2. String-Manipulate before converting to held expression, then treat result before evaluating or writing back
3. Use Import as "InactivatedExpression" (which keeps Format inactivated) and work with the result, which seems much more complicated to me and might have additional problems when the code itself makes use of Inactive and friends.

The following is a simplified example which would rename the symbol name in code which makes some definitions for it (for easier experiments I'm getting the code from strings instead of files and also convert back to strings instead of files). Does anyone have a better suggestion how I could get that to work?

codestr="
  Format[bin[x_,y_],TraditionalForm]:=MatrixForm[{{x},{y}}]
  bin[x_?NumericQ,y_?NumericQ]:=Binomial[x,y];
 ";

origcodexpr=ImportString[codestr,{"Package","HeldExpressions"}];

newcodeexpr=origcodexpr/.HoldPattern[bin]:>mybinomial;

newcodeexpr /. {HoldComplete -> 
  Function[c,ToString[Unevaluated[c],InputForm],HoldAllComplete]}

The last line returns:

{"RowBox[{\"mybinomial\", \"(\", 
   RowBox[{\"x_\", \",\", \"y_\"}], \")\"}] := MatrixForm[{{x}, {y}}]; ", 
 "mybinomial[(x_)?NumericQ, (y_)?NumericQ] := Binomial[x, y]; "}

but what I would want is:

{"Format[mybinomial[x_,y_],TraditionalForm]:= MatrixForm[{{x}, {y}}]; ",
  "mybinomial[(x_)?NumericQ, (y_)?NumericQ] := Binomial[x, y]; "}

Of course I am also interested whether there are other functions/constructs than Format which would make the above break...

EDIT

As Leonid (who else) immediatly clarified the problem is actually not the import which works as intendend and advertised. The problem is that when I looked at it in InputForm in the notebook the Format was interpreted. If I had looked at FullForm instead, I would have seen that both ways to import unevaluated expressions does indeed HoldComplete.

Now my problem turns out to be that I don't know how to convert these held expressions back to a string which I can export to a file (of course direct export to a file would also be acceptable...). I tried to follow Leonids idea but unfortunately I can't manage to produce a FullForm string of the unevaluated expressions. To clarify: the ultimate goal for my current use case would be to write the manipulated held expressions to a package file which can then be loaded with Get or Needs. So my original idea to write back as InputForm is stricktly speaking not so important, writing FullForm would be just as good. I could even imagine to compress the unevaluated expression and write that to a file with a wrapper which uncompresses and evaluates the code. Any other suggestions would be appreciated, and of course for other applications it would be nice to know if it would be possible to write out as human readable InputForm nevertheless...

Answer 1

Edit: Quoted comments added to make the answer more general.

Use FullForm (like ToExpression[codestr, InputForm, HoldComplete] // FullForm), and you will see hat everything has actually been imported correctly, it's just the print form that plays tricks with you.

-Leonid Shifrin

So the problem is that ToString takes into account Format "rules". We can block Format then, it is global solution but in case of importing/exporting unchanged expressions to string this is what we need.

Block[{Format},
 newcodeexpr /. {
    HoldComplete -> Function[c, ToString[Unevaluated[c], InputForm], HoldAllComplete]
 }
]

 {
   "Format[mybinomial[x_, y_], TraditionalForm] := MatrixForm[{{x}, {y}}]",
   "mybinomial[(x_)?NumericQ, (y_)?NumericQ] := Binomial[x, y];"
}

Related links:

• Convert expression to string in a reversible manner

• Import package with correct symbol contexts

Answer 2

As Leonid indicated I was actually misinterpreting what I was seeing, the problem was not the import but the export of the held expressions.

Kuba has in his answer shown how to solve that easily.

The only remaining problem is that this is not a general solution but handles Format as a special case. This might well be good enough as I have not seen another case which would not work, but maybe if the code contains more formatting-like constructs there could be other problematic cases.

So to play it save I think the following technique is worth mentioning: We can double check by writing and reimporting whether there are any other problematic constructs in our code. If the result is equal (SameQ) to the held expressions we wrote, then everything seems alright. If not, we would need to further investigate what is going wrong, probably adding other symbols to the Block list:

exportedcodestr = StringRiffle[
   Block[{Format},
     Replace[newcodeexpr,
       Verbatim[HoldComplete][c__]:>ToString[Unevaluated[c],InputForm],
       {1}
     ]
   ],"\n\n"]

ImportString[exportedcodestr,{"Package","HeldExpressions"}]===newcodeexpr

Using the above technique on some more of my code I found two more cases which need special treatment:

1. Something like b*a^-1*2 will be changed to (b*2)/a when turning it to an input expression. It seems that for this special combination of Times and Power some specific rules are applied when converting to an InputForm string. That can be avoided by blocking Times just as Format.

2. Some non-ASCII characters will be read back wrong when we don't explicitly set the character encoding in ToString

so my current solution looks like this:

exportedcodestr = StringRiffle[
   Block[{Format,Times},
     Replace[
       newcodeexpr,
       Verbatim[HoldComplete][c__] :> 
          ToString[Unevaluated[c],InputForm,CharacterEncoding->"ASCII"],
       {1}
     ]
   ],"\n\n"]

Unfortunately that indicates that there might be more not so obvious cases which will need special treatment and the above should not be trusted without double checking...