10
$\begingroup$

The "PrintableASCII" characters set consists of 97 characters, including space (\[RawSpace]) and two control characters: the newline character \n (\[NewLine]) and the carriage return \r (\[RawReturn]), which are the only allowed WhitespaceCharacters in this encoding (highlighted by light blue in the following table):

printableASCIISet = 
  Cases[ToCharacterCode[FromCharacterCode[Range[0, 255]], "PrintableASCII"], k_Integer];

printableASCIISet // Length

Grid[
  Flatten[
   Transpose /@
    Partition[{#, FromCharacterCode[#]} & /@ printableASCIISet, UpTo@20], 1],
  Frame -> {All, {{True, False}}}, Spacings -> 0.2, FrameStyle -> LightGray] /.
 s_String /; StringMatchQ[s, WhitespaceCharacter] :>
  Item[StringTrim[ToString[InputForm@s], "\""], Background -> LightBlue]
97

output

The "ASCII" encoding allows all control characters and consists of the characters with codes from 0 to 127.

I wish to allow only newlines (\n), carriage returns (\r), spaces () and tabs (\t) as whitespace characters when exporting as ASCII. So the only difference with "PrintableASCII" is that tabs are also allowed, and of course all the special, extended-ASCII and Unicode characters must be converted into their corresponding FullForms.

Here are example strings with expected output in the exported textual file:

  1. Input:

    ExportAsASCII["test1.txt", "Here is a string in quotes:\n\"quoted string\""]
    

    Verbatim contents of the created file "test1.txt":

    Here is a string in quotes: "quoted string"

    (note the absence of the outer quotation marks and also that the inner quotes aren't escaped).

  2. Input:

    ExportAsASCII["test2.txt", "Lamé \[LongRightArrow] αβ+"]
    

    Verbatim contents of the created file "test2.txt":

    Lam\[EAcute] \[LongRightArrow] \[Alpha]\[Beta]+

    (note the é encoded as \[EAcute], not as \\[EAcute]).

  3. Input:

    s = "\!\(\r\n\*SubsuperscriptBox[\(\[Integral]\), \(-Pi\), \(Pi\)]\(g[\r\n\*SuperscriptBox[\(x\), \(2\)] + \r\n\*SuperscriptBox[\(x\), \(3\)] + \r\n\*SuperscriptBox[\(x\), \(6\)]] \[DifferentialD]x\)\)"
    
    ExportAsASCII["test3.txt", s]
    

    Verbatim contents of the created file "test3.txt":

    \!\( \*SubsuperscriptBox[\(\[Integral]\), \(-Pi\), \(Pi\)]\(g[ \*SuperscriptBox[\(x\), \(2\)] + \*SuperscriptBox[\(x\), \(3\)] + \*SuperscriptBox[\(x\), \(6\)]] \[DifferentialD]x\)\)

    (note that the newline characters aren't escaped and present here as actual new lines).

How is it possible to implement ExportAsASCII?


P.S. Actually the above-described behavior is (by the essence) what I expected for Exporting as "String" with CharacterEncoding -> "ASCII" according to the Documentation:

Export["test.txt", "Lamé \[LongRightArrow] αβ+", "String", CharacterEncoding -> "ASCII"]

But currently "ASCII" as well as "PrintableASCII" and None seem to be simply ignored by Export. :(

$\endgroup$
5
  • $\begingroup$ Any feedback from WRI about this and related issues? $\endgroup$
    – Kuba
    May 19, 2017 at 9:00
  • $\begingroup$ @Kuba I have not yet written to the support, still trying to figure it out on my own. $\endgroup$ May 19, 2017 at 9:06
  • 1
    $\begingroup$ @Kuba The Support responded: "There seems to be a disregard for certain characters when Exporting with ASCII as the CharacterEncoding, regardless of their character codes." As a workaround is proposed manual conversion of such characters into their full forms using helper function fullname[c_String]:=StringReplace[ToString[c,InputForm,CharacterEncoding->"ASCII"],"\"\\["~~a__~~"]\"":>"\\["~~a~~"]"]. $\endgroup$ Jun 6, 2017 at 7:51
  • 1
    $\begingroup$ You are misreading the documentation if you think "String" would be appropriate for this. "String" is a binary format, which by design ignores CharacterEncoding, and converts characters 0-255 to the corresponding byte values (and other characters into long names). $\endgroup$ Jul 7, 2017 at 23:24
  • $\begingroup$ @ItaiSeggev This comment serves as a proper answer to this question. Note that some Unicode characters aren't converted into long names, for example \[CirclePlus]. Should it be counted as a bug? $\endgroup$ Jul 8, 2017 at 4:51

1 Answer 1

9
$\begingroup$

ExportAsASCII

Currently I know only one well-documented and working (with exception for the NULL character \.00 and a few characters from Unicode Private Use Area which normally shouldn't appear in code) method to convert all special and non-ASCII characters in an arbitrary string into "PrintableASCII" programmatically:

toInputForm[s_String] := StringTake[
   ToString[s, InputForm, CharacterEncoding -> "PrintableASCII"], {2, -2}];

In general this method duplicates every backslash which is a part of a special character like \n, \!, \* etc.:

toInputForm["\!\n"]
"\\!\\n"

... and adds two backslashes before every escaping backslash:

toInputForm["\\"]
toInputForm["\""]
toInputForm["\\n"]
"\\\\"
"\\\""
"\\\\n"

AFAIK the only two characters which have to be escaped inside of the Mathematica strings are backslash \ (the escaping character and the integral part of every special character) and double quote " (which indicates the start and the end of the string). This is important since for performing the reverse transformation we have to care about every such character.

Based on these considerations we can write a list of replacement rules which will perform the reverse transformation according to our goals:

unescape[s_String] := 
  StringReplace[
   s, {"\\\\" -> "\\", "\\\"" -> "\"", "\\n" -> "\n", "\\r" -> "\r", "\\t" -> "\t"}];

Now we can implement ExportAsASCII as follows:

ExportAsASCII[fileName_String, s_String] := 
 Export[fileName, unescape[toInputForm[s]], "String"]

... or perhaps more efficiently using WriteString:

ExportAsASCII[fileName_String, s_String] :=
  Module[{str = OpenWrite[fileName, BinaryFormat -> True]},
   WriteString[str, unescape[toInputForm[s]]]; Close[str]];

... or BinaryWrite:

ExportAsASCII[fileName_String, s_String] :=
  Close[BinaryWrite[fileName, unescape[toInputForm[s]]]];

This implementation passes all the tests given in the question. Note however that this implementation has the same defect as Export[..., "String"] for Unicode characters: in the exported file escaped and unescaped versions of the same non-PrintableASCII character can't be distinguished. For example, both from "\!" and "\\!" Export produces a file containing \!. There is no way to avoid such inconsistency entirely since if we will write \! from "\!" and \\! from "\\!", we still have ambiguity for "\\!" and "\\\!" which both will be written as \\!. That's because we must write "\\" as single backslash \ it actually represents.


ImportAsASCII

Since non-PrintableASCII characters are presented in FullForm in the exported file, they will be imported already escaped like "\\[Paragraph]", "\\200" or "\\!". We must avoid escaping them again when we replace "\\" with "\\\\" (the reverse of what unescape does). Let us find all characters which may appear after \ in FullForm of a non-PrintableASCII character (excluding \t which I include in my character set):

mySet = Join[ToCharacterCode["\t"], 
   Cases[ToCharacterCode[FromCharacterCode[Range[0, 255]], "PrintableASCII"], k_Integer]];
disallowed = Complement[Range[0, 65535], mySet];
chs = toInputForm[FromCharacterCode[#]] & /@ disallowed;
sel = Select[chs, StringTake[#, 1] === "\\" &];
DeleteDuplicates[StringTake[#, {2}] & /@ sel]
{"0", "b", "f", "[", "1", "2", ":", ")", "!", "@", "%", "^", "&", "*", "(", "_", "+", "/", "`"}

The named characters (like \[Mod2Key]) and input forms \:xxxx (where xxxx is hexadecimal code) and \xxx (where xxx contains only decimals) can be matched by "\\" ~~ RegularExpression["\\[[A-z12]+\\]|:[a-f\\d]{4}|\\d{3}"] (note that this regex is exhaustive but still excessive, and will match also invalid inputs like \[foo] or \999). There also is partially allowed input form \.xx which normally isn't generated by toInputForm, so I do not account for it.

Hence a regular expression matching "\\" which will protect such characters may be written as

temperedBackslash = 
  RegularExpression["\\\\(?![bf()!@%^&*_+`/]|\\[[A-z12]+\\]|:[a-f\\d]{4}|\\d{3})"];

Here is a function which performs the reverse operation:

escape[s_String] := 
  StringReplace[s, {temperedBackslash -> "\\\\", "\"" -> "\\\"", 
                    "\n" -> "\\n", "\r" -> "\\r", "\t" -> "\\t"}];

ImportAsASCII[fileName_String] :=
 ToExpression["\"" <> escape[FromCharacterCode[BinaryReadList[fileName]]] <> "\"", 
  InputForm]

Consistency check:

string = "\!\(\r\n\*Lamé \[LongRightArrow] αβ+\\\\\\\\\)⊕\[Wolf]\[RuleDelayed]\"";
string === ImportAsASCII[ExportAsASCII["test1.txt", string]]
True
$\endgroup$
2
  • 2
    $\begingroup$ Games like this are dangerous. Consider the string s="\"\\\"\\n\\\"\"". It will not export correctly (it produces a file containing "\"\n\""), and on import will generated an invalid syntax error. Which isn't to say your function is not useful in practice. I don't think there is a built-in function which does this, in part because you are wanting to have it both ways: some special characters are themselves, others are not. How is the function supposed to know which is which? (Also, there are special rules for interpretting the contents insde \!\(...\) which are being ignored.) $\endgroup$ Jul 7, 2017 at 23:34
  • $\begingroup$ @ItaiSeggev Thank you for this example. With your string Export (as "String") behaves exactly as ExportAsASCII. In some other cases this introduces inconsistency since, for example, both from "\!" and "\\!" Export produces a file containing \!. I do not know whether this is intended behavior or a bug. Obviously it isn't appropriate for ExportAsASCII but I still can't figure out how to avoid such inconsistency. I've improved ImportAsASCII: now it works correctly with your example. $\endgroup$ Jul 8, 2017 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.