3
$\begingroup$

I am using Mathematica to create a CSV file to be used by another program (SRIM if needed). The file must end with the EOF character. I get an error from SRIM telling me that the file does not have the EOF character included, so Mathematica does not include it by default.

How do I add the EOF character in a CSV file?

It should be the character ASCII#26

$\endgroup$
  • $\begingroup$ Sounds implausible ... it is quite possible that the problem is not what you think it is. But you can always Import a file as "Byte", add the 26, then export again as"Byte". $\endgroup$ – Szabolcs Jan 31 '19 at 15:35
  • $\begingroup$ It's indeed in the SRIM manual: "A second requirement is that at the TRIM.DAT data, there must be an EOF character (EOF = End Of File). Most editors do this automatically and these characters are essential. If you finish your calculation and suddenly get Error #62, then your data file is missing the EOF character." It sounds some 30 years out of date and DOS/Windows specific ... e.g. on Unix it's #4 (^D) that marks the end of a file, but it should not be explicitly stored. $\endgroup$ – Szabolcs Jan 31 '19 at 15:40
6
$\begingroup$

Adding EOF When Creating The File

If we are creating the CSV file ourselves then we can open a stream, export the CSV, and write the EOF character separately at the end:

$data = {{1, 2, 3}, {4, 5, 6}};

$stream = OpenWrite["test.csv"];
Export[$stream, $data, "CSV"];
WriteString[$stream, FromCharacterCode[26]];
Close[$stream];

We can verify that the result has the EOF character by inspecting the bytes of the file:

Import["test.csv", "Byte"]

(* {49, 44, 50, 44, 51, 13, 10, 52, 44, 53, 44, 54, 13, 10, 26} *)

Note the last character code, 26.

Adding EOF To An Existing File

If the CSV file already exists, then we can update it using OpenAppend. For test purposes, we will create a CSV file that lacks an EOF character:

$data = {{1, 2, 3}, {4, 5, 6}};

Export["test2.csv", $data];

Import["test2.csv", "Binary"]

(* {49, 44, 50, 44, 51, 13, 10, 52, 44, 53, 44, 54, 13, 10} *)

We can append an EOF character as follows:

$stream = OpenAppend["test2.csv"];
WriteString[$stream, FromCharacterCode[26]];
Close[$stream];

So then:

Import["test2.csv", "Byte"]

(* {49, 44, 50, 44, 51, 13, 10, 52, 44, 53, 44, 54, 13, 10, 26} *)

Character Encodings

Given that having EOF characters at the ends of files is a Windows-emulating-DOS-emulating-CP/M phenomenon, it seems likely that the file encoding in use is WindowsANSI. The bytes values shown in these examples reflect that. On non-Windows machines, WindowsANSI should be specified explicitly.

The character encoding should always be explicit if we wished to add text to the end of a file for other purposes, particularly if we are adjusting a pre-existing file. For instance, UTF-16 encoding (known as "Unicode" to Mathematica, formerly styled "UCS-2"):

$stream = OpenWrite["newutf16.csv", CharacterEncoding -> "Unicode"]

or

$stream = OpenAppend["existingutf16.csv", CharacterEncoding -> "Unicode"];

This detail can be important since appending a single character in other encodings might actually write multiple bytes to the file.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Great use of passing Stream to Export here, the other option here is to append FromCharacterCode[26] to the last datum. $\endgroup$ – GenericAccountName Feb 2 '19 at 17:49
  • $\begingroup$ @GenericAccountName You might consider writing an answer showing how to do that. The only ways I can think of would end up writing the EOF character before the final line ending (or introduce a spurious double quote character if added as an extra row). $\endgroup$ – WReach Feb 2 '19 at 18:49
  • $\begingroup$ you are right I forgot about the newline at the end of the file, but delimiters can be turned off with "TextDelimiters" -> None. So there's no way to use Export directly, but if one wanted to remove the \n char this is an Export only solution: Export["test.csv", StringDrop[ExportString[{{"a", "b", "c"}, {"d", "e", "f"}}, "CSV"], -1] <> FromCharacterCode[26], "String"] . I prefer your solution, but this does look a bit simpler. Remove the StringDrop to keep the \n before the EOF obviously. $\endgroup$ – GenericAccountName Feb 2 '19 at 19:55
  • $\begingroup$ BTW CharacterEncoding -> "Unicode" is "Raw 16 byte unicode" which is UCS2, not UTF-16.. I looked up the spec and it looks like SRIM may expect CRLF which WindowsANSI encoding in WL will still use LF.. I am going to add a minimal example as an answer so this doesn't get buried in comments. $\endgroup$ – GenericAccountName Feb 2 '19 at 20:02
  • $\begingroup$ @GenericAccountName Thanks for the comments. I have updated the wording in the character encoding discussion to reflect that it is really only relevant in a more general case -- not the specific case of adding an EOF to a legacy DOS file. $\endgroup$ – WReach Feb 2 '19 at 21:11
4
$\begingroup$

WReach's answer using Streams is nice, I probably would prefer that. But then the line separators are wrong for SRIM (should be CRLF not LF). This should do the job (WindowsANSI because this looks like a DOS program, I'm sure ASCII / ISOLatin1 would be fine as well):

Export["test.csv",
    StringReplace[
        StringDrop[#,-1]&@ (* Remove this line if a CRLF is desired at the end of the file before the EOF*)
        ExportString[
            {{"a","b","c"},{"d","e","f"}}, (* data *)
            "CSV",
            CharacterEncoding -> "WindowsANSI"
        ]
        , 
        "\n"->"\r\n"
    ] <> FromCharacterCode[26]
    ,
    "String"
]
| improve this answer | |
$\endgroup$
1
$\begingroup$

It may be that the file DOES have an EOF character, but it gets consumed due to another data error. Check that you don't have any unclosed strings. For example, the missing quote after Mary:

"John","Bob","Mary,"Phillip"

Or an unescaped quote, like in the third cell here:

"period is .","parenthesis is )","quote is "","plus is +"

Depending on how the parser is reading the file, it might be reading what it thinks is the string inside quotes, and runs into the end of the file, consuming the EOF as part of the string. Then, when the string parser finally fails in some way, the part of the program expecting the EOF after the final line is disappointed, since the EOF has already passed by.

Ideally, the parser would give an error more like "unclosed quote"...but you never know.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.