3
$\begingroup$

I have a LARGE file (14.6 GB) that I want to separate into several smaller files. Because of its size, using Import/Export is not feasible, I think the most reasonable way of handling this would be using the file as a stream

stream = OpenRead[filepath] 

then with

Read[stream, String]

I can select what I want, and hold the strings in a temporary list until that is large enough to warrant being exported and cleared.

The problem is that when Read[stream,String] encounters ASCII character 0x1A, Mathematica interprets it as an EndOfFile, even when it's not. Any suggestions to get around this?

Edit: https://www.dropbox.com/s/sm51m2s19wtnoqa/allascii.txt is a text file with all ASCII characters, if read in as a stream, this error should be reproduced

$\endgroup$
3
  • 1
    $\begingroup$ Looks like you need BinaryReadList. $\endgroup$ Jun 28, 2014 at 6:56
  • $\begingroup$ 1A is really a kind of EOF: see this. $\endgroup$ Jun 28, 2014 at 7:02
  • $\begingroup$ Is this a binary file or a plain text file? Plain text files shouldn't have that character ... if it's a binary file, use BinaryFormat -> True in OpenRead. $\endgroup$
    – Szabolcs
    Jun 28, 2014 at 13:54

1 Answer 1

3
$\begingroup$

Long, long ago, but right here and not in a galaxy far, far away, there was an operating system called MS-DOS that used the character 0x1A as an end-of-file flag. Windows inherited this in its early days. I would have thought that it was eliminated by now, but perhaps not. Mathematica is old enough that it has a system constant $IgnoreEOF. It should be True by default, but it isn't, so

$IgnoreEOF = True

should fix things.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.