Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a LARGE file (14.6 GB) that I want to separate into several smaller files. Because of its size, using Import/Export is not feasible, I think the most reasonable way of handling this would be using the file as a stream

stream = OpenRead[filepath] 

then with

Read[stream, String]

I can select what I want, and hold the strings in a temporary list until that is large enough to warrant being exported and cleared.

The problem is that when Read[stream,String] encounters ASCII character 0x1A, Mathematica interprets it as an EndOfFile, even when it's not. Any suggestions to get around this?

Edit: https://www.dropbox.com/s/sm51m2s19wtnoqa/allascii.txt is a text file with all ASCII characters, if read in as a stream, this error should be reproduced

share|improve this question
1  
Looks like you need BinaryReadList. –  Sjoerd C. de Vries Jun 28 at 6:56
    
1A is really a kind of EOF: see this. –  Sjoerd C. de Vries Jun 28 at 7:02
    
Is this a binary file or a plain text file? Plain text files shouldn't have that character ... if it's a binary file, use BinaryFormat -> True in OpenRead. –  Szabolcs Jun 28 at 13:54

1 Answer 1

Long, long ago, but right here and not in a galaxy far, far away, there was an operating system called MS-DOS that used the character 0x1A as an end-of-file flag. Windows inherited this in its early days. I would have thought that it was eliminated by now, but perhaps not. Mathematica is old enough that it has a system constant $IgnoreEOF. It should be True by default, but it isn't, so

$IgnoreEOF = True

should fix things.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.