6
$\begingroup$

I have a very large binary file (several hundred GB) which I would like to process using Mathematica. I only need to process a few bytes of this file at a time, and I know in advance which bytes I would like to read.

How can I directly read the $N$th byte of this file? Functions like OpenRead and BinaryReadList seem to only be able to read from the beginning of the file.

$\endgroup$
  • 6
    $\begingroup$ Have a look at StreamPosition / SetStreamPosition / Skip. $\endgroup$ – Leonid Shifrin Dec 16 '15 at 22:43
  • $\begingroup$ @LeonidShifrin SetStreamPosition is definitely the missing piece I was looking for! Can't believe I missed that. I'll gladly accept if you make that an answer. $\endgroup$ – David Zhang Dec 16 '15 at 22:58
  • $\begingroup$ I don't think Leonid is interested in writing answers that are less than 2,000 words :-) You may need to write it as an answer yourself so that it doesn't remain an open question. $\endgroup$ – Jason B. Dec 17 '15 at 9:29
2
$\begingroup$

As @LeonidShifrin points out in the comments above, the answer is to first use OpenRead to open the file as an InputStream object, then SetStreamPosition to jump directly to the desired location. Then, BinaryRead and BinaryReadList can be used to read the desired bytes.

For the benefit of future readers, do note that BinaryReadList works with InputStreams, even though its Information only shows it working with file names. Also, SetStreamPosition does not use 1-based indexing like the rest of Mathematica; to read the 5th byte from a file, use SetStreamPosition[stream, 4].

$\endgroup$
  • $\begingroup$ If you are reading a binary file, make sure to include BinaryFormat -> True in OpenRead $\endgroup$ – Tom Jul 22 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.