6
$\begingroup$

I have a very large binary file (several hundred GB) which I would like to process using Mathematica. I only need to process a few bytes of this file at a time, and I know in advance which bytes I would like to read.

How can I directly read the $N$th byte of this file? Functions like OpenRead and BinaryReadList seem to only be able to read from the beginning of the file.

Improve this question
$\endgroup$
3
  • 6
    $\begingroup$ Have a look at StreamPosition / SetStreamPosition / Skip. $\endgroup$
    – Leonid Shifrin
    Dec 16, 2015 at 22:43
  • $\begingroup$ @LeonidShifrin SetStreamPosition is definitely the missing piece I was looking for! Can't believe I missed that. I'll gladly accept if you make that an answer. $\endgroup$
    – David Zhang
    Dec 16, 2015 at 22:58
  • $\begingroup$ I don't think Leonid is interested in writing answers that are less than 2,000 words :-) You may need to write it as an answer yourself so that it doesn't remain an open question. $\endgroup$
    – Jason B.
    Dec 17, 2015 at 9:29

1 Answer 1

Reset to default
2
$\begingroup$

As @LeonidShifrin points out in the comments above, the answer is to first use OpenRead to open the file as an InputStream object, then SetStreamPosition to jump directly to the desired location. Then, BinaryRead and BinaryReadList can be used to read the desired bytes.

For the benefit of future readers, do note that BinaryReadList works with InputStreams, even though its Information only shows it working with file names. Also, SetStreamPosition does not use 1-based indexing like the rest of Mathematica; to read the 5th byte from a file, use SetStreamPosition[stream, 4].

Improve this answer
$\endgroup$
1
  • $\begingroup$ If you are reading a binary file, make sure to include BinaryFormat -> True in OpenRead $\endgroup$
    – Tom
    Jul 22, 2019 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.