Read a binary file to an image in memory then analyze it

Question

Is it possible to read a whole file (~100 MBs) into memory, and then use BinaryRead[] function on the data loaded into memory to read/cast it to the types I need with a proper StreamPosition pointer?

I need this because the process is very slow if I do it directly to files, so I prefer to have all manipulations done in memory, especially that I require many memory accesses with many BinaryRead[] calls.

Thank you for any efforts.

Answer 1

take a look at this:

Create a binary file with mixed types:

 f=OpenWrite["tmp2",BinaryFormat->True];
 BinaryWrite[f,4545.4343,"Real64"];
 BinaryWrite[f,"abc","TerminatedString"];
 BinaryWrite[f,354245,"UnsignedInteger32"];
 Close[f];

read it back in as all "Byte"

f=OpenRead["tmp2",BinaryFormat->True];
bytes=BinaryReadList[f,"Byte"];
Close[f];

recover the data:

BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[1;;8]])]  , "Real64" ]
BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[9;;12]])]  , "TerminatedString" ]
BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[13;;16]])]  , "UnsignedInteger32" ]

  ->  4545.4343 , "abc" , 354245

I don't know why you'd do this vs just reading in the proper format in the first place, unless of course you don't know the format in advance.

Edit..just noticed this can be expressed a bit more compactly, eg.

 BinaryRead[StringToStream@FromCharacterCode@bytes[[13;;16]]  ,
            "UnsignedInteger32" ]

Caveat --- Each "Byte" is stored by mathematica as a 64 bit integer, so your memory usage is 8x the file size.. Don't know a clean way around that..

Edit 2 here's the fix.. stuff everything into a big string:

buf=FromCharacterCode@BinaryReadList[f,"Byte"];

Then read back like this:

BinaryRead[StringToStream@StringTake[buf,{13,16}],"UnsignedInteger32"]

Answer 2

A little bit late after 5 years...

buff = Import["file_to_read", "String"];

This will read byte by byte the complete content of "file_to_read" into buff. buff will result into a string of 8 bit characters (with integer value 0..255), the StringLength[buff] will exactly match the file size FileByteCount["file_to_read"].