1
$\begingroup$

Is it possible to read a whole file (~100 MBs) into memory, and then use BinaryRead[] function on the data loaded into memory to read/cast it to the types I need with a proper StreamPosition pointer?

I need this because the process is very slow if I do it directly to files, so I prefer to have all manipulations done in memory, especially that I require many memory accesses with many BinaryRead[] calls.

Thank you for any efforts.

Improve this question
$\endgroup$
2
  • $\begingroup$ have you looked into setting up a memory resident virtual file system ? system dependent obviously... $\endgroup$
    – george2079
    Jan 21 '14 at 13:28
  • $\begingroup$ @bills it's not only integers. There are various types I wanna read, and I can't find any casting functionality in Mathematica, that's why I'm asking. $\endgroup$
    – The Quantum Physicist
    Jan 22 '14 at 10:18
4
$\begingroup$

take a look at this:

Create a binary file with mixed types:

 f=OpenWrite["tmp2",BinaryFormat->True];
 BinaryWrite[f,4545.4343,"Real64"];
 BinaryWrite[f,"abc","TerminatedString"];
 BinaryWrite[f,354245,"UnsignedInteger32"];
 Close[f];

read it back in as all "Byte"

f=OpenRead["tmp2",BinaryFormat->True];
bytes=BinaryReadList[f,"Byte"];
Close[f];

recover the data:

BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[1;;8]])]  , "Real64" ]
BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[9;;12]])]  , "TerminatedString" ]
BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[13;;16]])]  , "UnsignedInteger32" ]

  ->  4545.4343 , "abc" , 354245

I don't know why you'd do this vs just reading in the proper format in the first place, unless of course you don't know the format in advance.

Edit..just noticed this can be expressed a bit more compactly, eg.

 BinaryRead[StringToStream@FromCharacterCode@bytes[[13;;16]]  ,
            "UnsignedInteger32" ]

Caveat --- Each "Byte" is stored by mathematica as a 64 bit integer, so your memory usage is 8x the file size.. Don't know a clean way around that..

Edit 2 here's the fix.. stuff everything into a big string:

buf=FromCharacterCode@BinaryReadList[f,"Byte"];

Then read back like this:

BinaryRead[StringToStream@StringTake[buf,{13,16}],"UnsignedInteger32"]
Improve this answer
$\endgroup$
0
$\begingroup$

A little bit late after 5 years...

buff = Import["file_to_read", "String"];

This will read byte by byte the complete content of "file_to_read" into buff. buff will result into a string of 8 bit characters (with integer value 0..255), the StringLength[buff] will exactly match the file size FileByteCount["file_to_read"].

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.