Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Is it possible to read a whole file (~100 MBs) into memory, and then use BinaryRead[] function on the data loaded into memory to read/cast it to the types I need with a proper StreamPosition pointer?

I need this because the process is very slow if I do it directly to files, so I prefer to have all manipulations done in memory, especially that I require many memory accesses with many BinaryRead[] calls.

Thank you for any efforts.

share|improve this question
    
have you looked into setting up a memory resident virtual file system ? system dependent obviously... –  george2079 Jan 21 at 13:28
    
@bills it's not only integers. There are various types I wanna read, and I can't find any casting functionality in Mathematica, that's why I'm asking. –  The Quantum Physicist Jan 22 at 10:18

1 Answer 1

up vote 3 down vote accepted

take a look at this:

Create a binary file with mixed types:

 f=OpenWrite["tmp2",BinaryFormat->True];
 BinaryWrite[f,4545.4343,"Real64"];
 BinaryWrite[f,"abc","TerminatedString"];
 BinaryWrite[f,354245,"UnsignedInteger32"];
 Close[f];

read it back in as all "Byte"

f=OpenRead["tmp2",BinaryFormat->True];
bytes=BinaryReadList[f,"Byte"];
Close[f];

recover the data:

BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[1;;8]])]  , "Real64" ]
BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[9;;12]])]  , "TerminatedString" ]
BinaryRead[ StringToStream[StringJoin@(FromCharacterCode[#]&/@
          bytes[[13;;16]])]  , "UnsignedInteger32" ]

  ->  4545.4343 , "abc" , 354245

I don't know why you'd do this vs just reading in the proper format in the first place, unless of course you don't know the format in advance.

Edit..just noticed this can be expressed a bit more compactly, eg.

 BinaryRead[StringToStream@FromCharacterCode@bytes[[13;;16]]  ,
            "UnsignedInteger32" ]

Caveat --- Each "Byte" is stored by mathematica as a 64 bit integer, so your memory usage is 8x the file size.. Don't know a clean way around that..

Edit 2 here's the fix.. stuff everything into a big string:

buf=FromCharacterCode@BinaryReadList[f,"Byte"];

Then read back like this:

BinaryRead[StringToStream@StringTake[buf,{13,16}],"UnsignedInteger32"]
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.