7
$\begingroup$

I want to read in large files as byte strings.

Import[..., "Byte"] yields a list of Integers that takes up much more space (16 times more) than the file.

ByteArray would be the proper way to store such sequences in memory, but how can I read a file into a ByteArray directly?

I know I can later apply ByteArray to the result of Import, but I would prefer to not use so much memory intermittently.

Importing as Integer64 uses much less extra memory, but it's not an option because it does not read in everything when the size is not divisible by 4.

Importing as "String" seems to not use much more memory than the file, but I am not sure whether this might lead to problems since I have arbitrary byte sequences. I'm also not sure how I would index the n-th byte (rather than the n-th multibyte character). String does store multibyte variable length characters (utf-8 style), right? It's not just a wchar_t/short/int array, is it?

Any ideas?

$\endgroup$
  • $\begingroup$ To begin answering this would require knowing what your file looks like, i.e., how the data is stored in the file. $\endgroup$ – bill s Nov 5 '16 at 14:15
  • 1
    $\begingroup$ I don't think multibyte encodings are relevant to your main question, but to answer it: as far as I know, in v11 strings still can't handle characters that can handle UTF16 surrogate pairs. That's because Mathematica got unicode support before the unicode spec was finalized, and got stuck with a 16-bit representation (i.e. can only handle the basic multilingual plane). This should not concern you because when importing as "String", only 8-bit "characters" are returned (i.e. characters represent bytes) $\endgroup$ – Szabolcs Nov 5 '16 at 15:43
  • $\begingroup$ Updated my answer. $\endgroup$ – Szabolcs Nov 6 '16 at 9:54
  • $\begingroup$ @bills I'm interested in the raw bytes in the file, I don't need to know the structure for now. $\endgroup$ – masterxilo Nov 8 '16 at 22:38
  • $\begingroup$ Related: (a/151073). $\endgroup$ – Alexey Popkov Jul 10 '17 at 2:17
7
$\begingroup$

Import[..., "String"] and Export[..., "String"] are meant precisely for this and will not cause problems. This is guaranteed to give you "character codes" between 0..255, and the string can represent the file contents exactly.

This differs significantly from Import[..., "Text"] which will handle character encodings, line endings, etc. and is meant for text, not for arbitrary binary data.


I have not used ByteArray and I am not sure about its purpose, but I got the impression that it is meant for the cryptography functionality as I couldn't find any other high-level functions that work with it. It does work with basic list manipulation functions though.

I know I can later apply ByteArray to the result of Import, but I would prefer to not use so much memory intermittently.

How about reading in chunks, and packing each chunk into a ByteArray, to avoid high memory usage?

chunkSize = 300*1024; (* 300 kB due to the size of my test file *)

stream = OpenRead["file.pdf", BinaryFormat -> True];

ba = Join @@ First@Last@Reap@While[True,
      res = BinaryReadList[stream, "Byte", chunkSize];
      If[res === {}, Break[]]; (* there was no more data to read *)
      Sow[ByteArray[res]]
     ]

Close[stream]
$\endgroup$
  • $\begingroup$ I'm curious to know more specifically what you mean when you say ByteArray "doesn't integrate well with the rest of the system". Could you please elaborate? $\endgroup$ – dionys Nov 5 '16 at 19:38
  • $\begingroup$ @dionys It was unfair of me to say that, so I corrected it. $\endgroup$ – Szabolcs Nov 6 '16 at 9:44
  • 1
    $\begingroup$ Thank you. Another "high-level function" supporting ByteArray is URLRead with element "BodyByteArray". $\endgroup$ – masterxilo Nov 8 '16 at 22:37
  • $\begingroup$ @masterxilo Thanks, it's exactly the kind of function I was looking for but I couldn't find. $\endgroup$ – Szabolcs Nov 9 '16 at 7:51
3
$\begingroup$

You can use the new in M11.3 function ReadByteArray:

path = ExampleData[{"TestImage", "Mandrill"}, "FilePath"];
MaxMemoryUsed[ba = ReadByteArray[path]] //AbsoluteTiming
Head[ba]

{0.000439, 723768}

ByteArray

$\endgroup$
2
$\begingroup$

It seems that using new in version 11.2 StringToByteArray you can reduce the memory requirements:

file = ExampleData[{"TestImage", "Mandrill"}, "FilePath"];

byteList = BinaryReadList[path];
string = Import[path, "String"];
byteList // ByteCount
string // ByteCount
byteArray = StringToByteArray[string]; // MaxMemoryUsed
byteArray    
5024024

945208

945504

ByteArrray

As one can see from the above, String requires 5 times lesser memory than packed array of integers returned by BinaryReadList while StringToByteArray (without second argument) takes only a tiny amount of additional memory.

A more memory efficient representation of the data can be obtained by specifying "ISO8859-1" encoding by the cost of increasing intermediate memory requirements:

byteArray2 = StringToByteArray[string, "ISO8859-1"]; // MaxMemoryUsed
byteArray2
1573240

ByteArray

Hence it would be useful to have an option to import a file as ByteArrray directly.


String does store multibyte variable length characters (utf-8 style), right? It's not just a wchar_t/short/int array, is it?

Citing Itai Seggev:

We internally encode strings in a variant of UTF-8. Now, of course, any byte can be faithfully converted to/from ISO8859-1, but that encoding only equals UTF-8 for the lower 7 bits. For other values, you need to use multiple bytes per character. So using a string to store byte data is both less space efficient and time efficient (since you need to ensure to correct conversion between the two encodings.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.