6
$\begingroup$

I have a .npy file which is created by numpy of Python as below:-

import numpy as np
x = np.arange(60)
y=x.reshape(3,4,5)
np.save("C:\\Users\\H42\\Desktop\\matrix001.npy", y)

I tried to read this in Mathematica, and thus I used:-

data = BinaryReadList["C:\Users\H42\Desktop\matrix001.npy"];
data

However, the output is not what I want. The original matrix is just integers from 0 to 59, with size 3x4x5. But the matrix that I got in Mathematica is totally different, with size 368... Why and what should I do?

Many thanks!

$\endgroup$
  • $\begingroup$ If you are able to change the original python code, you could also use no.savetxt() to save as a text file that is easily read by MMA. If you’re stuck with a .npy file, the solutions below are much better. $\endgroup$ – MassDefect Apr 26 at 14:33
  • $\begingroup$ Thanks for your reply. But I got the files from somewhere else so I can't save it as the format I want. I can import it in Python first and then converted to another format, but that would be better if I can read it in Mathematica directly. $\endgroup$ – H42 Apr 26 at 18:56
  • $\begingroup$ No worries, I figured you probably wouldn't be asking if np.savetxt() was an option, but thought I'd mention it just in case! $\endgroup$ – MassDefect Apr 26 at 19:20
4
$\begingroup$

This worked on Linux.

str = Import["matrix001.npy", "String", Path -> NotebookDirectory[]];
meta = Flatten@StringCases[str, "{" ~~ __ ~~ "}"];

dims = Flatten@StringCases[meta,
    "(" ~~ z__ ~~ ")" :> ToExpression["{" <> z <> "}"]];
nElems = Times @@ dims;
sizeOfInteger = 8 (* bytes *);

binData = StringTake[str, -sizeOfInteger*nElems ;;];
bstream = StringToStream[binData];
raw = BinaryReadList[bstream, "Integer64"];

On[Assert]
Assert[Length[raw] == nElems, 
     "Did not read the correct number of integers"]
Close[bstream];

data = ArrayReshape[raw, dims]

The idea is to read the entire file as a string. The beginning of the string is text metadata, including the array dimensions. The binary data follows. The code parses out the dimensions, but assumes the data is 8-byte integers, but that could have been parsed out also. Then the code reads the end of the string as a binary stream and reshapes the list into the original shape.

It's completely experimental, but seems to work in this case.

$\endgroup$
  • $\begingroup$ The format is specified here. $\endgroup$ – b.gates.you.know.what Apr 26 at 8:17
  • $\begingroup$ Thanks for your reply. I am using Windows 10. I tried the above code but it failed at the first line str = Import["C:\Users\H42\Desktop\matrix001.npy", "String", Path -> NotebookDirectory[]];. What documentation can I look at so that I can fix this line for Windows? $\endgroup$ – H42 Apr 26 at 18:54
  • $\begingroup$ Try it without the Path-> ... option, since the path is already in your filename. $\endgroup$ – LouisB Apr 26 at 19:00
  • $\begingroup$ It doesn't work in Windows. str is just NUMPY\.00v\.00{'descr': '<i4', 'fortran_order': False, 'shape': (3, 4, 5), } and some unrelated characters... but many thanks for help! $\endgroup$ – H42 Apr 26 at 20:22
  • $\begingroup$ Your header contains '<i4'. That's important information. It means numpy wrote 4-type integers. My numpy wrote 8-byte integers. You must change sizeOfInteger to 4 and change the binary element type to "Integer32", for 32 bits. My file is 608 bytes long. That's 128 bytes for the header and 480 bytes for the data. Your file is 368 bytes long, which is 128 bytes for the header and 240 bytes for the data. $\endgroup$ – LouisB Apr 26 at 21:38
6
$\begingroup$

I would suggest reading it with numpy and not Mathematica. It seems to me that this is not an exchange format. It is a format meant to be used only by numpy.

Mathematica 12.0 has significantly improved ExternalEvaluate, and now you can transfer data from Python to Mathematica quite efficiently.

ExternalEvaluate["Python",
 "import numpy as np
 x = np.arange(60)
 y=x.reshape(3,4,5)
 y"
]

enter image description here

Normal[%]
(* {{{0, 1, 2, 3, 4}, {5, 6, 7, 8, 9}, {10, 11, 12, 13, 14}, {15,
    16, 17, 18, 19}}, {{20, 21, 22, 23, 24}, {25, 26, 27, 28, 
   29}, {30, 31, 32, 33, 34}, {35, 36, 37, 38, 39}}, {{40, 41, 42, 43,
    44}, {45, 46, 47, 48, 49}, {50, 51, 52, 53, 54}, {55, 56, 57, 58, 
   59}}} *)
$\endgroup$
  • $\begingroup$ Seems doesn't work for Mathematica 11.3. I will test it again on Mathematica 12 once it's available to me. Thanks! $\endgroup$ – H42 Apr 26 at 20:27
  • $\begingroup$ @H42 It requires Mathematica 12.0. $\endgroup$ – Szabolcs Apr 26 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.