28
$\begingroup$

I've been using np.savetxt(location, result, delimiter=',') in Python followed by Import[location,"CSV"] in Mathematica. Some of my files are around 1GB, opening seems a bit slow, what else do people use for this?

$\endgroup$
32
$\begingroup$

You can use the binary format to speed up the process:

python side

import numpy as np
array = np.random.rand(100000000);
array.astype('float32').tofile('np.dat')

Mathematica side

data = 
   BinaryReadList["np.dat", "Real32"]; // AbsoluteTiming
(* {2.56679, Null} *)

data // Dimensions
(* {100000000} *)
$\endgroup$
  • $\begingroup$ Amazing, that cut my data loading time from 26.7544 seconds to 0.068415 seconds $\endgroup$ – Yaroslav Bulatov Apr 28 '17 at 18:21
  • 2
    $\begingroup$ @YaroslavBulatov Glad it is helpful. I'm wondering how often do you use Mathematica in your work in ML? I'm considering switch my career to ML and am wondering whether my Mathematica skills are transferable. $\endgroup$ – xslittlegrass Apr 28 '17 at 19:54
  • $\begingroup$ I guess it depends on the kind of ML. I work on neural networks, and I use Mathematica for small scale experiments like this, as well as checking math and making pretty plots. After sanity checks are done, rewrite everything in TensorFlow for large scale experiments. $\endgroup$ – Yaroslav Bulatov Apr 28 '17 at 20:06
  • 3
    $\begingroup$ One way to get into OpenAI is to have state of the art implementations of some learning algorithms. Winning vizdoom contest, or imagenet challenge or some other hard contest would look pretty good. $\endgroup$ – Yaroslav Bulatov Apr 28 '17 at 23:32
  • 5
    $\begingroup$ @YaroslavBulatov xslittlegrass I would also try HDF5 as the transfer format. It knows the type, so you don't have the remember that the file was 32-bit floating point. $\endgroup$ – Szabolcs Apr 29 '17 at 6:34
10
$\begingroup$

Here is a template to read a numpy binary ".npy" file created simply by

numpy.save(filename,array)

this file format has the array structure encoded as a python string that we need to parse..

{'descr': '<f8', 'fortran_order': False, 'shape': (3, 4, 5), }

the byte order is also encoded so that this format is portable across hardware.

all of this code is parsing a small header and the actual data read is a single BinaryReadList so it should be very fast..

getnpy[file_] := 
  Module[{a, f = OpenRead[file, BinaryFormat -> True], version, 
    headerlen, header, dims, type, typ, byto},
    a = If[
     BinaryRead[f, "Byte"] == 147 &&         
      BinaryReadList[f, "Character8", 5] == Characters["NUMPY"] ,
     version = BinaryReadList[f, "Byte", 2];
     headerlen = BinaryRead[f, "Integer16", ByteOrdering -> -1];
     header = StringJoin@BinaryReadList[f, "Character8", headerlen];
     dims = StringCases[header,"'shape':" ~~ Whitespace ~~ "(" ~~ 
                 s : {NumberString, ",", Whitespace} .. ~~ ")" :> 
            ToExpression[
            "{" ~~
            If[StringTake[s, -1] == ",", StringDrop[s, -1], s] ~~ 
            "}"]][[1]];
     type = 
      StringCases[header, 
        "'descr':" ~~ Whitespace ~~ 
          Shortest["'" ~~ s : _ ... ~~ "'"] :> s][[1]];
     byto = 
      Switch[StringTake[type, 1], "<", -1, ">", 1, _, $ByteOrdering];
     If[MemberQ[{"<", ">","|","="}, StringTake[type, 1]], 
      type = StringDrop[type, 1]];
     typ = 
      Switch[ type ,
        "f8" , "Real64" ,
        "i8" , "Integer64" ,
          _  , Print["unknown type", header]; 0];
     If[typ != 0, 
      ArrayReshape[BinaryReadList[f, typ, ByteOrdering -> byto], 
       dims], 0 ], Print["not a npy"]; 0];
   Close[f]; a];

getnpy["test.npy"]

note I only put a couple of types you might encounter in the Switch . See the manual under BinaryRead if you need to add others. Also I did not implement the 'fortran_order' key , just assume the default false.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.