4
$\begingroup$

A list of {Integer32, Real32} items in Mathematica seems to use 12-times as much memory than the C equivalent array of structs. And it looks like it can't be stored in a packed array, due to the integer/real mix. Besides splitting the time-series into two lists, one of timestamps and the other of values, so that each one can be packed individually, is there any other strategy?

Also, Mathematica 9 has a new function – TemporalData. Is that more efficient than a simple list?

$\endgroup$
5
  • $\begingroup$ Mathematica's AbsoluteTime returns the Unix Time which is per default represented by a Real. Isn't it a possibility to use this? $\endgroup$
    – halirutan
    Commented Oct 15, 2013 at 18:04
  • $\begingroup$ @halirutan at least it's almost unix time: unixTime[date___]:=AbsoluteTime[date] - AbsoluteTime[{1970, 1, 1}] $\endgroup$
    – ssch
    Commented Oct 15, 2013 at 18:16
  • $\begingroup$ Hmm, the source is UNIX time, but Mathematica seems to convert them to Integer when I use the following function: Epoch[timestamp_] = AbsoluteTime[{1970}] + timestamp $\endgroup$
    – Meh
    Commented Oct 15, 2013 at 18:54
  • $\begingroup$ How do you assert that an integer is represented by Integer32 in memory? As far as I can tell, you have no control on that. $\endgroup$
    – Hector
    Commented Oct 16, 2013 at 1:23
  • $\begingroup$ @Hector I don't, I just load them using BinaryReadList using {Integer32, Real32}. Even if Mathematica uses 64-bit ints/floats, there is still a 6x overhead over C. $\endgroup$
    – Meh
    Commented Oct 16, 2013 at 1:46

1 Answer 1

1
$\begingroup$

I am not sure what seems to be the problem.

Module[{fn = "D:\\Temp\\exp.mx", lt = Transpose@{Range[10], N[Range[10]/3]}, at}, 
 BinaryWrite[fn, lt, {"Integer32", "Real32"}]; Close@fn; 
 Print[FileByteCount@fn]; 
 Print[at = BinaryReadList[fn, {"Integer32", "Real32"}, 10]]; 
 Print[lt];
 at - lt]

That file is as packaged as it can be; and, besides rounding errors, the information is recovered.

As for the contents in memory, you do not have control over those. I do not think Mathematica uses 64-bits integers but as intermediate objects. Otherwise, it would not be able to make arbitrary precision arithmetic. For example, you can fit $2^{30}$ in a 32-bit integer and store in disk as such. Once loaded to memory, Mathematica has no problem finding $(2^{30})^5$, which cannot fit in a machine-size integer. The 12x overhead space is the price we pay for the polymorphism that Mathematica gives.

$\endgroup$
1
  • 2
    $\begingroup$ I was asking about in memory usage, not on disk. ByteCount[ts] shows 12x overhead $\endgroup$
    – Meh
    Commented Oct 15, 2013 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.