A list of {Integer32, Real32} items in Mathematica seems to use 12-times as much memory than the C equivalent array of structs. And it looks like it can't be stored in a packed array, due to the integer/real mix. Besides splitting the time-series into two lists, one of timestamps and the other of values, so that each one can be packed individually, is there any other strategy?

Also, Mathematica 9 has a new function – TemporalData. Is that more efficient than a simple list?

  • $\begingroup$ Mathematica's AbsoluteTime returns the Unix Time which is per default represented by a Real. Isn't it a possibility to use this? $\endgroup$ – halirutan Oct 15 '13 at 18:04
  • $\begingroup$ @halirutan at least it's almost unix time: unixTime[date___]:=AbsoluteTime[date] - AbsoluteTime[{1970, 1, 1}] $\endgroup$ – ssch Oct 15 '13 at 18:16
  • $\begingroup$ Hmm, the source is UNIX time, but Mathematica seems to convert them to Integer when I use the following function: Epoch[timestamp_] = AbsoluteTime[{1970}] + timestamp $\endgroup$ – Meh Oct 15 '13 at 18:54
  • $\begingroup$ How do you assert that an integer is represented by Integer32 in memory? As far as I can tell, you have no control on that. $\endgroup$ – Hector Oct 16 '13 at 1:23
  • $\begingroup$ @Hector I don't, I just load them using BinaryReadList using {Integer32, Real32}. Even if Mathematica uses 64-bit ints/floats, there is still a 6x overhead over C. $\endgroup$ – Meh Oct 16 '13 at 1:46

I am not sure what seems to be the problem.

Module[{fn = "D:\\Temp\\exp.mx", lt = Transpose@{Range[10], N[Range[10]/3]}, at}, 
 BinaryWrite[fn, lt, {"Integer32", "Real32"}]; Close@fn; 
 Print[at = BinaryReadList[fn, {"Integer32", "Real32"}, 10]]; 
 at - lt]

That file is as packaged as it can be; and, besides rounding errors, the information is recovered.

As for the contents in memory, you do not have control over those. I do not think Mathematica uses 64-bits integers but as intermediate objects. Otherwise, it would not be able to make arbitrary precision arithmetic. For example, you can fit $2^{30}$ in a 32-bit integer and store in disk as such. Once loaded to memory, Mathematica has no problem finding $(2^{30})^5$, which cannot fit in a machine-size integer. The 12x overhead space is the price we pay for the polymorphism that Mathematica gives.

| improve this answer | |
  • 2
    $\begingroup$ I was asking about in memory usage, not on disk. ByteCount[ts] shows 12x overhead $\endgroup$ – Meh Oct 15 '13 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.