1
$\begingroup$

I couldn't find the answer to my question, so here it is:

I have quite a large table, which I generate in Matlab (for some reasons i use both Matlab and Mathematica, I'd like to use only one of them, but plots look better in Mathematica and programming is more clear for me in Matlab). Matlab says that data I have is "121x121x181 double", I save it as *mat file - its size is only 1.4 KB. Then I import it in Mathematica and flatten, because of some wierd extra dimension:

data1 = Flatten[Import["file.mat"],1];
ByteCount[data1];

21200328

So, its over 20 MB already. After that I transform the table a little bit further, to make it appropriate for interpolation:

m = Max[data1];
data2 = Flatten[
Table[{{i1, i2, i3}, data1[[i3, i2, i1]]/m}, 
{i1, 1, 181}, {i2, 1, 121}, {i3, 1, 121}], 
2];
ByteCount[data2]

508804112

Well, it is more than 500MB. Now things are getting slow. Not to say, I can't handle bigger tables. So, can anybody explain, how to work with such big tables safely? Because I really don't need this infinite precision, I believe Mathematica uses each time I do anything.

Improve this question
$\endgroup$
6
  • $\begingroup$ Have you seen the Arbitrary-Precision Numbers tutorial? In particular $MaxPrecision. $\endgroup$
    – Edmund
    Commented Jun 20, 2016 at 20:12
  • $\begingroup$ The best way to minimize size is to keep your data as a regular array, i.e. a tensor. That way you can use packed arrays. $\endgroup$
    – Greg Hurst
    Commented Jun 20, 2016 at 20:16
  • $\begingroup$ As you haven't explained what you are trying to do with the result, it is difficult to explain how you can achieve it without creating the array that is causing the problem. But functions such as MapIndexed may allow you to apply a function to array values without explicitly tagging each with its index. $\endgroup$
    – mikado
    Commented Jun 20, 2016 at 21:14
  • $\begingroup$ @mikado This table contains distrbution of some physical property in 3d-space. I'd like to be able to plot it in arbitrary planes or integrate over some areas. To do so I use Interpolation[], because it is very natural in Mathematica to define a special function for a given distribution. But, for some reason, when I interpolate data with more than 2 dimensions, it requires indices, so I add them. It's ok, but I just don't get why it becomes so huge after that. $\endgroup$
    – vasya
    Commented Jun 21, 2016 at 14:32
  • $\begingroup$ @Chip Thanks, I'll check it out. As I mentioned, I format the table for Interpolation[]. But if there's nothing I can do with its huge size, I'll look for another way to solve my problem. $\endgroup$
    – vasya
    Commented Jun 21, 2016 at 14:36

1 Answer 1

Reset to default
1
$\begingroup$

I generate some test data.

SeedRandom[42];
data1 = RandomReal[100., {5, 4, 3}];
ByteCount  @ data1

600

Then I calculate what I think you want.

m = Max @ data1

99.6966

Scale and reorder the data.

data2 = Transpose[data1/m, {3, 2, 1}]

data2

ByteCount @ data2

600

I think data2 would all you need to make further computations in Mathematica, but if you insist on attaching the indices, then the following will do it.

data3 = Flatten[MapIndexed[{#2, #1} &, data2, {-1}], 2]

data3

ByteCount @ data3

12576

Now your way of doing it.

data3 = 
  Flatten[
    Table[{{i1, i2, i3}, data1[[i3, i2, i1]]/m}, {i1, 1, 3}, {i2, 1, 4}, {i3, 1, 5}],
    2]
data3 == data4

True

Naturally data3 and data4 are larger than data2, They contain the indices and have a much more complex list structure to accommodate them.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks. I tried your way to generate "data3" - it actually takes a little bit less (6s/7.7s) time, but the size of the result is bigger (551MB/508Mb). Anyway, they both are quite big. This data represents some physical property in 3d-space, so I'd like to work with this spatial distribution, especially to be able to plot distributions in arbitrary planes and integrate over arbitrary areas. To do so, I use Interpolation[]. For some reason, if I try to interpolate data with more than two dimensions - attaching of indices is required. But I stiil don't understand why data becomes so big. $\endgroup$
    – vasya
    Commented Jun 21, 2016 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.