10
$\begingroup$

I'm working on something that involves using both R and Mathematica. For this something, I first generate a large R list (1000 matrices, each 200x1000, mixed integer and real values) and then need to load it in Mathematica.

Exporting to text files and importing in Mathematica is not an option (a plain .txt would be about 1.6Gb in size and too time consuming to be written and read). So I tried Rlink. What I do is basically this:

  1. output a .RData file containing the list (roughly 55Mb, vs the 1.6Gb) from R with the save() command
  2. fire up Rlink in Mathematica and load() the .RData file, so that it's accessible through REvaluate[].

What I would like to do is to define a Mathematica object to work with directly, but doing something like

tmp = REvaluate["nameofthelist"]

leads to a memory allocation error within the Rlink R session itself. If I try with smaller subsets of the list (say 5 200x1000 matrices at once) it works, but takes forever to complete.

Any suggestion?

Here's a sample code to recreate a smaller version of the problem, set list length (here 10) to your liking/computational possibilities:

Needs["RLink`"]
InstallR[]
REvaluate["tmplist <- vector('list',10)"]
REvaluate["for (i in 1:10) tmplist[[i]] <- matrix(rnorm(200*1000),200,1000)"]

I was wondering if creating an object and then filling it by means of a loop would work (extract single columns/matrices from the list is not so computationally heavy using REvaluate[]), but I'm no expert of Mathematica, so I'm looking for advice..

Thanks in advance!

$\endgroup$
7
$\begingroup$

The following works for me in reasonable time, when I have about 3Gb free of RAM:

Needs["RLink`"]
InstallR[]

Create the large list in R:

REvaluate["tmplist <- vector('list',1000);"];
REvaluate[
 "for (i in 1:1000) tmplist[[i]] <- matrix(rnorm(200*1000),200,1000)"
]; // AbsoluteTiming

(* {15.8774, Null} *)

Test the memory use on the R side:

REvaluate["object.size(tmplist)"]

(* RObject[{1.60021*10^9}, RAttributes["class" :> {"object_size"}]] *)

Define a function (closure) to extract the i-th part and transfer it to Mathematica:

ithPart = REvaluate["function(i){tmplist[[i]]}"]

(*  
    RFunction["closure", RCode["function (i) 
      {
        tmplist[[i]]
      }"], 1, RAttributes[]
    ]
*)

Construct the full array in Mathematica:

(temp = Array[ithPart, {1000}]); // AbsoluteTiming

(* {40.4831, Null} *)

It takes about 40 seconds on my machine, which is not that unreasonable to transfer 1.6 Gb of data:

ByteCount[temp]

(* 1600160200 *)

Check dimensions:

Dimensions[temp]

(* {1000, 200, 1000} *)

Clean up everything (have to call R's garbage collection gc as well):

REvaluate["rm(tmplist)"]
REvaluate["gc()"];
Remove[temp]

You should ovserve memory having been fully released at both Mathematica and R side after this step. However, during the transfer, this scheme requires roughly twice the memory needed to store the data, which is about 3.2 Gb in this case.

$\endgroup$
  • $\begingroup$ Dear Leonid, thanks so much for your answer. I got by myself as far as to define an 'ithPart' function, but didn't know about the Array[] construct in Mathematica, so I couldn't solve a thing. This should work, I just have to figure out why it still takes ~10 secs to ithPart a single matrix! $\endgroup$ – Federico Andreis Nov 11 '15 at 12:31
  • $\begingroup$ @FedericoAndreis Glad it helped. Re: single matrix - if it is a mix of integer and double types, then the transfer may be slower or much slower. Is each single matrix of the same type (integers or doubles), or is it a mix? Also, do you have anough RAM? If you are running short of RAM, it can slow down the transfer. As you could see, in my example, single matrix was transferred in about 0.04 seconds. Can you just run exact same code as I posted, and tell if the timings are similar or much different in your case? $\endgroup$ – Leonid Shifrin Nov 11 '15 at 12:37
  • $\begingroup$ I was doing just what you suggested. It takes more than 70 seconds just to create the example list tmplist.. as for what concerns the creation of temp, the Java platform SE binary behind works occupies ~2.2Gb of RAM that, incidentally, is ~2% of the total RAM... total time to create temp is slightly less than 200 seconds. Which I guess is ok for what I need, but still far from your result! I wonder... $\endgroup$ – Federico Andreis Nov 11 '15 at 12:50
  • $\begingroup$ @FedericoAndreis What kind of computer you have, and on what platform? It sounds like your computer performs about 5x slower than mine, and I have a 3 yrear old MacBook Pro 2.9 GHz Core i7 (2 cores) with 8Gb RAM, and I was running this code on Mathematica 10.2. So it is not clear for me why the code runs so much slower in your case, since my computer is pretty average. $\endgroup$ – Leonid Shifrin Nov 11 '15 at 13:09
  • $\begingroup$ Mathematica 10.2.. AMD Opteron, 128Gb RAM (48 cores though I'm running on one for this I guess).. It's a server, there are other users connected but no more than 25% of the total RAM is being used atm.. Possibly just a bad day for the cpu temperature :D $\endgroup$ – Federico Andreis Nov 11 '15 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.