Is there an efficient way to load a large list to Mathematica from R within Rlink?

Question

I'm working on something that involves using both R and Mathematica. For this something, I first generate a large R list (1000 matrices, each 200x1000, mixed integer and real values) and then need to load it in Mathematica.

Exporting to text files and importing in Mathematica is not an option (a plain .txt would be about 1.6Gb in size and too time consuming to be written and read). So I tried Rlink. What I do is basically this:

1. output a .RData file containing the list (roughly 55Mb, vs the 1.6Gb) from R with the save() command
2. fire up Rlink in Mathematica and load() the .RData file, so that it's accessible through REvaluate[].

What I would like to do is to define a Mathematica object to work with directly, but doing something like

tmp = REvaluate["nameofthelist"]

leads to a memory allocation error within the Rlink R session itself. If I try with smaller subsets of the list (say 5 200x1000 matrices at once) it works, but takes forever to complete.

Any suggestion?

Here's a sample code to recreate a smaller version of the problem, set list length (here 10) to your liking/computational possibilities:

Needs["RLink`"]
InstallR[]
REvaluate["tmplist <- vector('list',10)"]
REvaluate["for (i in 1:10) tmplist[[i]] <- matrix(rnorm(200*1000),200,1000)"]

I was wondering if creating an object and then filling it by means of a loop would work (extract single columns/matrices from the list is not so computationally heavy using REvaluate[]), but I'm no expert of Mathematica, so I'm looking for advice..

Thanks in advance!

Answer 1

The following works for me in reasonable time, when I have about 3Gb free of RAM:

Needs["RLink`"]
InstallR[]

Create the large list in R:

REvaluate["tmplist <- vector('list',1000);"];
REvaluate[
 "for (i in 1:1000) tmplist[[i]] <- matrix(rnorm(200*1000),200,1000)"
]; // AbsoluteTiming

(* {15.8774, Null} *)

Test the memory use on the R side:

REvaluate["object.size(tmplist)"]

(* RObject[{1.60021*10^9}, RAttributes["class" :> {"object_size"}]] *)

Define a function (closure) to extract the i-th part and transfer it to Mathematica:

ithPart = REvaluate["function(i){tmplist[[i]]}"]

(*  
    RFunction["closure", RCode["function (i) 
      {
        tmplist[[i]]
      }"], 1, RAttributes[]
    ]
*)

Construct the full array in Mathematica:

(temp = Array[ithPart, {1000}]); // AbsoluteTiming

(* {40.4831, Null} *)

It takes about 40 seconds on my machine, which is not that unreasonable to transfer 1.6 Gb of data:

ByteCount[temp]

(* 1600160200 *)

Check dimensions:

Dimensions[temp]

(* {1000, 200, 1000} *)

Clean up everything (have to call R's garbage collection gc as well):

REvaluate["rm(tmplist)"]
REvaluate["gc()"];
Remove[temp]

You should ovserve memory having been fully released at both Mathematica and R side after this step. However, during the transfer, this scheme requires roughly twice the memory needed to store the data, which is about 3.2 Gb in this case.