4
$\begingroup$

I am trying to pass a matrix of 150*4000 real numbers to R with RLink. If I try to pass the whole matrix the process seems to take forever, without stopping for errors. The interesting part is that if i pass the first 606 columns then the data is passed immediately, but if I add a single column more then it takes forever.

I guess that I am reaching some threshold on data size (which is approx. 2MB with ByteCount[]). Can I overcome this?

Edit: Trying to reproduce the problem on random data I've realized that the 607th column of my data contains some Integers, mixed with the rest of Real numbers. This leads to a simple solution that I'll post below

I am on macosx 10.12.4, using R 3.2.2 and mma 10.2

The code I use to install RLink and pass the data, following Szabolcs' Guide:

SetEnvironment["DYLD_LIBRARY_PATH" -> "/Library/Frameworks/R.framework/Resources/lib"]
Needs["JLink`"]
SetOptions[InstallJava, JVMArguments -> "-Xmx3g"];
Needs["RLink`"]
InstallR["RHomeLocation" -> "/Library/Frameworks/R.framework/Resources/", "RVersion" -> 3]
RSet["data", data]
$\endgroup$
5
  • $\begingroup$ Can you share exact code you use to pass the data? With the timings on your machine? $\endgroup$ Apr 19 '17 at 12:15
  • $\begingroup$ Trying to reproduce the problem on random data I've made some progress. I have updated the question accordingly $\endgroup$
    – CupiDio
    Apr 19 '17 at 13:13
  • 2
    $\begingroup$ Thanks. The reason why mixed representation takes much more time is that it can no longer be represented as a matrix of native type in R (at least, automatically), and then is interpreted as an R list. And R lists are much slower to send to R or get back from R, for a number of reasons. But you are right that RLink could perhaps be improved, may be issue some warnings in such cases, also this should be mentioned in the documentation. I will see if I can make this happen for 11.2 $\endgroup$ Apr 19 '17 at 13:20
  • $\begingroup$ Also, in case if you want for some reason to preserve integers as integers: if you provide a realistic sample of your mixed Integer / Real data, it might be possible to find some better way. Otherwise, I will take is as "problem solved" for now - if that is the case, then perhaps you could post a self-answer from the detail in your edit. $\endgroup$ Apr 19 '17 at 13:24
  • $\begingroup$ For me the problem is solved, I'll post this as a self-answer. Thanks! $\endgroup$
    – CupiDio
    Apr 19 '17 at 13:31
2
$\begingroup$

Apparently trying to send a matrix with mixed types takes extremely longer. I've tried sending 3 different types of matrices, of different sizes, and here are the timings:

(*All Real Numbers*)
3*3: 0.0059s
11*21: 0.0071s

(*All Integer Numbers*)
3*3: 0.0080s
11*21: 0.0065s

(*Mixed Reals and Integers*)
3*3: 0.0697s
11*21: 1.5256s

So passing mixed types not only takes much more but seems to scale very differently with size. The problem was that integers appear for the first time in the 607th column, but at that point the matrix is so big that it should take a very long time to pass it (provided it really goes trough in the end.)

The problem is easily solved by converting everything into float numbers with N[] in this particular case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.