28
$\begingroup$

I would like to import the World Value Survey data file into Mathematica, but it's only given in spss, sas or stata formats, which are statistical analysis softwares format I think.

I can't get to import these files in Mathematica, they are not recognized.

I've found a Wolfram executable that should allow me to import SAS files but it can't make it work on Linux. Have you already been able to import such files in Mathematica?

$\endgroup$
12
  • 1
    $\begingroup$ @MarcoB As in the original question, the idea was to get the data from World Value Survey, in any of the forms available in the Longitudinal Data Files, i.e R, SAS, SATA, SPSS, but I would not discard retrieving particular variables directly from the web. $\endgroup$
    – rhermans
    Commented May 26, 2015 at 7:19
  • 2
    $\begingroup$ Its frankly bizzare that the data dictionary is available in excel but the actual data isn't - at least it would be more universal than any of the other formats.. Have you tried mailing them and suggesting supporting at least one non-proprietary format should be strongly considered? $\endgroup$ Commented May 26, 2015 at 7:53
  • 1
    $\begingroup$ Looks like the quickest way is to download R and use this code here to export to CSV. Since you only need to do it as a once off there is very little point in messing around with Rlink. $\endgroup$ Commented May 26, 2015 at 7:57
  • 1
    $\begingroup$ @GordonCoale R is bundled with Mathematica and can be called with the RLink package. $\endgroup$ Commented May 27, 2015 at 11:12
  • 1
    $\begingroup$ @rhermans - working on it but never used Rlink before ;) $\endgroup$ Commented May 29, 2015 at 11:28

3 Answers 3

14
+50
$\begingroup$

This should do the trick - disclaimer the file is 1.4Gb so everything takes a veerryyy long time on my MacBook air, and you will need an active internet connection for InstallR to work. Note you have to escape \ internal R file paths and anything else that uses a double quote.

Firstly download the rdata version of the file from the link given.

Needs["RLink`"];

InstallR[];

REvaluate[
"load(\"~/Downloads/WVS_Longitudinal_1981_2014_R_v2015_04_18.rdata\"\)"];

(*{".Traceback","WVS_Longitudinal_1981_2014_R_v2015_04_18"}*)

Writes the data into an R Dataset called WVS_Longitudinal_1981_2014_R_v2015_04_18

REvaluate["write.csv(WVS_Longitudinal_1981_2014_R_v2015_04_18,file=\
"WVS_data_rlink.csv\")"];

Writes the data back out to a CSV. Note in this case the file is a simple tabular one. This might be more problematic for some of the more complex data structures that .rdata format permits.

Theoreticall you can actually import the WVS_Longitudinal_1981_2014_R_v2015_04_18 data directly into MMA and work with it natively from that point but requires me working through the rLink tutorial a bit more than I have time for right now. :)

EDIT : Some useful tips to explore the data directly.

Get Column Names

REvaluate["names(WVS_Longitudinal_1981_2014_R_v2015_04_18)"] // Short

Get a single column in this case S001

REvaluate["WVS_Longitudinal_1981_2014_R_v2015_04_18$S001"]

Get the first 10 lines of the R data.frame note the R data objects in there. head is like the unix head not Head[] This data is so wide and deep its hard to get it in a readable format without subsetting and slicing and dicing.

REvaluate["head(WVS_Longitudinal_1981_2014_R_v2015_04_18,n=10)"]
$\endgroup$
10
$\begingroup$

You are right - those formats are for statistical software packages.
I've noticed in Big Data discussions that Mathematica does not want to extend the range of acceptable data formats leaving the job to third-party software. I would recommend to use software like 'Stat/Transfer', which is a usual practice for such tasks. If you take data from WVS, say Stata's .dta which I prefer, you may easily transfer it to .CSV or .DAT file.
Just take an option ASCII - Delimited as shown below. enter image description here
Then .csv is imported to Mathematica as usual.
Another option is to use ASCII Fixed Format + All Programs which will produce .dat file also easily imported to Mathematica (I've checked both cases).

$\endgroup$
2
  • 2
    $\begingroup$ Thanks for the answer, 'Stat/Transfer' unfortunately its a paid software. $\endgroup$
    – rhermans
    Commented May 26, 2015 at 7:24
  • 1
    $\begingroup$ I'm not aware about free analogs, indeed - it is very specific stuff and works fine. $\endgroup$
    – garej
    Commented May 26, 2015 at 14:47
7
$\begingroup$

With the ability to connect to Python I found this snippet of code helpful. It imports a STATA .dta file using pandas and then outputs it as a Wolfram Dataset.

     importDTAMP[fileName_] := Module[{}, 
 python37 = 
       StartExternalSession[<|"System" -> "Python", "Version" -> "3.7.3"|>];
      pythonFileName = StringReplace[fileName, "\\" -> "/"];
      
      ds = ExternalEvaluate[
        python37,
        "
         import pandas as pd;
         path = '" <> pythonFileName <> "';
         df = pd.read_stata(path);
         df
         "
        ];
      
      DeleteObject[python37];
      ds
      
      ]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.