I work with big data on AWS and scientific data in all the standard formats such as these...

enter image description here

Is there any way to Import/Export Datasets or Associations to any of these non-primitive (txt, csv, json...) like HDF or SQL?

  • $\begingroup$ I think there is currently no way to import to Dataset and Association directly in MMA. Also looking at your comments in the answer below - Dataset is not aware of foreign keys in external sources. Dataset is an object for allowing database-like manipulation of data within Mathematica and not particularly optimised for importing existing database structures into MMA. Simply put MMA will only behave with the data in the way you tell it. SQL queries will need the joins already present, 2 tables imported separately and then converted to Dataset will need JoinAcross etc etc. $\endgroup$ – Gordon Coale Feb 9 '15 at 9:26
  • $\begingroup$ If you are familiar with SQL I recommend SQLExecute rather than the other MMA SQL commands - then you can use the full range of SQL native to your database. $\endgroup$ – Gordon Coale Feb 9 '15 at 9:28
  • $\begingroup$ Is this any help? mathematica.stackexchange.com/questions/3965/… $\endgroup$ – Gordon Coale Feb 9 '15 at 9:37
  • $\begingroup$ Have you tried SemanticImport? $\endgroup$ – Stefan R Feb 11 '15 at 21:14
  • $\begingroup$ @StefanR, here's a assessment of SemanticImport and related functions on dates extracted from contracts and other documents: mathematica.stackexchange.com/questions/79241/… $\endgroup$ – alancalvitti May 27 '15 at 17:27

Running $ExportFormats in a Mathematica notebook gives:

{"3DS", "ACO", "AIFF", "AU", "AVI", "Base64", "Binary", "Bit", "BMP", \ "Byte", "BYU", "BZIP2", "C", "CDF", "Character16", "Character8", \ "Complex128", "Complex256", "Complex64", "CSV", "CUR", "DICOM", \ "DIF", "DIMACS", "DOT", "DXF", "EMF", "EPS", "ExpressionML", "FASTA", \ "FASTQ", "FCS", "FITS", "FLAC", "FLV", "GIF", "Graph6", "Graphlet", \ "GraphML", "GXL", "GZIP", "HarwellBoeing", "HDF", "HDF5", "HTML", \ "HTMLFragment", "ICNS", "ICO", "Integer128", "Integer16", \ "Integer24", "Integer32", "Integer64", "Integer8", "JPEG", \ "JPEG2000", "JSON", "JVX", "KML", "LEDA", "List", "LWO", "MAT", \ "MathML", "Maya", "MGF", "MIDI", "MOL", "MOL2", "MP3", "MTX", "MX", \ "NASACDF", "NB", "NetCDF", "NEXUS", "NOFF", "OBJ", "OFF", "OGG", \ "Package", "Pajek", "PBM", "PCX", "PDB", "PDF", "PGM", "PICT", "PLY", \ "PNG", "PNM", "POV", "PPM", "PXR", "QuickTime", "RawBitmap", \ "Real128", "Real32", "Real64", "RIB", "RTF", "SCT", "SDF", "SND", \ "Sparse6", "STL", "String", "SurferGrid", "SVG", "SWF", "Table", \ "TAR", "TerminatedString", "TeX", "TeXFragment", "Text", "TGA", \ "TGF", "TIFF", "TSV", "UnsignedInteger128", "UnsignedInteger16", \ "UnsignedInteger24", "UnsignedInteger32", "UnsignedInteger64", \ "UnsignedInteger8", "UUE", "VideoFrames", "VRML", "VTK", "WAV", \ "Wave64", "WDX", "WebP", "X3D", "XBM", "XHTML", "XHTMLMathML", "XLS", \ "XLSX", "XML", "XYZ", "ZIP", "ZPR"}

So yes for HDF, NetCDF and FITS. No, for Silo.

  • $\begingroup$ That was the first thing I did. But it's not clear to me at all how this integrates with Dataset... For instance when you Import SQLTables with DatabaseLink into a Dataset, e.g. how exactly are foreign key relations handled? Maybe I missing some tutorial somewhere in the documentation, but I could really use some guidance on importing nontrival datasets. $\endgroup$ – M.R. Feb 9 '15 at 0:30
  • 2
    $\begingroup$ I don't know anything about SQLTables and I've never worked with those file types before. I guess the best thing you can do would be to make some simple sample data sets in those formats (HDF, NetCDF and FITS), in Mathematica to export or outside and imported into Mathematica. See if anything gets screwed up by exporting or importing. That's the best I can offer. Why do you need to bring the data into Mathematica? $\endgroup$ – wgwz Feb 9 '15 at 2:22

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.