9
$\begingroup$

Bug introduced in 11.2.0

TL;DR

  • Issue 1 (bug): Import of CSV files with column specification fails when "HeaderLines" or "SkipLines" option is used:

    Import["ExampleData/financialtimeseries.csv", {"Data", All, 2}, "HeaderLines" -> 1]

    screesnhot

  • Issue 2 (explained in Itai's answer below): Import of version 11.2 incorrectly imports some CSV files created with previous versions. An example:

    (* evaluate in version 11.1)
Export["test.csv", {{11, 12, 13},
  {21, Missing["Uncertain", {"giant", "subgiant"}], 23}}, "CSV"];

    In the output file second line is written as
    21,Missing["Uncertain", {"giant", "subgiant"}],23
    Now an attempt to import such file with version 11.2 produces incorrect result:

    (* evaluate in version 11.2.0)
Import@"test.csv"

    {{11, 12, 13}, {21, "Missing[\"Uncertain\"", "{\"giant\"", "subgiant\"}],23"}}

    (the Missing statement is split into three parts).

Original question

Created additional posting at: Wolfram Community Posting

In Mathematica 11.1 I've created many CSV files from StarData:

sunMass = StarData["Sun", "Mass"];
sunLuminosity = StarData["Sun", "Luminosity"];
sunTemperature = StarData["Sun", "EffectiveTemperature"];
sunGravity = StarData["Sun", "Gravity"];
sunDensity = StarData["Sun", "Density"];
sunVolume = StarData["Sun", "Volume"];
sunDiameter = StarData["Sun", "Diameter"];

SetDirectory[$UserDocumentsDirectory]

listData1 = Take[StarData[EntityClass["Star", All]], {1, 10000}];

CloseKernels[]; LaunchKernels[4]
AbsoluteTiming[
 Length[
  data =
   Transpose[
    ParallelMap[
     StarData[listData1, #] &,
     {"Name", "Metallicity", "SpectralClass", "BVColorIndex", 
      "EffectiveTemperature",
      "Mass", "Luminosity", "AbsoluteMagnitude", "Gravity", "Density",
       "Diameter",
      "DistanceFromEarth", "MainSequenceLifetime", "Parallax",
      "RadialVelocity", "Radius", "StarEndState", "StarType", 
      "SurfaceArea",
      "VariablePeriod", "Volume", "HDName"}]]]]



zeroData = data /. {Missing["NotAvailable"] -> 0}; 

noUnitsData = 
  zeroData /. {c1_, c2_, c3_, c4_, c5_, c6_, c7_, c8_, c9_, c10_, 
     c11_, c12_, c13_, c14_, c15_, c16_, c17_, c18_, c19_, c20_, c21_,
      c22_} -> {c1, c2, c3, c4, QuantityMagnitude[c5], 
     QuantityMagnitude[c6/sunMass], 
     QuantityMagnitude[c7/sunLuminosity], c8, 
     QuantityMagnitude[c9/sunGravity], QuantityMagnitude[c10], 
     QuantityMagnitude[c11/sunDiameter], QuantityMagnitude[c12], 
     QuantityMagnitude[c13], QuantityMagnitude[c14]
     , QuantityMagnitude[c15], QuantityMagnitude[c16], c17, c18, 
     QuantityMagnitude[c19], QuantityMagnitude[c20], 
     QuantityMagnitude[c21/sunVolume], c22};

Length[noUnitsData]

prePendData = 
  Prepend[noUnitsData, {"Name", "Metallicity", "SpectralClass", 
    "BVColorIndex", "EffectiveTemperature",
    "Mass", "Luminosity", "AbsoluteMagnitude", "Gravity", "Density", 
    "Diameter",
    "DistanceFromEarth", "MainSequenceLifetime", "Parallax",
    "RadialVelocity", "Radius", "StarEndState", "StarType", 
    "SurfaceArea",
    "VariablePeriod", "Volume", "HDName"}];

TableForm[Take[prePendData, 5]]

Export["allStars1.csv", prePendData, "CSV"]

This creates a CSV file with 10,000 rows of comma separated data. Works great for all 108,939 rows of StarData, by creating 11 CSV files.

Importing each CSV file in a new notebook is pretty straight forward

    Length[data1 = 
  Import["allStarData1.csv", {"Data", {All}}, 
    "HeaderLines" -> 1] /. {c1_, c2_, c3_, c4_, c5_, c6_, c7_, c8_, 
     c9_, c10_, c11_, c12_, c13_, c14_, c15_, c16_, c17_, c18_, c19_, 
     c20_, c21_, c22_} -> {c1, c2, c3, c4, c5, c6, c7, c8, c9, c10, 
     c11, c12, c13, c14
     , c15, c16, c17, c18, c19, c20, c21, c22}
 ]

This worked fine across different releases of Mathematica 11 for a large number of CSV files.

Upgraded today to Mathematica 11.2.

None of the existing notebooks work.

Import statements now take forever, generating error messages, and truncating large numbers of rows from existing CSV files.

One workaround that I'm currently testing, is running the StarData extract code listed above under Mathematica 11.2 creating new CSV files, and then importing the new CSV files.

This worked for the first 10,000 row StarData extract. No errors, no truncation. But still runs very slow. Will have to run the other 10 extracts and create new 10,000 row CSV files for each.

Feels like this is a bug in Mathematica 11.2 Import statement internal code. Where new internal data verification checks are incompatible to previously created CSV files.

Anyone else run into this issue?

Also I turned off the error messages to try to get through existing code, but don't know how to turn the error messages back on, so that I can include them in this post. Anyone know how?

Thanks

Including JPEG of Import errors:

Import errors, where each file has a length of 10,000 records

Mathematica 11.2 documentation points to updates in CSV Import & Export functions:

enter image description here

Same files created as Export under Mathematica 11.2 and Import under Mathematica 11.2 are not truncated, and have full 10,000 records per file.

enter image description here

"TextDelimiters"->"" fixed the problem. Eliminated row truncation. Import is still horribly slow under Mathematica 11.2 for a 10,000 row CSV file, taking over 250 seconds. Thank you. Workaround I tested was to create all files under Mathematica 11.2 Export and then Import. No Truncation. The Part::partw warning messages are new under 11.2. They did not show up under Mathematica 11.1. Turning them off via Off[Part::part] did not improve the performance elapsed time of the Import.

enter image description here

Falling back to Mathematica 11.1. Importing CSV files under Mathematica 11.2 is 100 times slower.

enter image description here

Improve this question
$\endgroup$
2

3 Answers 3

Reset to default
12
$\begingroup$

The CSV importer was completely rewritten in version 11.2 in order to improve performance and fix a number of long-standing issues. For example, if you imported your csv files back in version 11.1 and din't know to specify a CharacterEncoding option, you would find that all special characters were corrupted.

Also, if you tried to open your 11.1-generated files in a spreadsheet (I used libreoffice), they would only open with 34 rows and lose some data. This is because earlier versions did not quote double quotes inside individual cells, so Missing["Uncertain"] would appear malformed. I don't think that behavior was "wrong" b/c CSV is a somewhat underspecified format, but it certainly isn't the currently dominant standard. I don't recall all the details beacuse I was only very peripherally involved, but this is certainly something which was considered carefully.

Version 11.2 properly quotes double quotes and now follows the standard of fairly agressively using quotes in any "text" field. You can get it to import old CSV files by manually adding the option "TextDelimiters"->"". Files generated in 11.2 should import fine into older versions, but will likely have doubled quotes inside of text strings due to the new quoting behavior. (Meaning, in particular, ToExpression won't work unless you eliminate the doubled double-quotes first.)

Improve this answer
$\endgroup$
7
  • $\begingroup$ Is there quicker fix than going through all Export[*.csv] commands in every script? (I've been relying on quote-free csv's exported from Mathematica for years as input elsewhere.) I tried adding this to my initialization SetOptions[Export,"TextDelimiters"->""] but it doesn't recognize Export options. $\endgroup$
    – kukako-muka
    Sep 17, 2017 at 11:03
  • 7
    $\begingroup$ Because people tend to speak up only when something is wrong, let me say that personally I am very happy with the improvement to Table/CSV import in 11.2. $\endgroup$
    – Szabolcs
    Sep 17, 2017 at 12:28
  • 1
    $\begingroup$ "TextDelimiters"->"" fixed the problem. Eliminated row truncation. Import is still horribly slow under Mathematica 11.2 for a 10,000 row CSV file, taking over 250 seconds. Thank you. Workaround I tested was to create all files under Mathematica 11.2 Export and then Import. No Truncation. The Part::partw warning messages are new under 11.2. They did not show up under Mathematica 11.1. Turning them off via Off[Part::part] did not improve the performance elapsed time of the Import. $\endgroup$
    – Joseph Karpinski
    Sep 17, 2017 at 14:54
  • $\begingroup$ Falling back to Mathematica 11.1. Importing CSV files under Mathematica 11.2 is 100 times slower. Makes all my programs unusable. $\endgroup$
    – Joseph Karpinski
    Sep 17, 2017 at 16:11
  • 1
    $\begingroup$ @AlexeyPopkov look at RFC4180, the CSV spec. 7. If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote. For example: "aaa","b""bb","ccc" So both the ImportString and ExportString examples above are correct. Note: In[450]:= ImportString[ExportString[Missing["Uncertain"], "CSV"], "CSV"] // InputForm Out[450]//InputForm= {{"Missing[\"Uncertain\"]"}} $\endgroup$
    – GenericAccountName
    Sep 19, 2017 at 23:11
1
$\begingroup$

I have experienced similar issues using the new Import function, which are only apparent when using specific column or row specifications, eg. {"Data",All,{1,3}}. The issues seem to relate to the use of

"HeaderLines" -> 1

or 

"SkipLines"-> 1

The options do not work for me when importing directly from a file, versus the online Mathematica 11.2 documentation where the examples are using ImportString. For example from the documentation:

Import["ExampleData/financialtimeseries.csv", "Data","HeaderLines" -> 1]

works. Also the following works:

Import["ExampleData/financialtimeseries.csv", {"Data", All, 2}]

However adding the column specification and "HeaderLines" options together fails:

Import["ExampleData/financialtimeseries.csv", {"Data", All, 2},"HeaderLines" -> 1]
Improve this answer
$\endgroup$
5
  • $\begingroup$ Please report this to the tech support. $\endgroup$
    – Alexey Popkov
    Sep 18, 2017 at 16:30
  • $\begingroup$ I have reported it - will update in case I hear back. $\endgroup$
    – GIM
    Sep 18, 2017 at 19:02
  • 2
    $\begingroup$ Wolfram Response: The problem is reproducible in 11.2 and has been reported to developers so that they can fix it in a future version of Mathematica. $\endgroup$
    – GIM
    Sep 18, 2017 at 23:48
  • $\begingroup$ FYI: the issue is fixed in our nightly build of the next major version. It's too early to tell whether we will have a minor bugfix release prior to the next major version which might also pick benefit from the fix. $\endgroup$
    – Itai Seggev
    Sep 25, 2017 at 3:11
  • $\begingroup$ Great - many thanks for the update! $\endgroup$
    – GIM
    Sep 25, 2017 at 19:47
0
$\begingroup$

Falling back to Mathematica 11.1. Importing CSV files under Mathematica 11.2 is 100 times slower.

enter image description here

Improve this answer
$\endgroup$
3
  • $\begingroup$ Why are you using {"Data",All}? That's essentially the default. And without it, 11.2 is almost 70% faster than 11.1 (0.35 second versus 0.6 seconds on my machine). $\endgroup$
    – Itai Seggev
    Sep 17, 2017 at 19:41
  • $\begingroup$ I've been using that version of the Import statement on CSV datasets for a while, across different versions of Mathematica with no issues, until Mathematica 11.2. There were two issues with the Import statement under that newer release. The first was that it truncated rows of data on an import without any notification, unless you were looking for it with a Length function. The"TextDelimiters"->"" fixed the problem, but it should of never occurred, and will impact other users, without them realizing it. The second issue is the 100 times increase in elapsed time. I've just opened a bug report. $\endgroup$
    – Joseph Karpinski
    Sep 18, 2017 at 0:18
  • $\begingroup$ Thank you for your help!!! $\endgroup$
    – Joseph Karpinski
    Sep 18, 2017 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.