5
$\begingroup$

I have a data file contains several lines of numbers that separated by "," and I cannot load it correctly with ReadList and Import.

Here is a example:

with data contains "," and "\n", simple ReadList doesn't work.

ReadList[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], Real]
(* {1099.5, $Failed} *)

If we use separators, it works, but if we group the types, it doesn't work.

ReadList[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], Real,
  RecordSeparators -> {",", "\n"}]
(* {1099.5, 0., -166.79, 1098.4, 0., -166.79} *)

ReadList[StringToStream[
  "1099.5,0,-166.79\n1098.4,0,-166.79\n"], {Real, Real, Real}, 
 RecordSeparators -> {",", "\n"}]
(* {$Failed} *)

And Import cannot import correctly as it treats each line as a string.

Import[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], "Table"]
StringQ /@ 
 Flatten@Import[
   StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], "Table"]
(*
  {{"1099.5,0,-166.79"}, {"1098.4,0,-166.79"}}
  {True, True}

*)

So why they are not working in these cases? And How to make ReadList and Import to work without using Partition etc. after the import?

Version: 9.0 on Mac.

$\endgroup$
  • $\begingroup$ Could you give the reason why this question should be closed? I think this is a practical problem when I have a large data set to import. $\endgroup$ – xslittlegrass Nov 19 '13 at 0:45
  • $\begingroup$ If I had to guess, it would be because you have comma separated values (CSV) and you're not using the direct option — "CSV" to do that. ReadList needs to have Record instead of Real. I think the latter is not very well known, but your question's main focus seems to be on performance with large files. You should focus on that instead. In any case, that would probably make this a dupe of the post that RunnyKine linked to... $\endgroup$ – rm -rf Nov 19 '13 at 1:46
5
$\begingroup$

Have you tried using "CSV" instead of "Table". Seems to work for me.

Import[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], "CSV"]
{{1099.5, 0, -166.79}, {1098.4, 0, -166.79}}

Update

The problem with ReadList turned out to rather hard to track down. It seems that only white space can be used to delimited fields within records when ReadList is used. That is, the following will work.

ReadList[StringToStream["1099.5\t0\t-166.79\n1098.4\t0\t-166.79\n"],
  {Number, Number, Number}]

{{1099.5, 0, -166.79}, {1098.4, 0, -166.79}}

and so will numerous variations of which I give a few examples.

ReadList[StringToStream["1099.5 0 -166.79\n1098.4 0 -166.79\n"],
  {Number, Number, Number}]

ReadList[StringToStream["1099.5 0 -166.79\n1098.4 0 -166.79\n"], 
  Number, RecordLists -> True]

ReadList[StringToStream["1099.5\t0\t-166.79\n1098.4\t0\t-166.79\n"], 
  Number, RecordLists -> True]

Putting "," into the RecordSeparators list doesn't help because that makes every separate number a record, but you want three numbers in a record. Unfortunately, ReadList doesn't support a FieldSeparator option, which is what you really need,

$\endgroup$
  • 1
    $\begingroup$ Thanks. Do you know why ReadList doesn't work? I have a large data set and Import is quiet slow. $\endgroup$ – xslittlegrass Nov 19 '13 at 0:49
  • $\begingroup$ FieldSeparator ! That's the option we need! I really don't understand why after so many years, there is still no FieldSeparator to make Read file as common as csv format convenient and efficient. So, at the moment, the only efficient way is still to read csv file is read it as record, and then convert to number? $\endgroup$ – matheorem Mar 10 at 2:20
4
$\begingroup$

Try this:

ReadList[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], 
  Table[Record, {3}], RecordSeparators -> {",", "\n"}] // ToExpression

{{1099.5, 0, -166.79}, {1098.4, 0, -166.79}}

$\endgroup$
  • $\begingroup$ Thanks, this work, but ToExpression seem drains the performance on large data files. $\endgroup$ – xslittlegrass Nov 19 '13 at 0:48
  • $\begingroup$ @xslittlegrass No matter what the performance hit is (And I don't think it's much), it will still be much faster than Import. In terms of ReadList this is the best you can do. Also if you want to deal with large CSV files look here for my answer: mathematica.stackexchange.com/questions/35371/… $\endgroup$ – RunnyKine Nov 19 '13 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.