5
$\begingroup$

I have a data file contains several lines of numbers that separated by "," and I cannot load it correctly with ReadList and Import.

Here is a example:

with data contains "," and "\n", simple ReadList doesn't work.

ReadList[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], Real]
(* {1099.5, $Failed} *)

If we use separators, it works, but if we group the types, it doesn't work.

ReadList[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], Real,
  RecordSeparators -> {",", "\n"}]
(* {1099.5, 0., -166.79, 1098.4, 0., -166.79} *)

ReadList[StringToStream[
  "1099.5,0,-166.79\n1098.4,0,-166.79\n"], {Real, Real, Real}, 
 RecordSeparators -> {",", "\n"}]
(* {$Failed} *)

And Import cannot import correctly as it treats each line as a string.

Import[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], "Table"]
StringQ /@ 
 Flatten@Import[
   StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], "Table"]
(*
  {{"1099.5,0,-166.79"}, {"1098.4,0,-166.79"}}
  {True, True}

*)

So why they are not working in these cases? And How to make ReadList and Import to work without using Partition etc. after the import?

Version: 9.0 on Mac.

$\endgroup$
2
  • $\begingroup$ Could you give the reason why this question should be closed? I think this is a practical problem when I have a large data set to import. $\endgroup$ Commented Nov 19, 2013 at 0:45
  • $\begingroup$ If I had to guess, it would be because you have comma separated values (CSV) and you're not using the direct option — "CSV" to do that. ReadList needs to have Record instead of Real. I think the latter is not very well known, but your question's main focus seems to be on performance with large files. You should focus on that instead. In any case, that would probably make this a dupe of the post that RunnyKine linked to... $\endgroup$
    – rm -rf
    Commented Nov 19, 2013 at 1:46

2 Answers 2

5
$\begingroup$

Have you tried using "CSV" instead of "Table". Seems to work for me.

Import[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], "CSV"]
{{1099.5, 0, -166.79}, {1098.4, 0, -166.79}}

Update

The problem with ReadList turned out to rather hard to track down. It seems that only white space can be used to delimited fields within records when ReadList is used. That is, the following will work.

ReadList[StringToStream["1099.5\t0\t-166.79\n1098.4\t0\t-166.79\n"],
  {Number, Number, Number}]

{{1099.5, 0, -166.79}, {1098.4, 0, -166.79}}

and so will numerous variations of which I give a few examples.

ReadList[StringToStream["1099.5 0 -166.79\n1098.4 0 -166.79\n"],
  {Number, Number, Number}]

ReadList[StringToStream["1099.5 0 -166.79\n1098.4 0 -166.79\n"], 
  Number, RecordLists -> True]

ReadList[StringToStream["1099.5\t0\t-166.79\n1098.4\t0\t-166.79\n"], 
  Number, RecordLists -> True]

Putting "," into the RecordSeparators list doesn't help because that makes every separate number a record, but you want three numbers in a record. Unfortunately, ReadList doesn't support a FieldSeparator option, which is what you really need,

$\endgroup$
2
  • 1
    $\begingroup$ Thanks. Do you know why ReadList doesn't work? I have a large data set and Import is quiet slow. $\endgroup$ Commented Nov 19, 2013 at 0:49
  • $\begingroup$ FieldSeparator ! That's the option we need! I really don't understand why after so many years, there is still no FieldSeparator to make Read file as common as csv format convenient and efficient. So, at the moment, the only efficient way is still to read csv file is read it as record, and then convert to number? $\endgroup$
    – matheorem
    Commented Mar 10, 2019 at 2:20
4
$\begingroup$

Try this:

ReadList[StringToStream["1099.5,0,-166.79\n1098.4,0,-166.79\n"], 
  Table[Record, {3}], RecordSeparators -> {",", "\n"}] // ToExpression

{{1099.5, 0, -166.79}, {1098.4, 0, -166.79}}

$\endgroup$
2
  • $\begingroup$ Thanks, this work, but ToExpression seem drains the performance on large data files. $\endgroup$ Commented Nov 19, 2013 at 0:48
  • $\begingroup$ @xslittlegrass No matter what the performance hit is (And I don't think it's much), it will still be much faster than Import. In terms of ReadList this is the best you can do. Also if you want to deal with large CSV files look here for my answer: mathematica.stackexchange.com/questions/35371/… $\endgroup$
    – RunnyKine
    Commented Nov 19, 2013 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.