1
$\begingroup$
  • Well, the problem was solved today.

Since "testAll.csv" is generated by Java/Eclipse, and I noticed that the LineSeperator is "\r", so I just need to assign RecordSeperator->"\r".

here is the sample file

 fList1=Import["testAll.csv","Lines"];
 length1=fList1//Length

I get 775108 which is right.

However when I'm trying:

 fList2=ReadList["testAll.csv","Records"];
 length2=fList2//Length

I get 768065 which is the incorrect line number, and StringSplit couldn't keep all lines with right column number, ie length1 is not the same as length2.

I also tried to partition the file into 100 files[by Export], and ReadList works correctly!

Additionally, I've changed the .csv file to a .txt file, ReadList works well.


And I found another thing is: when I export the fList=Import["testAll.csv","Lines"]; as 'testAll_new.csv'.

Then ReadList["testAll_new.csv","Records"]; is right...


I found without right options, Import function also has the similar problem in auto parsing the file.

enter image description here

$\endgroup$
4
  • 2
    $\begingroup$ You need to provide the file or an example that shows the problem or no one here can really help. $\endgroup$
    – Ymareth
    Nov 20, 2014 at 9:40
  • $\begingroup$ You probably need to specify the correct RecordSeparator-s. It might be that your file contains a couple of them. $\endgroup$ Nov 20, 2014 at 10:07
  • $\begingroup$ @SjoerdC.deVries strange, I found ReadList the new .csv file Generated by Import and Export by Mathematica is OK... $\endgroup$ Nov 20, 2014 at 10:19
  • $\begingroup$ Is there a typo in your new edit ("lines" instead of "Lines")? $\endgroup$ Nov 20, 2014 at 10:21

1 Answer 1

4
$\begingroup$

Possible cause:

 f = OpenWrite["test.csv"];
 WriteString[f, "1,2,3\n\n4,5,6\n"];
 Close[f];
 FilePrint["test.csv"]

enter image description here

 Through[{Length, MatrixForm}@Import["test.csv", "Lines"]]
 Through[{Length, MatrixForm}@ReadList["test.csv", "Record"]]

enter image description here

Readlist doesn't count a blank line (double line ending) as a "record" while Import does

You can fix this with the option NullRecords -> True to ReadList

Interestingly if you have extra returns at the end of the file WriteString[f, "1,2,3\n\n4,5,6\n\n\n\n"]; Import ignores those, while ReadList[...,NullRecords->True] picks them up. The "Lines" import format does not seem to be documented by the way.

(note with the plural "Records" I get "Records is not a valid format specification" )

$\endgroup$
1
  • $\begingroup$ I found my line seperator is "\r" that's the cause... $\endgroup$ Nov 21, 2014 at 1:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.