Timeline for How can I import a huge CSV file quickly?
Current License: CC BY-SA 3.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 24, 2015 at 19:39 | history | edited | Sjoerd C. de Vries | CC BY-SA 3.0 |
added 817 characters in body
|
| Sep 23, 2015 at 0:35 | comment | added | SquareOne |
@SjoerdC.deVries I made some more precise tests taking files with increasing number of lines (1000,5000,10000). For your code the speed varies actually from 1.6x faster to 1.3x faster than Import and i observe the reverse tendency for Jacob's code, it goes from 6.4x to 7.1x faster. (I am running mma 10.2, on OSX macbook)
|
|
| Sep 22, 2015 at 20:03 | comment | added | Sjoerd C. de Vries | Interesting, any idea why my code runs so much faster on your system than on my own (win8.1; mma10.2)? | |
| Sep 22, 2015 at 19:53 | comment | added | SquareOne |
According to my test it is not 20% speed increase, it is 2x faster than Import but 4x slower than Jacob's. However it is safer than ToExpression.
|
|
| Sep 22, 2015 at 14:08 | comment | added | Jacob Akkerboom |
Yeah I fed it full unsplitted lines/rows, see also my answer. It only gave errors after row 5000. Kind of unexpected that the quoted fields would give the trouble, but then again, it makes sense. By the way I wonder how ToExpression is programmed. "The Wolfram language" with all its infix notation may require a bloaty parser. I like your comparison with FromDigits and StringToDouble.
|
|
| Sep 21, 2015 at 17:40 | comment | added | Sjoerd C. de Vries |
ToExpression did throw errors (not unexpectedly) when it encountered the date fields. Not in your case then? Did you feed it full unsplitted lines?
|
|
| Sep 21, 2015 at 11:42 | comment | added | Jacob Akkerboom |
Which unfortunately trips up ToExpression. Otherwise I think ToExpression should not be too slow, when we feed it entire rows. Using ToExpression on the first 5000 rows (out of 7000) takes 5.49675 seconds for me.
|
|
| Sep 21, 2015 at 11:41 | comment | added | Jacob Akkerboom |
Nice answer, I think you cover all the best ingredients. However, I have some notes. First of all, using StringSplit["", ","]& fails here, because there are a few instances where there is a comma inside a field. That's why there are quotes surrounding the fields, the comma's are supposed to be shielded by the quotes. I suppose the same thing may happen if there are returns in fields (which this standard) allows, so that ReadList may fail, but that is more pathological. The csv file also contains an entry "PSYCHOLOGIST\CLINICAL PSYCHOLOGIST"... [cont]
|
|
| Sep 20, 2015 at 19:30 | history | undeleted | Sjoerd C. de Vries | ||
| Sep 20, 2015 at 19:30 | history | edited | Sjoerd C. de Vries | CC BY-SA 3.0 |
added 129 characters in body
|
| Sep 20, 2015 at 19:11 | history | edited | Sjoerd C. de Vries | CC BY-SA 3.0 |
added 868 characters in body
|
| Sep 16, 2015 at 20:31 | history | deleted | Sjoerd C. de Vries | via Vote | |
| Sep 16, 2015 at 20:30 | history | undeleted | Sjoerd C. de Vries | ||
| Sep 16, 2015 at 20:29 | history | deleted | Sjoerd C. de Vries | via Vote | |
| Sep 16, 2015 at 20:27 | comment | added | Sjoerd C. de Vries | I have been doing quite some editing in the answer box for readability, which may have caused some problems. I'll look into that. Btw I have not forgotten about file closure, I just did not copy it from my notebook. I'll delete the answer for now until I have time to work on this. | |
| Sep 16, 2015 at 14:31 | history | edited | Sjoerd C. de Vries | CC BY-SA 3.0 |
edited body
|
| Sep 16, 2015 at 12:07 | history | edited | Sjoerd C. de Vries | CC BY-SA 3.0 |
added 83 characters in body
|
| Sep 16, 2015 at 11:59 | history | edited | Sjoerd C. de Vries | CC BY-SA 3.0 |
added 1041 characters in body
|
| Sep 16, 2015 at 10:52 | history | answered | Sjoerd C. de Vries | CC BY-SA 3.0 |