This is a complementary answer, which shows mostly how to reduce memory use rather than speed (although later I might update it to address the speed issue as well). This answer is based on an undocumented functionality, so the usual warning applies: there is no guarantee that the method suggested below will work in future versions.
Using undocumented Streaming` module to load the file as LazyList object
I will show one way, based on a low-level part of the undocumented Streaming` functionality, available since version 10.1. It will not solve the speed issue in a straight-forward fashion, but, assuming that you want to work with your file later, it will improve the speed of loading all subsequent times, while keeping memory use pretty low.
Needs["Streaming`"]
We now import your file as a LazyList object, with a chunk size of 1000 rows:
(imported =
Streaming`LazyListImport["/Users/apple/Downloads/train-7000.csv", "CSV",1000]
)//AbsoluteTiming
(*
{134.237,« LazyList[{ID,VAR_0001,VAR_0002,VAR_0003,<<1926>>,VAR_1932,VAR_1933,VAR_1934,target},
{2,H,224,0,4300,C,0,0,false,<<1916>>,98,998,999999998,998,998,9998,9998,IAPS,0},...]»
}
*)
It admittedly takes time, but let's look at the memory usage:
MaxMemoryUsed[]
(* 68890440 *)
This is only 68Mb, compared to an almost 1Gb needed by Import - as shown in other answers. Moreover, the memory use won't dramatically increase for streaming import, even for a much larger file size, like your original file for example.
Working with LazyList objects
What can you do with LazyList? First of all, you can easily convert it to a normal list, using Normal:
Normal@imported//ByteCount//AbsoluteTiming
(* {0.887823,341779872} *)
which takes less than a second for your list. But, many core List operations are supported for LazyList. For example, you may try:
Take[imported, {1000, 2000}]
Drop[imported, 1000]
Select[imported, #[[4]] == 1 &]
imported[[10]]
imported[[{1,2,3,4}]]
Length[imported]
All of these (except single part extraction and Length) will return LazyList as a result. You can always convert it to a usual List with Normal, or continue working with LazyList objects, at any stage. Most operations which have special implementations for LazyList, are performed in the out-of-core fashion, so they don't require a list to be loaded into memory in full.
Some operations such as Take, Drop, Select are lazy by default, which means that they will only do real work when some elements are extracted from the list. You can force the eager execution for most of them, by wrapping them in Strict wrapper. For example:
Strict[Select[imported, #[[4]] == 1 &]] // AbsoluteTiming
(* {0.766582, Streaming`LazyList[...]} *)
Eager operations are still done out-of-core (so the memory use stays low), but they are performed immediately, on entire list.
Persistence and faster loads for subsequent work with the data
You can persist any given LazyList object into a directory, which you must first create:
CreateDirectory[mydir = FileNameJoin[{$TemporaryDirectory, "train-7000"}]]
(* "/var/folders/8r/lhqmmmj93hjgxbsx08g_5nhw0000gn/T/train-7000" *)
Now, here is how to persist LazyList:
Streaming`LazyListPut[imported, mydir] // AbsoluteTiming
(* {11.8504, "/var/folders/8r/lhqmmmj93hjgxbsx08g_5nhw0000gn/T/train-7000"} *)
It still takes time, although is an order of magnitude faster than importing the file. But the best part is that you can load the persisted LazyList back to memory in future sessions pretty fast:
Streaming`LazyListGet[mydir]//AbsoluteTiming
(*
{1.31455, « LazyList[{ID,VAR_0001,VAR_0002,VAR_0003,<<1926>>,VAR_1932,VAR_1933,VAR_1934,target},
{2,H,224,0,4300,C,0,0,false,<<1916>>,98,998,999999998,998,998,9998,9998,IAPS,0},...]»}
*)
Summary
The speed of import for the method I desrcibed is in fact worse than using plain Import, at least for smaller files (for larger files it may be different, because Import uses a lot of RAM, which makes it slow). However, the memory savings can be very substantial, and the described method should scale well for larger file sizes. One can also persist the resulting LazyList object on disk, and it will load much faster in all future work sessions.
The above method is based in undocumented functionality, so there is no guarantee that it will work in future versions.
Any feedback is more than welcome!
ImportandExportin CSV format." $\endgroup$