Timeline for How can I import a huge CSV file quickly?
Current License: CC BY-SA 3.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 21, 2015 at 19:03 | comment | added | alancalvitti |
@LeonidShifrin, re Association & Dataset I described timing studies showing a recursive trie deserialization Query looks linear in input. I don't the actual exponent but can't imagine the theoretical lower bound is sub-linear.
|
|
| Sep 21, 2015 at 18:59 | comment | added | alancalvitti |
@LeonidShifrin, someone at WTC2013 or '14 showed results of Mathematica vs C++ physics simulation, I think 5.5 vs 5 hours, marginal diff. Don't have the details as to what was compiled, but presumabily there's a lot of List involved. I'd like to see more benchmarks in real world tasks on all facets of data: volume, velocity, variety.
|
|
| Sep 21, 2015 at 18:36 | comment | added | Leonid Shifrin |
@alancalvitti I mean, that general-purpose structures like Lists and Associations are not the most efficient building blocks to store large volumes of, for example, tabular (numerical, string or otherwise some simple type) data. Standard operations would include typical sql-type operations (select, groupby, sort, part extraction, may be joins). The overhead of using top-level data structures may be anything from 20x to 100x and more in peformance, and 10x - 40x in memory use.
|
|
| Sep 21, 2015 at 18:23 | comment | added | alancalvitti |
@LeonidShifrin, can likely take advantage of composite keys as index to factor redundant keys threaded through tables and time-series . But my q is, since Query works without Dataset, what other core operations and requirements do you mean?
|
|
| Sep 21, 2015 at 17:26 | comment | added | Leonid Shifrin |
@alancalvitti There are several places where efficiency is lost in current Dataset. By far the main one is that Dataset is designed to store very general Mathematica data, and at the moment doesn't optimize neither core operations on it nor storage format for more specific data formats, like tabular data. This results in the large overhead in both memory use and the performance. It's not that such optimizations can't be done for Dataset, but that simply didn't happen yet. For that to happen, arguably its query language should become more formalized.
|
|
| Sep 21, 2015 at 17:18 | comment | added | alancalvitti |
@LeonidShifrin, re efficiency, since Query can be applied to arbitrarily nested List and Association w/o Dataset (unless it's generated internally- doc is unclear), do you mean specifically Dataset with its type system overhead? Or Query and operator semantics in general?
|
|
| Sep 21, 2015 at 4:18 | comment | added | alancalvitti | @SjoerdC.deVries, some ppl still program Commodore 64. | |
| Sep 20, 2015 at 15:10 | comment | added | Sjoerd C. de Vries | @alancalvitti Although memory prices may plummet, this not necessarily means that all users have sufficient memory. For a variety of reasons companies may still use 32 bit OSes which means that you're restricted to less than 4 GB. In my case, for security reasons, I'm restricted to using a 32 bit VDI, so memory efficiency is of utmost importance to me. | |
| Sep 18, 2015 at 19:27 | comment | added | Leonid Shifrin |
@alancalvitti Mathematica in general, and Dataset in particular, are not very efficient in terms of memory use. To some extent, you indeed can alleviate this with more memory. But take a 10x larger dataset and you face similar limits. I don't totally disagree with you, but speed and memory are related. LazyList doesn't just save memory - it saves memory while keeping the performance often on par, or 2-4 times worse, than built-in functions acting purely in-memory. There is never enough memory and speed - so saving either one is a good thing.
|
|
| Sep 18, 2015 at 18:51 | comment | added | alancalvitti | @LeonidShifrin, I use all 32GB on my Mac at work. We index ~4GB in Dataset w/ convenient but bulky Keys overhead. There's the consumer space, then there's the engr and industrial. Xeon, 64GB. Then there's cloud scalability EC2/R3. Gretzky's quote is skate where the puck will be. | |
| Sep 18, 2015 at 17:44 | comment | added | Leonid Shifrin | @alancalvitti Re: speed vs memory - I mostly agree. The memory is also important however, since most even medium-sized data sets are beyond the capacity of a typical box (with say 8Gb or 16 Gb of RAM), when you work with it in Mathematica. One has to take into account that in Mathematica, for code to be fast, it has to operate on large chunks of data at once , and also symbolic representation often requires more memory. So, for a typical user, memory issues are perhaps just as important, when working with Mathematica and reasonably large data sets. | |
| Sep 18, 2015 at 17:41 | comment | added | Leonid Shifrin |
@alancalvitti LazyList already partially supports Part, Span, All etc specs. But, to make it efficient during the import stage, one has to write an optimized importer. That's surely possible, but has not been done yet. Right now, you have to first import the file into LazyList entirely, and then work with LazyList. There is an option to import lazily too (so that it will only read as much of a file as needed to get the elements you want) - but not random access (like, last 1000 rows without importing the previous). So, indeed, a lot of room for optimization here.
|
|
| Sep 18, 2015 at 17:18 | comment | added | alancalvitti | @LeonidShifrin, Re: "slow speed ... the memory savings" - speed is more important than memory. The economy is moving to the real time limit, while the cost of storage is tumbling, eg FlashArray. Google gives Photos storage free just to mine the images >> Exa-scale like Horizon2020. | |
| Sep 18, 2015 at 17:12 | comment | added | alancalvitti |
@LeonidShifrin, This is great. But will LazyListImport support Part, Span, All , eg stream CSV 1st and 4th columns only or the last 1000 rows for example? More generally for trees you see where that's going.
|
|
| Sep 16, 2015 at 12:51 | comment | added | Leonid Shifrin |
@SjoerdC.deVries I agree, of course. I may update this post later with a much faster importer, if I have the time. LazyList has a mini-framework which makes it pretty easy to plug in custom importers. For example, I can construct one from the code in your answer - which is perhaps one thing I will do and post here, when I have the time.
|
|
| Sep 16, 2015 at 12:36 | comment | added | Sjoerd C. de Vries |
Lazy lists are great of course, but importing with LazyListImport or Import just takes too long at this moment. Nobody really wants to plan his lunch break so that it coincides with his data imports. Other systems can do this much faster and so should Mathematica if it wants to compete in the Big Data area.
|
|
| Sep 16, 2015 at 12:03 | history | answered | Leonid Shifrin | CC BY-SA 3.0 |