Timeline for Dealing with large files
Current License: CC BY-SA 3.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 22, 2014 at 9:02 | answer | added | rhermans | timeline score: 6 | |
| Nov 15, 2012 at 20:00 | comment | added | s0rce | For large files I find it nice to monitor the progress of the Read, see my answer at: mathematica.stackexchange.com/questions/4640/… | |
| Nov 15, 2012 at 19:50 | history | edited | István Zachar | CC BY-SA 3.0 |
added 9 characters in body; edited title
|
| Nov 15, 2012 at 19:18 | answer | added | image_doctor | timeline score: 13 | |
| Nov 15, 2012 at 18:55 | comment | added | Leonid Shifrin | My framework has not been yet optimized to work with lists where every part is very small but the number of parts is just huge. OTOH, it is relatively straightforward to do. The framework itself need not be modified, since this scenario can be addressed by writing some code on top of it. I will try to work out some example tomorrow. | |
| Nov 15, 2012 at 17:41 | comment | added | Luap Nalehw | @rm-rf , I think I agree with tkott. The conference presentation suggests they are working on this for future releases ... let's hope so. | |
| Nov 15, 2012 at 17:16 | comment | added | Gustavo Delfino | There was a talk about this in the 2011 conference: "BigData: Demystifying Large Datasets in Mathematica with Nick Lariviere" wolfram.com/events/technology-conference/2011/videos.html | |
| Nov 15, 2012 at 17:09 | comment | added | tkott | @rm-rf I think Leonid's answer to that question doesn't quite explain how to go from one gigantic file to his system. Perhaps all is needed is that tutorial? | |
| Nov 15, 2012 at 16:55 | comment | added | rm -rf♦ | I too think this is a duplicate of the question that silvia linked to | |
| Nov 15, 2012 at 16:50 | comment | added | Silvia | this answer may be relevant. | |
| Nov 15, 2012 at 16:43 | history | tweeted | twitter.com/#!/StackMma/status/269118696969101312 | ||
| Nov 15, 2012 at 15:45 | comment | added | Murta | Unfortunately, at the moment Mathematica is good only in handle data on memory, what it a problem for big files. I know that they are working to change that, but maybe just in Mathematica 10! So, for now I use bash too. | |
| Nov 15, 2012 at 15:35 | comment | added | Luap Nalehw | thanks cormullion. I was really looking for a mathematica solution. | |
| Nov 15, 2012 at 15:23 | comment | added | cormullion |
cat temp.dat | tr " " "\n" | split -l 500 would be one of many Unix commands. sed would also be a good bet.
|
|
| Nov 15, 2012 at 15:03 | comment | added | Luap Nalehw | split looks useful. But my first 500 sequences are not neccessarily the first 500 lines. I have no idea what lines they fall on so would need a way of splitting the first 500 { ... } 's which doesn't look like an option | |
| Nov 15, 2012 at 15:00 | history | edited | Luap Nalehw | CC BY-SA 3.0 |
deleted 4 characters in body
|
| Nov 15, 2012 at 14:59 | comment | added | canadian_scholar | Actually, this SO discussion "Way to deal with large data files in Wolfram Mathematica" looks pretty key for this. stackoverflow.com/questions/2370570/… | |
| Nov 15, 2012 at 14:59 | comment | added | canadian_scholar |
I'm interested in this too. When I've been dealing with plain text files, I often use the split command in bash. i.e. split -l 500 data. Here's the man page.
|
|
| Nov 15, 2012 at 14:50 | history | asked | Luap Nalehw | CC BY-SA 3.0 |