Timeline for Importing large data without using Import [closed]
Current License: CC BY-SA 3.0
29 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 9, 2015 at 8:43 | comment | added | Yves Klett | A MWE would still be tremendously useful to show and test alternative approaches. Those you could benchmark with your data and give feedback on their performance. But without the exact specs for your files this depends on too much speculation. | |
| May 8, 2015 at 23:34 | comment | added | Albert Retey |
concerning memory: do you know about $HistoryLength=0? if you don't do that Mathematica will remember all output you ever generated in a session. concerning import: if changing the simulation, I'd consider another format for your files, text is just not a very good encoding for numeric data. Other than that, you might find this question and answers to it helpful. If sticking with text, you can read lines with ReadList and type String instead of BindaryReadList as in the answer...
|
|
| May 8, 2015 at 22:33 | comment | added | HuShu | @AlbertRetey I have the output files in .out format, but the problem is, each output file has some unnecessary text which I want to remove, that's why all this hassle. Perhaps, I could go into the simulation itself and modify it so that it doesn't outputs those text. Hmmm! | |
| May 8, 2015 at 22:30 | comment | added | HuShu | @SjoerdC.deVries I have been trying to break up my code into cells and after I evaluate one cell and transfer the result to a variable into the next cell, I clear the variable of the previous cell. But this doesn't seem to work, my RAM consumption keeps increasing even though I clear the unnecessary variable, is there a way out? | |
| May 8, 2015 at 21:12 | comment | added | Albert Retey | and the code writes the presumably numeric results into text files? Can you change that and make that code save something else? It certainly is possible to read in these text files, but it might in total consume more time than changing the data format (in case that is possible)... | |
| May 8, 2015 at 19:29 | history | closed |
Yves Klett Bob Hanlon bbgodfrey Dr. belisarius gpap |
Not suitable for this site | |
| May 8, 2015 at 19:08 | comment | added | HuShu | @AlbertRetey No, it's generated from an N-body simulation like code. | |
| May 8, 2015 at 19:07 | comment | added | HuShu | @YvesKlett The actual data is around 250 MB and contains over 10^6 lines of numbers. A small working example will defeat the purpose of this question as I can very well import a small data list using either ReadList or Import. | |
| May 8, 2015 at 16:55 | answer | added | Mark Adler | timeline score: 2 | |
| S May 8, 2015 at 16:37 | history | suggested | Mark Adler | CC BY-SA 3.0 |
missing quote
|
| May 8, 2015 at 16:14 | review | Suggested edits | |||
| S May 8, 2015 at 16:37 | |||||
| May 8, 2015 at 14:50 | comment | added | Albert Retey |
do you generate the data with mathematica and reread it with mathematica? If yes, then use Dump and Get as you suggested (although I think Export[filename,expression,"MX"] is slightly cleaner). Anyway, if you want the data to be accessable with other programs you could also look at specific file formats like HDF5 which are at least partially supported from Mathematica and will be a much better choice to store large numeric arrays, even a whole collection in one file...
|
|
| May 8, 2015 at 12:48 | comment | added | Sjoerd C. de Vries |
You can use ReadList to read line-by-line or by blocks (use its third parameter). Erasing memory: just unset (=.) the variable you want to clear, or use ClearAll or Remove.
|
|
| May 8, 2015 at 12:21 | comment | added | george2079 | what are you talking about importing as strings and converting to numbers? I don,t see that in your code at all. | |
| May 8, 2015 at 9:51 | review | Close votes | |||
| May 8, 2015 at 19:29 | |||||
| May 8, 2015 at 9:34 | comment | added | Yves Klett | I guess it will be difficult to help you unless you provide a minimal working example. | |
| May 8, 2015 at 8:11 | comment | added | Gustavo Delfino | I would load the data files into a database like MySQL and then connect to it from Mathematica with DatabaseLink. MySQL's "LOAD DATA LOCAL INFILE ..." is incredibly fast. | |
| May 8, 2015 at 7:13 | comment | added | HuShu | @Algohi I tried using ReadList["file",String], but my computer completely froze, I checked that with ReadList the memory usage was much more compared to Import. Perhaps, I will import them in parts and Concatenate them. But still the data is stored in the memory, and my computer starts to act sluggishly. Is there a way, I can erase the memory once I have the data concatenated? | |
| May 8, 2015 at 6:41 | comment | added | Sektor | Plus, there's a missing quote in the code you provided. | |
| S May 8, 2015 at 6:38 | history | edited | Sektor | CC BY-SA 3.0 |
improve formatting
|
| S May 8, 2015 at 6:38 | history | suggested | Mark Adler | CC BY-SA 3.0 |
improve formatting
|
| May 8, 2015 at 6:20 | review | Suggested edits | |||
| S May 8, 2015 at 6:38 | |||||
| May 8, 2015 at 5:45 | comment | added | Basheer Algohi | Look at OpenRead and ReadList | |
| May 8, 2015 at 5:35 | comment | added | HuShu | Unfortunately I haven't found anything that specifically answers this question. | |
| May 8, 2015 at 5:07 | comment | added | Yves Klett | Please search the site for "Import large" and take a look at the listed threads. | |
| May 8, 2015 at 5:03 | comment | added | HuShu | I found the commands Get and Dump, but I am not sure how to apply a loop statement to it. | |
| May 8, 2015 at 5:02 | history | edited | HuShu | CC BY-SA 3.0 |
deleted 341 characters in body
|
| May 8, 2015 at 4:59 | comment | added | Yves Klett | Could you please format your code using linebreaks? It is very hard to read at the moment. Did you search the site for already existing solutions yet? | |
| May 8, 2015 at 4:42 | history | asked | HuShu | CC BY-SA 3.0 |