Timeline for Reading periodic elements from a large file
Current License: CC BY-SA 3.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:55 | history | edited | CommunityBot |
replaced http://mathematica.stackexchange.com/ with https://mathematica.stackexchange.com/
|
|
| Mar 6, 2013 at 23:23 | comment | added | Leonid Shifrin | @s0rce If this is a problem still, you could read the file in chunks, requesting these chunks from Mathematica. This would need some Java code modification though. | |
| Mar 6, 2013 at 23:22 | comment | added | Leonid Shifrin |
@s0rce Yes, Java heap space can be a problem. You can increase it by using command line parameters via "CommandLine" option in InstallJava (you will actually need to call ReinstallJava with the same option). You can read up on the web what the heap parameters should be, this is described in a million of places.
|
|
| Mar 6, 2013 at 23:13 | comment | added | Leonid Shifrin | @s0rce Ok, see here | |
| Mar 6, 2013 at 22:37 | comment | added | s0rce |
The error was: java.lang.OutOfMemoryError: Java heap space
|
|
| Mar 6, 2013 at 22:31 | comment | added | s0rce | Thanks for your quick response. I just realized I'm dumb. I wanted to read the whole file but I thought about it wrong and I put in a value for the rowSize but really that should just be 4 with 0 offset before and after, obviously. However, I'm still getting memory errors on OSX, might have to do with how much memory java is allocated? | |
| Mar 6, 2013 at 22:23 | comment | added | Leonid Shifrin |
@s0rce But why do you need to modify it at all? It has already a method getFloatColumn, and above there is an example of how to use it. All you have to do is to compute the offsets and total byte count of a row - and then just use it.
|
|
| Mar 6, 2013 at 22:14 | comment | added | s0rce | I've been using your invaluable java file loading code. I hope you don't mind me asking you a question here. I have some large files where I want to read in the whole file (all floats) and I figured I could simply edit your code to do that. However, I over simplified it and removed the buffered read (gist.github.com/lgordon/5103515) and now it runs out of memory loading large files. I've been trying to fix the buffered read based on (nadeausoftware.com/articles/2008/02/…) but I'm getting stuck :( (also I got it working on OSX). | |
| Nov 27, 2012 at 9:44 | comment | added | Leonid Shifrin |
@s0rce It is big endian, which is implicit in both the code for byteArrayToInt method and the getFloat() method of the ByteBuffer class that I use for the floats. Thanks for the accept,b.t.w. I hope this will help you. For really large files, some changes may be needed - one may have to launch JRE (Java) with more memory.
|
|
| Nov 27, 2012 at 3:11 | vote | accept | s0rce | ||
| Nov 27, 2012 at 3:11 | vote | accept | s0rce | ||
| Nov 27, 2012 at 3:11 | |||||
| Nov 26, 2012 at 23:56 | comment | added | s0rce | Sorry, if this is silly, but where is the endianness specified? | |
| Nov 26, 2012 at 10:43 | history | edited | Leonid Shifrin | CC BY-SA 3.0 |
added 247 characters in body
|
| Nov 26, 2012 at 10:30 | history | answered | Leonid Shifrin | CC BY-SA 3.0 |