Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am trying to import a 46Mb json file into mathematica (which by the way I exported using mathematica) but its failing with the following messages:

Java::pexcptn: A Java exception occurred after the result was returned to Mathematica: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3209)
    at java.lang.String.<init>(String.java:216)
    at java.lang.StringBuffer.toString(StringBuffer.java:585)
    at org.json.JSONTokener.nextString(JSONTokener.java:250)
    at org.json.JSONTokener.nextValue(JSONTokener.java:316)
    at org.json.JSONObject.<init>(JSONObject.java:190)
    at org.json.JSONTokener.nextValue(JSONTokener.java:319)
    at org.json.JSONObject.<init>(JSONObject.java:205)
    at org.json.JSONTokener.nextValue(JSONTokener.java:319)
    at org.json.JSONArray.<init>(JSONArray.java:120)
    at org.json.JSONTokener.nextValue(JSONTokener.java:322)
    at org.json.JSONObject.<init>(JSONObject.java:205)
    at org.json.JSONTokener.nextValue(JSONTokener.java:319)
    at org.json.JSONArray.<init>(JSONArray.java:120)
    at org.json.JSONTokener.nextValue(JSONTokener.java:322)
    at System.Convert.JSONDump.jDecodeJSONString(JSONDump.java:79).

(full msg: http://i.imgur.com/RUx7ic9.png)

How can I import the file? Is there another format I should use for large files in json? (I have tried .dat and other formats, but they seem to loose the json structure information) (or should I just not use json for large collections of data?)

share|improve this question
add comment

1 Answer 1

This seems to be a Java heap space error. I encounter this quite often when importing large Excel sheets, though I get a different error message: Import::nojmem. The help page for this error contains instructions that may be useful for you as well. Basically, you increase heap space using the following commands:

Needs["JLink`"]
ReinstallJava[JVMArguments -> "-Xmx512m"]

In my experience you may need to expand heap memory beyond that given in the above and quite often no memory you allocate is sufficient. It seems that many importers that take the Java route take an incredible and unreasonable amount of memory.

CSV/TSV type files can be read (using ReadList) much quicker and use up less memory in the process.

share|improve this answer
    
Does the amount of memory it takes depend on the file size? (I think so, because i didnt have this problem when my file was smaller) If so, I don't like this solution because it wont be effective when the file becomes larger –  Trollkemada Mar 24 at 22:04
    
@Trollkemada you can change the 512 is something larger, but I have found it is often ineffective. –  Sjoerd C. de Vries Mar 24 at 22:18
1  
There are various JSON to CSV converters available on the internet. Perhaps something to try? –  Sjoerd C. de Vries Mar 24 at 22:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.