10
$\begingroup$

I am trying to import a 46Mb json file into mathematica (which by the way I exported using mathematica) but its failing with the following messages:

Java::pexcptn: A Java exception occurred after the result was returned to Mathematica: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3209)
    at java.lang.String.<init>(String.java:216)
    at java.lang.StringBuffer.toString(StringBuffer.java:585)
    at org.json.JSONTokener.nextString(JSONTokener.java:250)
    at org.json.JSONTokener.nextValue(JSONTokener.java:316)
    at org.json.JSONObject.<init>(JSONObject.java:190)
    at org.json.JSONTokener.nextValue(JSONTokener.java:319)
    at org.json.JSONObject.<init>(JSONObject.java:205)
    at org.json.JSONTokener.nextValue(JSONTokener.java:319)
    at org.json.JSONArray.<init>(JSONArray.java:120)
    at org.json.JSONTokener.nextValue(JSONTokener.java:322)
    at org.json.JSONObject.<init>(JSONObject.java:205)
    at org.json.JSONTokener.nextValue(JSONTokener.java:319)
    at org.json.JSONArray.<init>(JSONArray.java:120)
    at org.json.JSONTokener.nextValue(JSONTokener.java:322)
    at System.Convert.JSONDump.jDecodeJSONString(JSONDump.java:79).

(full msg: https://i.sstatic.net/x5NJp.png)

How can I import the file? Is there another format I should use for large files in json? (I have tried .dat and other formats, but they seem to loose the json structure information) (or should I just not use json for large collections of data?)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

This seems to be a Java heap space error. I encounter this quite often when importing large Excel sheets, though I get a different error message: Import::nojmem. The help page for this error contains instructions that may be useful for you as well. Basically, you increase heap space using the following commands:

Needs["JLink`"]
ReinstallJava[JVMArguments -> "-Xmx512m"]

In my experience you may need to expand heap memory beyond that given in the above and quite often no memory you allocate is sufficient. It seems that many importers that take the Java route take an incredible and unreasonable amount of memory.

CSV/TSV type files can be read (using ReadList) much quicker and use up less memory in the process.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Does the amount of memory it takes depend on the file size? (I think so, because i didnt have this problem when my file was smaller) If so, I don't like this solution because it wont be effective when the file becomes larger $\endgroup$
    – José D.
    Commented Mar 24, 2014 at 22:04
  • $\begingroup$ @Trollkemada you can change the 512 is something larger, but I have found it is often ineffective. $\endgroup$
    – Sjoerd C. de Vries
    Commented Mar 24, 2014 at 22:18
  • 1
    $\begingroup$ There are various JSON to CSV converters available on the internet. Perhaps something to try? $\endgroup$
    – Sjoerd C. de Vries
    Commented Mar 24, 2014 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.