# How to import a large (~50Mb) json file in mathematica

I am trying to import a 46Mb json file into mathematica (which by the way I exported using mathematica) but its failing with the following messages:

Java::pexcptn: A Java exception occurred after the result was returned to Mathematica: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:216)
at java.lang.StringBuffer.toString(StringBuffer.java:585)
at org.json.JSONTokener.nextString(JSONTokener.java:250)
at org.json.JSONTokener.nextValue(JSONTokener.java:316)
at org.json.JSONObject.<init>(JSONObject.java:190)
at org.json.JSONTokener.nextValue(JSONTokener.java:319)
at org.json.JSONObject.<init>(JSONObject.java:205)
at org.json.JSONTokener.nextValue(JSONTokener.java:319)
at org.json.JSONArray.<init>(JSONArray.java:120)
at org.json.JSONTokener.nextValue(JSONTokener.java:322)
at org.json.JSONObject.<init>(JSONObject.java:205)
at org.json.JSONTokener.nextValue(JSONTokener.java:319)
at org.json.JSONArray.<init>(JSONArray.java:120)
at org.json.JSONTokener.nextValue(JSONTokener.java:322)
at System.Convert.JSONDump.jDecodeJSONString(JSONDump.java:79).


(full msg: http://i.imgur.com/RUx7ic9.png)

How can I import the file? Is there another format I should use for large files in json? (I have tried .dat and other formats, but they seem to loose the json structure information) (or should I just not use json for large collections of data?)

-

This seems to be a Java heap space error. I encounter this quite often when importing large Excel sheets, though I get a different error message: Import::nojmem. The help page for this error contains instructions that may be useful for you as well. Basically, you increase heap space using the following commands:

Needs["JLink"]
ReinstallJava[JVMArguments -> "-Xmx512m"]


In my experience you may need to expand heap memory beyond that given in the above and quite often no memory you allocate is sufficient. It seems that many importers that take the Java route take an incredible and unreasonable amount of memory.

CSV/TSV type files can be read (using ReadList) much quicker and use up less memory in the process.

-
Does the amount of memory it takes depend on the file size? (I think so, because i didnt have this problem when my file was smaller) If so, I don't like this solution because it wont be effective when the file becomes larger – José D. Mar 24 '14 at 22:04
@Trollkemada you can change the 512` is something larger, but I have found it is often ineffective. – Sjoerd C. de Vries Mar 24 '14 at 22:18
There are various JSON to CSV converters available on the internet. Perhaps something to try? – Sjoerd C. de Vries Mar 24 '14 at 22:22