How can I resolve an unexpected character error when trying to import selected data from a large JSON file?

Question

I'm downloading this datafile from kaggle.

I want to get all the IDs.

Normally for a JSON file I'd write

Import["./arxiv-metadata-oai-snapshot.json", {"Data", "id"}]

But I get the error

Import::jsonexpendofinput: Unexpected character found while looking for the end of input.
Import::jsonhintposandchar: An error occurred near character '"', at line 2:3

The only way I'd know to fix this is to import the ENTIRE file. It's massive. I really don't want to do that.

Answer 1

Any trick to import each line?

The key here is to open the file as a stream rather than trying to import it at once. Once you have it open as a stream use ReadLine to read in a single line at a time. Then you can ImportString the line itself and get its "id" field.

Something like this would work

stream = OpenRead @ "arxiv-metadata-oai-snapshot.json";
(* use a dynamic array to capture the results incrementally *)
res = CreateDataStructure @ "DynamicArray";
Monitor[
    lineNumber = 0;
    While[(line = ReadLine @ stream) =!= EndOfFile,
        id = ImportString[line, "RawJSON"]["id"];
        res["Append", id];
        lineNumber++
    ],
    lineNumber
];
(* now convert the dynamic array into a list *)
res = Normal @ res;

Note I'm using Monitor so you can keep track of how far you are into the 2 million lines in the file.

You may find the above to be a little slow, because it has to import the entire JSON string for each line and grab the id field. If I know that each line has what I want written exactly like "id":"the_id_i_want" then I could replace the

id = ImportString[....

line with

id = First[
  StringCases[line, 
   "\"id\":\"" ~~ Shortest[id__] ~~ "\"" :> id], $Failed]

and it goes much faster.