Any trick to import each line?
The key here is to open the file as a stream rather than trying to import it at once. Once you have it open as a stream use
ReadLine to read in a single line at a time. Then you can
ImportString the line itself and get its
Something like this would work
stream = OpenRead @ "arxiv-metadata-oai-snapshot.json";
(* use a dynamic array to capture the results incrementally *)
res = CreateDataStructure @ "DynamicArray";
lineNumber = 0;
While[(line = ReadLine @ stream) =!= EndOfFile,
id = ImportString[line, "RawJSON"]["id"];
(* now convert the dynamic array into a list *)
res = Normal @ res;
Note I'm using
Monitor so you can keep track of how far you are into the 2 million lines in the file.
You may find the above to be a little slow, because it has to import the entire JSON string for each line and grab the id field. If I know that each line has what I want written exactly like
"id":"the_id_i_want" then I could replace the
id = ImportString[....
id = First[
"\"id\":\"" ~~ Shortest[id__] ~~ "\"" :> id], $Failed]
and it goes much faster.