Read RDF N-Triples and replace them with a hashed index 'on the fly'

Question

I want to process a 181GB uncompressed RDF N-Triples file. You may download a small sample file (4KB) and read the file line by line.

ntFile = OpenRead["small.nt"];
line = ReadLine[ntFile];

While[line =!= EndOfFile,
   record = StringCases[line, {"<" ~~ Shortest[x__] ~~ ">" -> x, "\"" ~~ y__ ~~ "\"" -> y, "@" ~~ z__ ~~ " " -> z}];
   Print[record];
   line = ReadLine[ntFile]
];

So far so good, but now it comes the difficult part, (at least for me ;-) I want to replace each < http... > member of my record list with an index from a hash table.

(* Hash Table - e.g. for the first five lines of the text file you have
   1 http://example/astronomy.constellation.contains
   2 http://example/type.object.name
   3 http://example/type.property.expected_type
   4 http://example/astronomy.star
   5 http://example/type.property.schema
   6 http://example/astronomy.constellation
   7 http://example/rdf-syntax-ns#type
   8 http://example/type.property
   9 http://example/rdf-schema#Property
*)

The absolute requirement for such a task is that both the replacement and the creation of hash table must occur 'on the fly'.

An output of the first five records as it runs:

  (*
     {1, 2, "Stars", "en"}
     {1, 3, 4}
     {1, 5, 6}
     {1, 7, 8}
     {1, 7, 9}
  *)

What is the purpose of this processing ?

It will populate a database, possibly a graph database.

Perl Comparison

The solution in Perl, a friend of mine gave me some code, is 20 lines without spacing between. They define %myhash for hash table and then they have access to the table and the keys with the $ symbol e.g. $myhash.

About Compression

I have also a question about compression. If the file is compressed as gz then it is about 30GB. Is it possible to read lines with Wolfram Language, how it can be done ?

Answer 1

I worked out a solution, it might not be the most efficient one, but it works. I had to learn about Reap/Sow first and how you export Mathematica code to markdown ;-)

ntFile = OpenRead["small.nt"];
line = ReadLine[ntFile];
$hash = <||> ;
hashcnt = 1;

While[line =!= EndOfFile,
  record = StringCases[line, {"<" ~~ Shortest[x__] ~~ ">" -> x, "\"" ~~ y__ ~~ "\"" -> y, "@" ~~ z__ ~~ " " -> z}]];

  ndxrec = record /. Reap[
      If[StringMatchQ[#, "http:" ~~ __],
         ndx = Lookup[$hash, #];
	 If[MissingQ@ndx, 
            Sow[# -> hashcnt]; AppendTo[$hash, # -> hashcnt]; hashcnt++,
            Sow[# -> ndx]]
      ] & /@ record
  ][[2, 1]];

  (* Run as a script from command line to convert your nt file Or
     Add indexed record to the database .... *)
  Print[ndxrec];

  line = ReadLine[ntFile]
];

Close[ntFile];

HashTable Output

$hash //Dataset