Overwriting first line of large file

Question

As mentioned, I wish to read the first line of a file, and if needed, overwrite it with a new string. The aim is to have a CSV with a list of possible elements. Example:

Adding the elements: 5 A, 6 B, and 7 C to a blank CSV:
A B C
5 6 7
Adding 4 A, 9 D:
A B C D
5 6 7
4 0 0 9
Adding 2 B, 7 E
A B C D E
5 6 7
4 0 0 9
0 2 0 0 7

Rewriting the file is okay for small files, but the files are expected to grow to sizes of hundreds of MB, so repeatedly overwriting the entire file is not practiceable, since the entries are preferably done quickly.

How would one go about overwriting only the first line of the file? Reading the first line is possible from https://mathematica.stackexchange.com/questions/5179/how-to-read-data-file-quickly#=, but what about writing?

Answer 1

Low-level file operators like Write won't work here because OutputStreams (such as you get with OpenWrite and OpenAppend) can't have their StreamPosition set before the end of the file. In general overwriting characters in an existing file isn't terribly trivial; you can use c-functions likefile_ptr = fopen(file, "rb+") to do this, but you're overwriting character-by-character, so if you remove a character or add one it can throw things off.

Generally you're better off creating a copy of the data, along the lines of:

 replaceNthRecord[file_String, n_Integer, replaceWith_] := Module[
  {readStream = OpenRead[file], writeStream = OpenWrite[file <> "temp"], temp},
  Do[Write[writeStream, Read[readStream]], {n - 1}];
  Write[writeStream, replaceWith]; Read[readStream];
  While[temp =!= EndOfFile,
   temp = Read[readStream];
   If[UnsameQ[temp, EndOfFile], Write[writeStream, temp]];
   ]; Close /@ {readStream, writeStream}
  ]

Close[stream]

"file.dat"

replaceNthRecord["file.dat", 3, 99]

{"file.dat", "file.dattemp"}

Make sure that "file.dattemp" is showing what you want, and then delete the original file and replace it with the temp one:

ReadList["file.dattemp"]

{1, 2, 99, 4, 5, 6, 7, 8, 9, 10}

(then just add this to the end of the function:

CopyFile[file <> "temp", file, OverwriteTarget -> True];
DeleteFile[file <> "temp"]

Note that using ReadList appropriately here will be more efficient than Do[Write[_,Read[_]]] but you'll need to do a bit more work to figure out what an appropriate batch size is to do this in and make sure you stop in the right place.

Answer 2

Being fairly annoyed that mathematica can't do this straightforward thing..here is a solution using an external python script:

 Export["test.txt", 
      Join[{StringJoin[Join[{"*"}, ConstantArray[" ", {80}], {"*"}]]},
           RandomInteger[100, {3, 20}], {"end of file\n"}], "Table"]
 FilePrint["test.txt"]
 overwrite[file_, off_, string_] := 
       Run[ "python", "replacebytes.py", file, off, "'" <> string <> "'"];
 firstline = "some header 1 2";
 overwrite["test.txt", 2, firstline];
 FilePrint["test.txt"]
 firstline = firstline <> " 3";
 overwrite["test.txt", 2, firstline];
 FilePrint["test.txt"]

Where the python script "replacebytes.py" is just this:

 # usage python replacebytes.py file offset string
 # ovewrites file with string beginning at offset
 import sys
 if len(sys.argv) != 4 : sys.exit()
 f=open(sys.argv[1],'r+')
 f.seek(int(sys.argv[2]))
 f.write(sys.argv[3])
 f.close()

Note this will happily overwrite your data if firstline exceeds the length of the blank padded first line. (note also the asterisks are in there just so we can see whats going on, drop those in practice )