Can Mathematica edit large binary files in-place?

Question

I have a very large binary data file (several hundred GB) which I would like to edit using Mathematica. The only edits required are a few isolated byte replacements—no insertions or deletions. Is it possible to use Mathematica's Stream functions to perform these edits in-place?

Neither OpenWrite nor OpenAppend works for this purpose, since OpenWrite overwrites existing files, treating them as empty, and OpenAppend can only append to the end of a file. As noted in the comments, Mathematica will not allow SetStreamPosition to be called on a file opened with OpenAppend.

Answer 1

We can do it straightforward in Java using JLink:

<< JLink`
InstallJava[];

replaceBytes[file_, bytes_] := Module[{f, result}, 
   f = JavaNew["java.io.RandomAccessFile", file, "rw"];
   writeByte[ff_, {offset_, byte_}] := Module[{old},
     ff@seek[offset];
     old = ff@readByte[];
     ff@seek[offset];
     ff@write[byte];
     {offset, old}
     ];
   result = writeByte[f, #] & /@ Select[bytes, #[[1]] < f@length[] &];
   f@close[];
   result
   ];

old = replaceBytes["testJava.dat", 
    {{260, 5}, {1, 2}, {10664756161, 64}, {100 10^9,200}}]

This will replace bytes at positions 260, 1 and 10664756161 to 5, 2 and 64 accordingly, it will drop the last replacement because corresponding offset goes beyond file length. It will also return the bytes that were there before replacement.

I tested it with 10Gb file.

Answer 2

I have noticed that NETLink does not get a lot of love on this site (at least not as much as JLink does), so let me just post this solution based on NETLink. Note that you can use NETLink on any of the popular Operating Systems out there (.NET platform is also now open source), but if you're on Windows, NETLink is highly preferred to JLink. Anyhow, here is the code:

Needs["NETLink`"]
InstallNET[];

editBinary[file_String?FileExistsQ, pos : {{_, _} ..}] := NETBlock[
  Module[{str},
   LoadNETType /@ {"System.IO.FileMode", "System.IO.FileAccess"};
   str = NETNew["System.IO.FileStream", file, FileMode`Open, 
     FileAccess`ReadWrite];
   Scan[(str @ Position = #[[1]]; str @ WriteByte[#[[2]]]) &, pos];
   str @ Close[]
   ]
  ]

Usage:

bfile = "C:\\Path\\To\\Your\\file";
editBinary[bfile, {{1, 65}, {5, 86}}]

The first element in each input list is the position at which the data is to be replaced while the second item in each list is the corresponding data to be written.

For those that like terser code, here is a shorter version that replaces the Enums with their corresponding integer values, hence, no need to load their types:

editBinary[file_String?FileExistsQ, pos : {{_, _} ..}] := 
  Module[{str},
   str = NETNew["System.IO.FileStream", file, 3, 3];
   Scan[(str @ Position = #[[1]]; str @ WriteByte[#[[2]]]) &, pos];
   str @ Close[]
   ]

Usage is same as before.

Answer 3

Mathematica is not a great tool to edit large files; the documentation essentially admits this. The obvious answer is to work in another language; I use Java, where it would be trivial to read a file as a sequence of bytes. To work with very large files, as you describe, I wrote a program to split them up into manageable chunks; it has two modes: Split and Reassemble, depending on command line arguments. If you can split your files into chunks of 100's of megabytes, I would recommend taking a look at HHD Hex Editor Neo. it is a beautiful piece of work that makes editing binary files easy. The standard version is free, but if you download it you have the full version capabilities for a 30-day trial period. I doubt you will need them though; the standard version is all I have needed. The MNIST files are a database of pictures of handwritten digits developed for optical character recognition for the USPS to sort mail. To make these files easier to work with, they have been converted to CSV form. Mathematica is great at importing CSV files; it is fast and accurate. But again you may never be able to import a 1 gigabyte file as CSV, much less a 100 GB file. There is Python code all over the Internet to convert the original binary MNIST files to CSV, for instance here: https://www.snip2code.com/Snippet/257756/Python-script-for-converting-the-MNIST-d. Perhaps you could adapt it to your needs.