27
$\begingroup$

I have a very large binary data file (several hundred GB) which I would like to edit using Mathematica. The only edits required are a few isolated byte replacements—no insertions or deletions. Is it possible to use Mathematica's Stream functions to perform these edits in-place?

Neither OpenWrite nor OpenAppend works for this purpose, since OpenWrite overwrites existing files, treating them as empty, and OpenAppend can only append to the end of a file. As noted in the comments, Mathematica will not allow SetStreamPosition to be called on a file opened with OpenAppend.

$\endgroup$
  • 2
    $\begingroup$ Yes, in theory you can do that if you are very careful. I don't know how well it will work out in practice. I have never dealt with a binary file anywhere near as big as the one you want to work on. I'd worry about corrupting the file. I would work on a copy. Here is a reference you should look at Low-Level File Operations. Pay particular attention to the sections Seeking in Streams and Binary Data $\endgroup$ – m_goldberg Dec 21 '16 at 8:07
  • $\begingroup$ @m_goldberg I've looked at those functions, and in theory SetStreamPosition and BinaryWrite combined should be able to perform this task. The problem is that Mathematica will not allow you to simultaneously use these functions, because a file opened with OpenWrite is treated as empty, and a file opened with OpenAppend does not support SetStreamPosition. AFAIK there is no other way to open a file for writing. $\endgroup$ – David Zhang Dec 21 '16 at 16:04
  • $\begingroup$ Can you not open the file for append and then move the stream position to where you want to make your edits? $\endgroup$ – m_goldberg Dec 21 '16 at 17:02
  • 1
    $\begingroup$ @m_goldberg Nope! But don't take my word for it: f = OpenAppend["bigFile.dat"]; SetStreamPosition[f, 0] returns $Failed and issues the warning message SetStreamPosition::openw: OutputStream[...] is open for output. Like I said, OpenAppend does not support SetStreamPosition. $\endgroup$ – David Zhang Dec 21 '16 at 18:32
  • $\begingroup$ It's been a long time since I last tried to use low-level IO. My memory tells me that back then -- many releases of Mathematica ago -- one could move the stream position in a file open for writing. But my memory may be faulty. I have confirmed that Mathematica will not allow such an operation in V11.0.1. Sorry for misleading you. $\endgroup$ – m_goldberg Dec 22 '16 at 1:27
16
+150
$\begingroup$

We can do it straightforward in Java using JLink:

<< JLink`
InstallJava[];

replaceBytes[file_, bytes_] := Module[{f, result}, 
   f = JavaNew["java.io.RandomAccessFile", file, "rw"];
   writeByte[ff_, {offset_, byte_}] := Module[{old},
     ff@seek[offset];
     old = ff@readByte[];
     ff@seek[offset];
     ff@write[byte];
     {offset, old}
     ];
   result = writeByte[f, #] & /@ Select[bytes, #[[1]] < f@length[] &];
   f@close[];
   result
   ];

old = replaceBytes["testJava.dat", 
    {{260, 5}, {1, 2}, {10664756161, 64}, {100 10^9,200}}]

This will replace bytes at positions 260, 1 and 10664756161 to 5, 2 and 64 accordingly, it will drop the last replacement because corresponding offset goes beyond file length. It will also return the bytes that were there before replacement.

I tested it with 10Gb file.

$\endgroup$
7
$\begingroup$

I have noticed that NETLink does not get a lot of love on this site (at least not as much as JLink does), so let me just post this solution based on NETLink. Note that you can use NETLink on any of the popular Operating Systems out there (.NET platform is also now open source), but if you're on Windows, NETLink is highly preferred to JLink. Anyhow, here is the code:

Needs["NETLink`"]
InstallNET[];

editBinary[file_String?FileExistsQ, pos : {{_, _} ..}] := NETBlock[
  Module[{str},
   LoadNETType /@ {"System.IO.FileMode", "System.IO.FileAccess"};
   str = NETNew["System.IO.FileStream", file, FileMode`Open, 
     FileAccess`ReadWrite];
   Scan[(str @ Position = #[[1]]; str @ WriteByte[#[[2]]]) &, pos];
   str @ Close[]
   ]
  ]

Usage:

bfile = "C:\\Path\\To\\Your\\file";
editBinary[bfile, {{1, 65}, {5, 86}}]

The first element in each input list is the position at which the data is to be replaced while the second item in each list is the corresponding data to be written.

For those that like terser code, here is a shorter version that replaces the Enums with their corresponding integer values, hence, no need to load their types:

editBinary[file_String?FileExistsQ, pos : {{_, _} ..}] := 
  Module[{str},
   str = NETNew["System.IO.FileStream", file, 3, 3];
   Scan[(str @ Position = #[[1]]; str @ WriteByte[#[[2]]]) &, pos];
   str @ Close[]
   ]

Usage is same as before.

$\endgroup$
3
$\begingroup$

Mathematica is not a great tool to edit large files; the documentation essentially admits this. The obvious answer is to work in another language; I use Java, where it would be trivial to read a file as a sequence of bytes. To work with very large files, as you describe, I wrote a program to split them up into manageable chunks; it has two modes: Split and Reassemble, depending on command line arguments. If you can split your files into chunks of 100's of megabytes, I would recommend taking a look at HHD Hex Editor Neo. it is a beautiful piece of work that makes editing binary files easy. The standard version is free, but if you download it you have the full version capabilities for a 30-day trial period. I doubt you will need them though; the standard version is all I have needed. The MNIST files are a database of pictures of handwritten digits developed for optical character recognition for the USPS to sort mail. To make these files easier to work with, they have been converted to CSV form. Mathematica is great at importing CSV files; it is fast and accurate. But again you may never be able to import a 1 gigabyte file as CSV, much less a 100 GB file. There is Python code all over the Internet to convert the original binary MNIST files to CSV, for instance here: https://www.snip2code.com/Snippet/257756/Python-script-for-converting-the-MNIST-d. Perhaps you could adapt it to your needs.

$\endgroup$
  • 4
    $\begingroup$ If you are going for the bounty: I will only award the bounty for an answer that shows a workable solution within Mathematica. If that solution uses J/Link, that is fine. But it should be written in a way that someone familiar only with Mathematica (and general programming), but not Java, can take it and make use of it. I disagree with the idea that one should not want to do this using Mathematica, although I do agree that interfacing with other languages (such as Java through J/Link) is likely necessary. $\endgroup$ – Szabolcs Dec 28 '16 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.