# Efficiently deleting rows in many large text files

I have 500 text files each of about 200000 rows. I need to delete one row in each of them as fast as possible (later such operations will repeat). What is the best way to do this with Mathematica?

Solutions using NETLink, JLink etc. are welcome!

Update:

I have found a solution for C#: "Efficient way to delete a line from a text file" but I don't know how to adapt it for using through NETLink. Another option probably is to use the java.io.RandomAccessFile class in Java but I am not very familiar with Java and JLink. Any help will be appreciated!

-
Personally I think going with unix command line tools in this case will save you some pain. But you can call them from mma... Even if your on windows (like me) you can still get sed, awk, grep and the like. – Ajasja Apr 19 '12 at 7:44
@Ajasja Please provide an example of use of them under Windows. – Alexey Popkov Apr 19 '12 at 7:46
3 years ago, I have written a virtual file system in Java, which uses a single file as a back-end storage, and supports (sub-classes and provides own implementations of) Java InputStream, OutputStream and RandomAccessFile classes (which should be enough for any practical work). This file system was specifically designed to have fast insert/collapse operations for files, without a full file rewrite, and works orders of magnitude faster than the standard file system, for such operations. It has defragmentation utility, and is very customizable. – Leonid Shifrin Apr 19 '12 at 8:24
I just ran the unit tests and it works fine (I used a lot of stress testing at the time I wrote it). If you are interested, and would consider first copying your files into it (one-time operation), and then working with it, I can make a jar out of it, post a link to it, and provide some examples of use, hopefully later today or tomorrow. Of course, using it from Mathematica would involve JLink. The Java code itself is well-documented. – Leonid Shifrin Apr 19 '12 at 8:26
@Leonid I am interested, but I am not very familiar with Java. If you provide ready-to-use code for simple operations (get n'th line in the file(s), get lines fron n to k, delete n'th line, append line to file) it would be very helpful. And what is about fragmentation of the storage file itself? – Alexey Popkov Apr 19 '12 at 8:45

If you'll accept solutions involving other technologies, then how about using sed? On a unix system, to read in file1, delete row n and output to file2, you'd do:

sed -e 'nd' file1 > file2


The quotes in 'nd' are not necessary in this case, since there is only one instruction. However, if you're doing more complicated stuff with regular expressions, you should include the quotes so that the regex is not interpreted as something else. So it's safer to include it in general.

If you want to edit the file in place, then the command is

sed -i '' -e 'nd' test


Note that the ''after -i is necessary if you don't want a backup. If you omit that then sed assumes that the extension is being supplied via stdin, introducing additional files (which really are backups).

If you want to backup your file, just in case, then use -i.abc where abc is any string of your choice (need not be limited to 3 chars) and the file will be backed up to file.abc.

You can now run any of these commands from Mathematica via Run[command] (where command is the string with the actual command). Of course, replace n above with your actual row number or if variable, modify Run[...] accordingly.

I don't use Windows, but cygwin should provide you with a linux environment for Windows.

If you only require sed then cygwin is an overkill (takes a lot of space and is quite slow). A native build of sed can be found as part of GnuWin32. Download here.

-
Is it possible to modify the original file without reading/writing it entirely? I noticed that (for example) EmEditor saves a file with manually deleted row extremely fast (probably by modifying only a part of it on the disk). Are there any command-line tools with such a possibility? – Alexey Popkov Apr 19 '12 at 8:08
Also, sed has an inplace option if I remember correctly. But then the Q&A should be migrated to another stackexchange... – Ajasja Apr 19 '12 at 8:13
@Ajasja I added the in-place option. I don't think this should be migrated, because sometimes, the answer to how do I do X in mathematica is "Don't use Mathematica" :) – R. M. Apr 19 '12 at 8:47
@R.M I just have downloaded sed 4.2.1 and found that the correct syntax is sed -i -e nd test (without ''). – Alexey Popkov Apr 19 '12 at 10:10
Check out Gow (github.com/bmatzelle/gow/wiki) for some unix tools compiled natively on windows without cygwin. – s0rce May 12 '12 at 4:32