I have some huge logs, which are text files, containing up to a million rows each, and I need them sorted.
Because trying to import one obviously makes Mathematica run out of memory, we must rule out approaches like this
At the present moment I use a childish method, splitting the file to be ordered in ten files 0.txt ... 9.txt, where k.txt contains only lines starting with the figure k; ordering each (as above) and, finally, gathering them.
It works fine in some respects but it's deadly slow, even on a solid state hard disk.
Does Mathematica ( 8.0 or 9.0 ) provide any built-in support or package for sorting files?
Any suggestion is highly welcome!
ADDENDUM : 2014 03 21
Firstly I apologize for my poor English and I thank for you adjusting: I will attend english.stackexchange.com, promise. I'm good enough in "from English to Italian" but I still walk backward on a finely greased, splippery, floor.
I have put a piece of my code here: https://www.dropbox.com/s/i9l4atcfwhcmg5q/fileSorter%20ver.%204.2%20unformatted.%20.zip
It operates splitting a file over and over again, up to pieces loadable in RAM, sorting them and sticking all togheter.
It works as intended but deadly slowly: sorting a file containing 10^7 records, a string of length 10^3 each, takes around two hours on my pc (P4 1,8 GHz, 2 Gb RAM, SSD) because of a wasteful cutting method.
At present moment I'm improving it to a single pass knife.
I'm thoroughly studying the Stream based hint you provided, also
The dabaase to be sorted is precisely structured; it' a tab delimited list of files, located on external hard disks :
file size, unit serial number, unit label, file date, file hash (crc32), file hash (crcmd5), filedate, folder, file base name, file extension