0
$\begingroup$

I have a large text file containing numeric data of 1000 groups each of 200 lines. The lines are unsorted. There is a 9 line header at the beginning of each group, i.e., 209 ⨉ 1000 lines altogether.

I want to drop the headers to have a file containing 200 ⨉ 1000 lines. I also want the 200 lines of each output group sorted by the first number in the line. What is a way to do this efficiently in Mathematica?

My actual data set is much larger than 200 lines and 1000 groups. I made up these number for the sake of discussion.

|improve this question
$\endgroup$
  • 1
    $\begingroup$ Have you read the documentation for Read, ReadList, Skip, etc.? $\endgroup$ – Mr.Wizard♦ Dec 21 '12 at 14:21
  • $\begingroup$ Could you put a large text file on dropbox or somewhere else so we can play with it? $\endgroup$ – Rolf Mertig Dec 21 '12 at 14:22
  • $\begingroup$ here is the link to a file with 134 steps each 1584 lines (including 9 header lines). I want 134*1575 lines sorted 1,2,3...1575,1,2,3,...1575,...(ex. first 1575 lines are ordered) cheers, $\endgroup$ – Sesna Secna Dec 21 '12 at 15:41
  • $\begingroup$ sorry, forgot to add link in previous comment: dropbox.com/s/i1b12qxlowdcccm/2.txt $\endgroup$ – Sesna Secna Dec 21 '12 at 16:02
1
$\begingroup$

Maybe not the fastest way, but this just takes a few minutes to write. Read and looping and checking will be faster, but takes more time to code.

Since I could not use Import on your dropbox link, I copied 2.txt to my website. Then, if you have a fast internet connection and reasonable hardware, the following should run in less than 10 seconds. Fast enough?

dat = Import["http://www.mertig.com/2.txt", "Table"];
sdat= Sort /@ 
   Select[SplitBy[dat, MatchQ[#, {__?NumberQ}] &], Length[#] > 10 &];

Dimensions@sdat

(*
===> {134, 1575, 5}
*)
|improve this answer
$\endgroup$
  • $\begingroup$ Thanks. this works well. although I intended to know how to do this in a "loop". $\endgroup$ – Sesna Secna Jan 3 '13 at 10:57
  • $\begingroup$ Why use loops if you don't have to? $\endgroup$ – Rolf Mertig Jan 3 '13 at 15:34
  • $\begingroup$ alright, the text file I'm trying to import is too big (4GB). but I need only (i*1575) + (10th and 1575th) lines. {i,0,10000}. importing it uses memory in very huge files and crashes. can I read only lines and select? $\endgroup$ – Sesna Secna Feb 8 '13 at 14:08
  • $\begingroup$ @SesnaSecna you can. no time right now. later. $\endgroup$ – Rolf Mertig Feb 8 '13 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.