0
$\begingroup$

I have a large text file containing numeric data of 1000 groups each of 200 lines. The lines are unsorted. There is a 9 line header at the beginning of each group, i.e., 209 ⨉ 1000 lines altogether.

I want to drop the headers to have a file containing 200 ⨉ 1000 lines. I also want the 200 lines of each output group sorted by the first number in the line. What is a way to do this efficiently in Mathematica?

My actual data set is much larger than 200 lines and 1000 groups. I made up these number for the sake of discussion.

$\endgroup$
4
  • 1
    $\begingroup$ Have you read the documentation for Read, ReadList, Skip, etc.? $\endgroup$
    – Mr.Wizard
    Commented Dec 21, 2012 at 14:21
  • $\begingroup$ Could you put a large text file on dropbox or somewhere else so we can play with it? $\endgroup$ Commented Dec 21, 2012 at 14:22
  • $\begingroup$ here is the link to a file with 134 steps each 1584 lines (including 9 header lines). I want 134*1575 lines sorted 1,2,3...1575,1,2,3,...1575,...(ex. first 1575 lines are ordered) cheers, $\endgroup$ Commented Dec 21, 2012 at 15:41
  • $\begingroup$ sorry, forgot to add link in previous comment: dropbox.com/s/i1b12qxlowdcccm/2.txt $\endgroup$ Commented Dec 21, 2012 at 16:02

1 Answer 1

1
$\begingroup$

Maybe not the fastest way, but this just takes a few minutes to write. Read and looping and checking will be faster, but takes more time to code.

Since I could not use Import on your dropbox link, I copied 2.txt to my website. Then, if you have a fast internet connection and reasonable hardware, the following should run in less than 10 seconds. Fast enough?

dat = Import["http://www.mertig.com/2.txt", "Table"];
sdat= Sort /@ 
   Select[SplitBy[dat, MatchQ[#, {__?NumberQ}] &], Length[#] > 10 &];

Dimensions@sdat

(*
===> {134, 1575, 5}
*)
$\endgroup$
4
  • $\begingroup$ Thanks. this works well. although I intended to know how to do this in a "loop". $\endgroup$ Commented Jan 3, 2013 at 10:57
  • $\begingroup$ Why use loops if you don't have to? $\endgroup$ Commented Jan 3, 2013 at 15:34
  • $\begingroup$ alright, the text file I'm trying to import is too big (4GB). but I need only (i*1575) + (10th and 1575th) lines. {i,0,10000}. importing it uses memory in very huge files and crashes. can I read only lines and select? $\endgroup$ Commented Feb 8, 2013 at 14:08
  • $\begingroup$ @SesnaSecna you can. no time right now. later. $\endgroup$ Commented Feb 8, 2013 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.