So, say I have a list of strings, representing lines in a file, like so:
Pillsy`testLines =
{"foo",
"bar\\",
"baz\\",
"quux",
"wongle\\ bongle",
"wingle",
"pringle\\",
"prongle",
"blort"};
These use the (common?) convention that if a line ends with a backslash, it should be appended to the following line, giving a result like this:
Pillsy`testResult =
{"foo", "barbazquux", "wongle\\ bongle", "wingle", "pringleprongle", "blort"};
Now, the naïve way to accomplish this is to use pattern matching, but the performance is likely to be really awful if you've got a lot of lines:
Pillsy`naiveCatenateContinuedLines[lines : {___String}] :=
lines //.
{before___, line1_, line2_, after___} /;
StringMatchQ[line1, ___ ~~ "\\" ~~ EndOfString] :>
{before, StringDrop[line1, -1] <> line2, after};
You've got two potential performance hits, one with repeated breaking of the list into BlankNullSequence
s, and the other with repeated StringJoin
s (I'm actually not 100% sure that the last is avoidable, but it's certainly possible that StringJoin
ing many strings at once is efficient, and that would explain why StringJoin
has the Flat
attribute).
I ended up doing what any functional programmer does when confronted with a problem they don't know how to solve: I used Fold
to accumulate a linked list. This solution works, and seems like it could be efficient (in part by exploiting the Flat
ness of StringJoin
), but I haven't done actual performance testing. EDIT: Now I have a file to test with (linked below), and this solution runs in about 4 milliseconds. The naive solution takes about 25 milliseconds, and scales quadratically with length.
Pillsy`catenateContinuedLines[lines : {___String}] :=
Module[{catenating},
Attributes[catenating] = {HoldAllComplete};
Flatten@Fold[
Function[{acc, line},
With[{
(* This allows me to avoid repeating myself, but is a bit nuts. *)
thunk =
If[
0 < StringLength@line && StringTake[line, -1] == "\\",
With[{dropped = StringDrop[line, -1]},
catenating@StringJoin[#, dropped] &],
StringJoin[#, line] &]
},
acc /. {
{init_, catenating[arg_]} :> {init, thunk[Unevaluated@arg]},
_ -> {acc, thunk[""]}
}]],
{},
lines]]
I checked briefly to see if there was an option for Import[file, {"Text", "Lines"}]
that would allow you to specify a line continuation character, but nothing jumped out at me. I have a solution, but it seems needlessly convoluted.
EDIT to add: I have a semi-realistic test file that's long enough that I can do some timing, but unfortunately can't make it public.
EDIT again to add: OK, I munged any identifying information in the file beyond all recognition, so if you want something for test timings, you can find it here.