Joining some strings in a list of strings

Question

So, say I have a list of strings, representing lines in a file, like so:

Pillsy`testLines =
 {"foo",
  "bar\\",
  "baz\\",
  "quux",
  "wongle\\ bongle",
  "wingle",
  "pringle\\",
  "prongle",
  "blort"};

These use the (common?) convention that if a line ends with a backslash, it should be appended to the following line, giving a result like this:

Pillsy`testResult =
  {"foo", "barbazquux", "wongle\\ bongle", "wingle", "pringleprongle", "blort"};

Now, the naïve way to accomplish this is to use pattern matching, but the performance is likely to be really awful if you've got a lot of lines:

Pillsy`naiveCatenateContinuedLines[lines : {___String}] :=
 lines //.
  {before___, line1_, line2_, after___} /;
    StringMatchQ[line1, ___ ~~ "\\" ~~ EndOfString] :>
   {before, StringDrop[line1, -1] <> line2, after};

You've got two potential performance hits, one with repeated breaking of the list into BlankNullSequences, and the other with repeated StringJoins (I'm actually not 100% sure that the last is avoidable, but it's certainly possible that StringJoining many strings at once is efficient, and that would explain why StringJoin has the Flat attribute).

I ended up doing what any functional programmer does when confronted with a problem they don't know how to solve: I used Fold to accumulate a linked list. This solution works, and seems like it could be efficient (in part by exploiting the Flatness of StringJoin), but I haven't done actual performance testing. EDIT: Now I have a file to test with (linked below), and this solution runs in about 4 milliseconds. The naive solution takes about 25 milliseconds, and scales quadratically with length.

Pillsy`catenateContinuedLines[lines : {___String}] :=
 Module[{catenating},
  Attributes[catenating] = {HoldAllComplete};

  Flatten@Fold[
    Function[{acc, line},
     With[{
       (* This allows me to avoid repeating myself, but is a bit nuts. *)
       thunk =
        If[
         0 < StringLength@line && StringTake[line, -1] == "\\",
         With[{dropped = StringDrop[line, -1]},
          catenating@StringJoin[#, dropped] &],
         StringJoin[#, line] &]
       },
      acc /. {
        {init_, catenating[arg_]} :> {init, thunk[Unevaluated@arg]},
        _ -> {acc, thunk[""]}
        }]],
    {},
    lines]]

I checked briefly to see if there was an option for Import[file, {"Text", "Lines"}] that would allow you to specify a line continuation character, but nothing jumped out at me. I have a solution, but it seems needlessly convoluted.

EDIT to add: I have a semi-realistic test file that's long enough that I can do some timing, but unfortunately can't make it public.

EDIT again to add: OK, I munged any identifying information in the file beyond all recognition, so if you want something for test timings, you can find it here.

Answer 1

new one

This seems to be quite fast (didn't test with yours)

sc[{n_String}] := n;
sc[s_] := StringJoin[StringDrop[Most[s], -1]] <> Last[s]

sc /@ Split[testLines, StringMatchQ[#, "*\\"] &]

old one

StringJoin @@@ (
  Split[testLines, StringMatchQ[#, "*\\"] &] /. s : {Repeated[_String, {2, ∞}]} :> 
                                                     StringReplace[s, "\\" -> ""])

{"foo", "barbazquux", "wongle\\ bongle", "wingle", "pringleprongle", "blort"}

Notice that this is Split not SplitBy. Important difference because from adjacent elements we only test former one.

Answer 2

This solution requires that you know some character that does not occur in your string, in this example "|"

jacob[strList_] :=
 StringSplit[
  StringReplace[StringJoin[Riffle[strList, "|"]], "\\|" -> ""], "|"]

Maybe it will be fast because there are few function calls.

Modified definitions of other answers

sc[{n_String}] := n;
sc[s_] := StringJoin[StringDrop[Most[s], -1]] <> Last[s]

kuba[strList_] :=
 sc /@ Split[strList, StringMatchQ[#, "*\\"] &]

kubaOld[strList_] :=
 StringJoin @@@ (Split[strList, StringMatchQ[#, "*\\"] &] /. 
    s : {Repeated[_String, {2, \[Infinity]}]} :> 
     StringReplace[s, "\\" -> ""])

(*the solution by aky took more than 10 seconds*)
(*the solution in Pillsy's answer also took more than 10 seconds *)

Timing comparison

(jacobRes = jacob[words]) // Timing // First
(kubaRes = kuba[words]) // Timing // First
(kubaOldRes = kubaOld[words]) // Timing // First
(rmrfRes = joinWords[words]) // Timing // First
jacobRes === kubaRes === kubaOldRes == rmrfRes

gives

0.449849
1.694213
2.215579
4.350092
True

Answer 3

This is similar to Kuba's solution, but uses the listability of ToCharacterCode/FromCharacterCode instead of string patterns. Sometimes, this can be a bit faster than string manipulations:

joinWords[list_] := Join @@@ (Split[ToCharacterCode@list, Last[#] == 92 &] /. 
    {h__, 92} :> {h}) // FromCharacterCode

joinWords@Pillsy`testLines   
(* {"foo", "barbazquux", "wongle\\ bongle", "wingle", "pringleprongle", "blort"} *)

Answer 4

One alternative solution, that seems faster, but isn't really that much less horrible, uses Sow/Reap and increments a counter to use as a tag:

Pillsy`reapCatenateContinuedLines[lines : {___String}] :=
 Module[{counter = 0},
  StringJoin @@@ Last@Reap[
     Scan[
      If[StringMatchQ[#, ___ ~~ "\\"],
        Sow[StringDrop[#, -1], counter],
        Sow[#, counter++]] &,
      lines]]];

Answer 5

Haven't really evaluated this in terms of speed, but throwing it into the mix anyway:

f[] := Sequence[]
f[first_String, rest___String] := 
 If[StringTake[first, -1] == "\\", 
  f[StringDrop[first, -1] <> First@{rest}, 
   Sequence @@ Rest@{rest}], {first, f[rest]}]

Flatten[f @@ Pillsy`testLines]
(* {"foo", "barbazquux", "wongle\\ bongle", "wingle", "pringleprongle", "blort"} *)