3
$\begingroup$

I have three sets of text blocs (actually many more...) which show part of a complete text. However the partitioning of the original text was not done correctly as some sentences were split between two text blocks.

text1 = {"We will talk about data about model specification parameter \
estimation and model application and the context where we will apply \
the simple example.Is an application where we would like to analyze \
the market for electric cars because"};

text2 = {"we are interested in the market of electric cars.The choice \
that we are interested in is the choice of each individual to \
purchase an electric car or not And we will see how"};

text3 = {"to address this question. Furthermore, it needs to be noted that this is only a model text and there is no content associated with it. "};

For instance text2 starts with "we are interested in the market of electric cars." This is an incomplete first sentence, which actually started in text block 1 (see last sentence there).

I want to make sure that each text block ends with a complete sentence. So I want to move incomplete first sentences to the last text block. For instance here, the result would be:

 text1corr = {"We will talk about data about model specification parameter \
    estimation and model application and the context where we will apply \
    the simple example.Is an application where we would like to analyze \
    the market for electric cars because we are interested in the market of electric cars."};

text2corr = {"The choice that we are interested in is the choice of each individual to purchase an electric car or not And we will see how to address this question."};

text3corr = {"Furthermore, it needs to be noted that this is only a model text and there is no content associated with it. "};

How can I do it in Mathematica ? Is this even possible ?

$\endgroup$
7
  • 1
    $\begingroup$ "I want to move incomplete first sentences to the last text block." I read this, then I read your examples, and I don't see the connection. $\endgroup$
    – C. E.
    Oct 5 '18 at 20:17
  • $\begingroup$ @C.E. See how text-block 2 starts with ""we are interested in the market of electric cars." ... this is an incomplete first sentence, which actually started in text block 1 (see last sentence there). So I would like to attach it to text-block 1 and delete it from text block 2. Is that helping ? $\endgroup$
    – james
    Oct 5 '18 at 20:21
  • $\begingroup$ @C.E. I edited my question. $\endgroup$
    – james
    Oct 5 '18 at 20:25
  • $\begingroup$ ok, I see it now. $\endgroup$
    – C. E.
    Oct 5 '18 at 20:28
  • $\begingroup$ can we assume that we know the ordering of blocks? $\endgroup$
    – kglr
    Oct 5 '18 at 22:29
3
$\begingroup$

Update2: Using ReplaceRepeated:

ClearAll[correct]
correct = {StringRiffle@#} & /@ ReplaceRepeated[#, 
 {a___, {b___, c_String?(Not[StringMatchQ[StringTrim@#, "*."]] &)}, {d_,  e___}, f___} :>
   {a, {b, c, d}, {e}, f}] & @
    (TextSentences /@ StringReplace[First /@ #, "." ~~ a: LetterCharacter :> ". " <> a]) &

correct @{text1,text2,text3} ==  {text1corrected, text2corrected , text3corrected } 

True

{StringRiffle@#} & /@ 
  ReplaceRepeated[#, {a___, {b___, c_String?(Not[StringMatchQ[StringTrim @ #, "*."]] &)}, 
    {d_, e___}, f___} :> {a, {b, c, d}, {e}, f}] & @ sents

Update: If text blocks are already made:

texts = First /@ {text1, text2, text3}; 
sents = TextSentences /@ StringReplace[texts, "." ~~ a : LetterCharacter :> ". " <> a];
sents2 = If[StringMatchQ[StringTrim[#[[1, -1]]], "*."], #[[1]], 
     Append[#[[1]], #[[2, 1]]]] & /@ Partition[sents, 2, 1, 1] ;
{text1corrected, text2corrected , text3corrected } = {StringRiffle @ #} & /@ 
  FoldList[If[#[[-1]] == #2[[1]], Drop[#2, 1], #] &, sents2]

{{"We will talk about data about model specification parameter estimation and model application and the context where we will apply the simple example. Is an application where we would like to analyze the market for electric cars because we are interested in the market of electric cars."},
{"The choice that we are interested in is the choice of each individual to purchase an electric car or not And we will see how to address this question."},
{"Furthermore, it needs to be noted that this is only a model text and there is no content associated with it. "}}

Original answer:

Taking a step back and doing the chunking of the original text to make sure that sentences are not broken:

(1) Use TextSentences to split the source text into sentences.

(2) If needed, use Split to split sentences into chunks subject the constraint that chunk sizes cannot exceed maxchunksize. The function chunk below does this:

ClearAll[chunk]
chunk[sentences_, maxchunksize_] := Module[{t = 0}, Split[sentences,  
  Or[(t += StringLength[#]) <= maxchunksize - StringLength[#2],  t = 0] &]] 

Example:

loremipsum = StringTake[ExampleData[{"Text", "LoremIpsum"}], 600]

"Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Integer nunc augue, feugiat non, egestas ut, rutrum eu, purus. Vestibulum condimentum commodo pede. Nam in metus eu justo commodo posuere. Nunc varius dui id nulla. Fusce porttitor pretium leo. Quisque in diam. Nulla pellentesque. Donec vitae urna et arcu lobortis varius. Aenean velit eros, varius ac, semper sit amet, lacinia eget, sem. Phasellus mollis nunc a pede cursus fermentum. Proin et odio. Nullam turpis. Fusce eget arcu. Mauris placerat ultricies lectus. Nullam pede nisl, ullamcorper ullamcorper, hendrerit sit amet, molestie ac,"

sentences = TextSentences[loremipsum ]

{"Lorem ipsum dolor sit amet, consectetuer adipiscing elit.",
"Integer nunc augue, feugiat non, egestas ut, rutrum eu, purus.",
"Vestibulum condimentum commodo pede.",
"Nam in metus eu justo commodo posuere.",
"Nunc varius dui id nulla.",
"Fusce porttitor pretium leo.",
"Quisque in diam.",
"Nulla pellentesque.",
"Donec vitae urna et arcu lobortis varius.",
"Aenean velit eros, varius ac, semper sit amet, lacinia eget, sem.",
"Phasellus mollis nunc a pede cursus fermentum.",
"Proin et odio.",
"Nullam turpis.",
"Fusce eget arcu.",
"Mauris placerat ultricies lectus.",
"Nullam pede nisl, ullamcorper ullamcorper, hendrerit sit amet, molestie ac,"}

Length @ sentences

16

StringLength[sentences]

{57, 62, 36, 38, 25, 28, 16, 19, 41, 65, 46, 14, 14, 16, 33, 75}

maxchunksize = 200;
chunks = chunk[sentences, maxchunksize]
StringJoin /@ chunks 

{"Lorem ipsum dolor sit amet, consectetuer adipiscing elit.Integer nunc augue, feugiat non, egestas ut, rutrum eu, purus.Vestibulum condimentum commodo pede.Nam in metus eu justo commodo posuere.",
"Nunc varius dui id nulla.Fusce porttitor pretium leo.Quisque in diam.Nulla pellentesque.Donec vitae urna et arcu lobortis varius.Aenean velit eros, varius ac, semper sit amet, lacinia eget, sem.",
"Phasellus mollis nunc a pede cursus fermentum.Proin et odio.Nullam turpis.Fusce eget arcu.Mauris placerat ultricies lectus.Nullam pede nisl, ullamcorper ullamcorper, hendrerit sit amet, molestie ac,"}

Length@chunks

3

StringLength /@ StringJoin /@ chunks

{193, 194, 198}

$\endgroup$
4
  • $\begingroup$ Thank you very much for your answer! Unfortunately, I can't do that. :( The text blocs are already made. They were extracted from subtitles from videos and there size varies a lot. I want to keep the original size of the text block but simply make sure that each last sentence is not cut into two blocks. Is that possible ? $\endgroup$
    – james
    Oct 6 '18 at 8:33
  • $\begingroup$ @james, please see the update. $\endgroup$
    – kglr
    Oct 6 '18 at 10:40
  • $\begingroup$ Wow! Very nice answer ! Thanks a lot !! Would you mind to explain what "~~ a : LetterCharacter :> ". " <> a" does ? $\endgroup$
    – james
    Oct 6 '18 at 11:24
  • $\begingroup$ @james, it puts a space after "." . TextSentences doesn't treat "." as the end of a sentence if it is immediately followed by a letter character. $\endgroup$
    – kglr
    Oct 6 '18 at 11:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.