2
$\begingroup$

Context:

I have a long article (on Wordpress). And I'm using Google Translate API (v2) to translate this article. This article has about 300 000 characters.

However, due to the limit of Google Translate API.

  • I can only translate a segment of text with a limit of 5000 characters.

So now, I need to split my long article into several segments of sentences. Then I send these small segments of sentences to Google Translate API

Problem:

I have a very long string. For example this string.

string = "This is the first sentence. This is the second sentence."

I want to partition the string into non-overlapping substrings (sentence) of length n (maxcharacter). There is a function StringPartition["string",n], but it doesn't count for the sentence. So it will cut the string.

string = "This is the first sentence. This is the second sentence. \
This is the third sentence."
maxcharacter = 40;
StringPartition[string, maxcharacter]

The result:

    {"This is the first sentence. This is the ", "second sentence. This \
is the third sente"}

I would like to:

  • partition the long string into a sub-list of sentence
  • the length of the sub-list does not exceed the maxcharacter count
$\endgroup$
0

3 Answers 3

1
$\begingroup$

Okay, reading your updated question, maybe this is what you want:

Starting with some sample data to test with:

string = ExampleData[{"Text", "DeclarationOfIndependence"}];
StringLength@string
(* 8936 *)

Get the sentences:

sentences = TextSentences[string];
(* Thanks @MarcoB *)

Now, the strategy is to use Split to group sentences together if they're sufficiently short. Then we can map StringJoin over the result. Split can take a test predicate which takes two adjacent members of the list. We can add some statefulness to this process. It's a bit wonky, and maybe someone has a more elegant idea, but here's what I've got at this point (I've tried to format it to make everything easy to inspect):

results =
  Module[
    {len = StringLength@sentences[[1]],
     threshold = 1000},
    StringJoin /@
      Split[
        sentences,
        If[len + StringLength@#2 < threshold,
           len += StringLength@#2; True,
           len = StringLength@#2; False] &]]

Now, there are some consequences to this. Let's look at the length of the results:

StringLength[results]
(* {407, 965, 967, 879, 803, 1115, 925, 990, 943, 908} *)

Some snippets could be longer than the threshold (because of very long sentences). I assumed that you'd want to preserve long snippets rather than drop them.

$\endgroup$
3
  • $\begingroup$ It works perfectly! Thank you very very much! ❤️ $\endgroup$
    – Nam Nguyen
    Jun 16, 2022 at 22:04
  • $\begingroup$ Yes, I want to preserve long snippets rather than drop them. :) and your solutions works. The idea is right. $\endgroup$
    – Nam Nguyen
    Jun 16, 2022 at 22:08
  • $\begingroup$ The use of Split is very smart! Thank you! $\endgroup$
    – Nam Nguyen
    Jun 16, 2022 at 22:16
4
$\begingroup$

Starting with your data:

string = "This is the first sentence. This is the second sentence. This is the third sentence."

There are two (at least :) ) ways to get the sentences:

sentences1 = TextCases[string, "Sentence"];
sentences2 = StringSplit[string, ". "];

Notice that these produce slightly different results.

You can use StringTake to limit how long each string is (it works if given a list of string, so no need to use Map):

StringTake[sentences1, maxcharacter]

One problem here is that if a string is less than maxcharacter, then StringTake will generate errors. You can fix this with UpTo:

StringTake[sentences1, UpTo[maxcharacter]]
$\endgroup$
5
  • 2
    $\begingroup$ In the spirit of providing alternatives, TextSentences[string] would also accomplish the same as TextCases[string, "Sentence"]. $\endgroup$
    – MarcoB
    Jun 16, 2022 at 15:43
  • $\begingroup$ StringTake - Take the n characters from several sentences. But I don't mean to do that. I want to divide a long paragraph into a list of several sentences, and for these number of character of several sentences is smaller than the max_character. $\endgroup$
    – Nam Nguyen
    Jun 16, 2022 at 15:54
  • 1
    $\begingroup$ TextSentences is robust. StringSplit is not intelligent. Consider: StringSplit["Moon is 2.4 million miles away. Second sentence. Third \ sentence.", "."] $\endgroup$
    – Syed
    Jun 16, 2022 at 16:08
  • $\begingroup$ @Syed yeah, I didn't want to go into all of the details that would make StringSplit robust. RegularExpression would probably come into play eventually. $\endgroup$
    – lericr
    Jun 16, 2022 at 16:38
  • 2
    $\begingroup$ @NamNguyen So, you want full sentences, but just sentences shorter than maxcharacter? It might be helpful if you just post the desired output as part of your question. For your three sentences, and some given maxcharacter, what exactly would you like to see? $\endgroup$
    – lericr
    Jun 16, 2022 at 16:45
2
$\begingroup$

Let's get some text from Charles Dicken's Oliver Twist.

Define:

strCD = "Although I am not disposed to maintain that the being born \
in a workhouse,is in itself the most fortunate and enviable \
circumstance that can possibly befall a human being,I do mean to say \
that in this particular instance,it was the best thing for Oliver \
Twist that could by possibility have occurred.The fact is,that there \
was considerable difficulty in inducing Oliver to take upon himself \
the office of respiration,--a troublesome practice,but one which \
custom has rendered necessary to our easy existence;and for some time \
he lay gasping on a little flock mattress,rather unequally poised \
between this world and the next:the balance being decidedly in favour \
of the latter.Now,if,during this brief period,Oliver had been \
surrounded by careful grandmothers,anxious aunts,experienced \
nurses,and doctors of profound wisdom,he would most inevitably and \
indubitably have been killed in no time.There being nobody \
by,however,but a pauper old woman,who was rendered rather misty by an \
unwonted allowance of beer;and a parish surgeon who did such matters \
by contract;Oliver and Nature fought out the point between them.The \
result was,that,after a few struggles,Oliver breathed,sneezed,and \
proceeded to advertise to the inmates of the workhouse the fact of a \
new burden having been imposed upon the parish,by setting up as loud \
a cry as could reasonably have been expected from a male infant who \
had not been possessed of that very useful appendage,a voice,for a \
much longer space of time than three minutes and a quarter. As Oliver \
gave this first proof of the free and proper action of his lungs, the \
patchwork coverlet which was carelessly flung over the iron bedstead, \
rustled; the pale face of a young woman was raised feebly from the \
pillow; and a faint voice imperfectly articulated the words, 'Let me \
see the child, and die. The surgeon had been sitting with his face \
turned towards the fire: giving the palms of his hands a warm and a \
rub alternately. As the young woman spoke, he rose, and advancing to \
the bed's head, said, with more kindness than might have been \
expected of him:

'Oh, you must not talk about dying yet.'

'Lor bless her dear heart, no!' interposed the nurse, hastily \
depositing in her pocket a green glass bottle, the contents of which \
she had been tasting in a corner with evident satisfaction.

'Lor bless her dear heart, when she has lived as long as I have, sir, \
and had thirteen children of her own, and all on 'em dead except two, \
and them in the wurkus with me, she'll know better than to take on in \
that way, bless her dear heart! Think what it is to be a mother, \
there's a dear young lamb do.'

Apparently this consolatory perspective of a mother's prospects \
failed in producing its due effect. The patient shook her head, and \
stretched out her hand towards the child.

The surgeon deposited it in her arms. She imprinted her cold white \
lips passionately on its forehead; passed her hands over her face; \
gazed wildly round; shuddered; fell back--and died. They chafed her \
breast, hands, and temples; but the blood had stopped forever. They \
talked of hope and comfort. They had been strangers too long.

'It's all over, Mrs. Thingummy!' said the surgeon at last.

'Ah, poor dear, so it is!' said the nurse, picking up the cork of the \
green bottle, which had fallen out on the pillow, as she stooped to \
take up the child. 'Poor dear!'

'You needn't mind sending up to me, if the child cries, nurse,' said \
the surgeon, putting on his gloves with great deliberation. 'It's \
very likely it _will_ be troublesome. Give it a little gruel if it \
is.' He put on his hat, and, pausing by the bed-side on his way to \
the door, added, 'She was a good-looking girl, too; where did she \
come from?'

'She was brought here last night,' replied the old woman, 'by the \
overseer's order. She was found lying in the street. She had walked \
some distance, for her shoes were worn to pieces; but where she came \
from, or where she was going to, nobody knows.'";

Split into sentences:

slist = TextSentences[strCD]

These are the number of characters in the sentences.

StringLength /@ slist

{300, 376, 216, 222, 402, 309, 127, 130, 40, 190, 241, 63, 99, 73, \
37, 147, 78, 32, 33, 58, 156, 12, 128, 43, 173, 83, 34, 131}

Define: (a function that I used in a recent answer.

cSplit3[inList_List, f_Function] := Module[{rem = inList, i},
  Reap[
    While[rem != {},
     i = 1;
     While[i <= Length@rem && f@Take[rem, i],
      i++;
      ];
     i--;
     Sow@rem[[1 ;; i]];
     rem = Drop[rem, i];
     ]
    ][[2, 1]]
  ]

Usage: (Here I have chosen max count of 600)

(chunks = cSplit3[slist, Total@StringLength@# < 600 &]) // Column

StringLength /@ chunks
{{300}, {376, 216}, {222}, {402}, {309, 127, 130}, {40, 190, 241, 
  63}, {99, 73, 37, 147, 78, 32, 33, 58}, {156, 12, 128, 43, 173, 
  83}, {34, 131}}

Output: (* see chunks *)


Original answer

Manipulate[
 StringJoin @@@ (Partition[Characters@string, UpTo[n]]) // TableForm
 , {{n, 10}, 10, 50, 1}
 ]
$\endgroup$
2
  • 1
    $\begingroup$ This is what I could make of the OP's description. Cutting the sentence midway to meet a column width requirement is not done typically. Consider: If a sentence has more than 40 characters, it would not fit into any sublist. $\endgroup$
    – Syed
    Jun 16, 2022 at 16:00
  • 1
    $\begingroup$ Thank you very much @Syed ❤️. It works for max count of 600. But for max count of 200, it is really slow (I don't understand why). I will choose the answer of lericr $\endgroup$
    – Nam Nguyen
    Jun 16, 2022 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.