I try to calculate the average length (in words) of paragraphs in a document. The data look like:

  data=<|923 -> {"The food at snack is a selection of popular Greek dishes. 
  The appetizer tray is good as is the Greek salad. We were \=
  underwhelmed with the main courses. There are 4-5 tables here so it's 
  sometimes hard to get seated."}, 
  924 -> {"This little place in Soho is wonderful. I had a lamb 
  sandwich and a glass of wine. The price shocked me for how small the 
  serving was, but then again, this is Soho. The staff can be a little 
  snotty and rude, but the food is great, just don't expect world-class 
  service."}, 
  925 -> {"ordered lunch for 15 from Snack last Friday.  On time, 
  nothing missing and the food was great.  I have added it to the 
  regular company lunch list, as everyone enjoyed their meal.", 
  "ordered lunch for 15 from Snack last Friday.  On time, nothing 
  missing and the food was great.  I have added it to the regular 
  company lunch list, as everyone enjoyed their meal."}|>

where keys are a number of documents and values are the paragraphs. I calculate the average length in words with the following function:

averegL[text_] := Module[{l = Length@text, res},
res = Total[WordCount[#] & /@ text]/l]
ParallelMap[averegL[#] &, data]; // AbsoluteTiming

Unfortunately, it takes a lot of time to calculate for all dataset (the dataset is ~ 300000 keys). I look for a way to speed up this calculation.

  • Here is a MWE more suited to the assessment of performance: ExampleData[{"Text", "PrideAndPrejudice"}]; data = Association@MapIndexed[First@#2 -> # &, StringSplit[%, "."]]; (Sentences can serve as paragraphs for the purposes of computation, perhaps.) – Michael E2 May 6 at 13:08
up vote 4 down vote accepted

How picky are you about what counts as a word?:

ExampleData[{"Text", "PrideAndPrejudice"}];
assoc = Association@MapIndexed[First@#2 -> # &, StringSplit[%, "."]];

WordCount /@ assoc // Total // AbsoluteTiming
(*  {7.93687, 121851}  *)

Length@*StringSplit /@ assoc // Total // AbsoluteTiming
(*  {0.102869, 122541}  *)

Divide by Length@assoc to get the average (adds negligible time).

  • thank a lot StringSplit together with Length is faster than WordCount ... – Kiril Danilchenko May 6 at 13:34
  • 1
    Very good approach (+1 of course). Try also StringCount[#, WhitespaceCharacter]&. It saves an additional quarter of time (fewer memory bound operations). – Henrik Schumacher May 6 at 16:33

No need for averegL[#] & and WordCount[#] &. averegL and WordCount are already functions. (Not pure functions, though, but this is not of interest for Map.)

Here, I construct a pure function with \[Function], the infix form of Function.

ParallelMap[
 text \[Function] Total[WordCount /@ text]/N[Length[text]], 
 data
]

{0.015057, <|923 -> 40., 924 -> 52., 925 -> 32.|>}

  • Thank you @Henrik. this approach speeds up the calculation by 15% – Kiril Danilchenko May 6 at 13:21

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.