Average length (in words) of paragraphs in document

I try to calculate the average length (in words) of paragraphs in a document. The data look like:

  data=<|923 -> {"The food at snack is a selection of popular Greek dishes.
The appetizer tray is good as is the Greek salad. We were \=
underwhelmed with the main courses. There are 4-5 tables here so it's
sometimes hard to get seated."},
924 -> {"This little place in Soho is wonderful. I had a lamb
sandwich and a glass of wine. The price shocked me for how small the
serving was, but then again, this is Soho. The staff can be a little
snotty and rude, but the food is great, just don't expect world-class
service."},
925 -> {"ordered lunch for 15 from Snack last Friday.  On time,
nothing missing and the food was great.  I have added it to the
regular company lunch list, as everyone enjoyed their meal.",
"ordered lunch for 15 from Snack last Friday.  On time, nothing
missing and the food was great.  I have added it to the regular
company lunch list, as everyone enjoyed their meal."}|>


where keys are a number of documents and values are the paragraphs. I calculate the average length in words with the following function:

averegL[text_] := Module[{l = Length@text, res},
res = Total[WordCount[#] & /@ text]/l]
ParallelMap[averegL[#] &, data]; // AbsoluteTiming


Unfortunately, it takes a lot of time to calculate for all dataset (the dataset is ~ 300000 keys). I look for a way to speed up this calculation.

• Here is a MWE more suited to the assessment of performance: ExampleData[{"Text", "PrideAndPrejudice"}]; data = Association@MapIndexed[First@#2 -> # &, StringSplit[%, "."]]; (Sentences can serve as paragraphs for the purposes of computation, perhaps.) – Michael E2 May 6 '18 at 13:08

How picky are you about what counts as a word?:

ExampleData[{"Text", "PrideAndPrejudice"}];
assoc = Association@MapIndexed[First@#2 -> # &, StringSplit[%, "."]];

WordCount /@ assoc // Total // AbsoluteTiming
(*  {7.93687, 121851}  *)

Length@*StringSplit /@ assoc // Total // AbsoluteTiming
(*  {0.102869, 122541}  *)


Divide by Length@assoc to get the average (adds negligible time).

• thank a lot StringSplit together with Length is faster than WordCount ... – Kiril Danilchenko May 6 '18 at 13:34
• Very good approach (+1 of course). Try also StringCount[#, WhitespaceCharacter]&. It saves an additional quarter of time (fewer memory bound operations). – Henrik Schumacher May 6 '18 at 16:33

No need for averegL[#] & and WordCount[#] &. averegL and WordCount are already functions. (Not pure functions, though, but this is not of interest for Map.)

Here, I construct a pure function with \[Function], the infix form of Function.

ParallelMap[
text \[Function] Total[WordCount /@ text]/N[Length[text]],
data
]


{0.015057, <|923 -> 40., 924 -> 52., 925 -> 32.|>}

• Thank you @Henrik. this approach speeds up the calculation by 15% – Kiril Danilchenko May 6 '18 at 13:21