6
$\begingroup$

I try to calculate the average length (in words) of paragraphs in a document. The data look like:

  data=<|923 -> {"The food at snack is a selection of popular Greek dishes. 
  The appetizer tray is good as is the Greek salad. We were \=
  underwhelmed with the main courses. There are 4-5 tables here so it's 
  sometimes hard to get seated."}, 
  924 -> {"This little place in Soho is wonderful. I had a lamb 
  sandwich and a glass of wine. The price shocked me for how small the 
  serving was, but then again, this is Soho. The staff can be a little 
  snotty and rude, but the food is great, just don't expect world-class 
  service."}, 
  925 -> {"ordered lunch for 15 from Snack last Friday.  On time, 
  nothing missing and the food was great.  I have added it to the 
  regular company lunch list, as everyone enjoyed their meal.", 
  "ordered lunch for 15 from Snack last Friday.  On time, nothing 
  missing and the food was great.  I have added it to the regular 
  company lunch list, as everyone enjoyed their meal."}|>

where keys are a number of documents and values are the paragraphs. I calculate the average length in words with the following function:

averegL[text_] := Module[{l = Length@text, res},
res = Total[WordCount[#] & /@ text]/l]
ParallelMap[averegL[#] &, data]; // AbsoluteTiming

Unfortunately, it takes a lot of time to calculate for all dataset (the dataset is ~ 300000 keys). I look for a way to speed up this calculation.

Improve this question
$\endgroup$
1
  • $\begingroup$ Here is a MWE more suited to the assessment of performance: ExampleData[{"Text", "PrideAndPrejudice"}]; data = Association@MapIndexed[First@#2 -> # &, StringSplit[%, "."]]; (Sentences can serve as paragraphs for the purposes of computation, perhaps.) $\endgroup$
    – Michael E2
    May 6, 2018 at 13:08

2 Answers 2

Reset to default
4
$\begingroup$

How picky are you about what counts as a word?:

ExampleData[{"Text", "PrideAndPrejudice"}];
assoc = Association@MapIndexed[First@#2 -> # &, StringSplit[%, "."]];

WordCount /@ assoc // Total // AbsoluteTiming
(*  {7.93687, 121851}  *)

Length@*StringSplit /@ assoc // Total // AbsoluteTiming
(*  {0.102869, 122541}  *)

Divide by Length@assoc to get the average (adds negligible time).

Improve this answer
$\endgroup$
2
  • $\begingroup$ thank a lot StringSplit together with Length is faster than WordCount ... $\endgroup$
    – Kiril Danilchenko
    May 6, 2018 at 13:34
  • 1
    $\begingroup$ Very good approach (+1 of course). Try also StringCount[#, WhitespaceCharacter]&. It saves an additional quarter of time (fewer memory bound operations). $\endgroup$
    – Henrik Schumacher
    May 6, 2018 at 16:33
3
$\begingroup$

No need for averegL[#] & and WordCount[#] &. averegL and WordCount are already functions. (Not pure functions, though, but this is not of interest for Map.)

Here, I construct a pure function with \[Function], the infix form of Function. 

ParallelMap[
 text \[Function] Total[WordCount /@ text]/N[Length[text]], 
 data
]

{0.015057, <|923 -> 40., 924 -> 52., 925 -> 32.|>}

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you @Henrik. this approach speeds up the calculation by 15% $\endgroup$
    – Kiril Danilchenko
    May 6, 2018 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.