0
$\begingroup$

I want to make a toy mapreduce demonstration in Mathematica using iterator. Iterator allows to simulate real case when we get input row by row, not nowing in advance how many rows are there in source. Consider the simple job known as Words Count. Each row is represented by a word in the text. We may simulate a text as list = {foo, foo, quux, labs, foo, bar, quux}.

To simulate row-by-row throwing of words the following iterator is fine [1]:

ClearAll[mIterator];
mIterator[fname_Symbol, a_List] :=
  With[{indices = a},
   With[{len  = Length[indices]},
    Module[{i = 0},
      ClearAll[fname];
      fname[] := With[{ind = ++i}, indices[[ind]] /; ind <= len];
      fname[] := Null;
 ]]];

We write:

mIterator[next, Thread[{{foo, foo, quux, labs, foo, bar, quux}, {1,1,1,1,1,1,1}}]]

Each next[] will return {"word", 1}, thus we simulate a mapper job. What I need is a reducer. (To be certain sorter and reducer at once as output of mIterator is not sorted.)

How to make a reducer function that takes the output of mIterator and returns (prints or saves to a dictionary):

bar     1
foo     3
labs    1
quux    2

? Limitation. Reducer does not know how may rows mapper will produce. I now realize that the question concerns procedural style, but even in "Mathematica Gets Big Data with HadoopLink" (blog.wolfram.com/2013/07/31/mathematica-gets-bigdata-with-hadooplink/) procedural style is used.

$\endgroup$
5
  • 4
    $\begingroup$ Check Tally and Counts. $\endgroup$ May 4, 2017 at 7:13
  • $\begingroup$ @Kuba, is it better now? $\endgroup$
    – iot
    May 4, 2017 at 16:07
  • 1
    $\begingroup$ @io_tuta Don't know if enough to get answers you look for but better, thanks! $\endgroup$
    – Kuba
    May 4, 2017 at 21:14
  • 1
    $\begingroup$ Possible duplicate of Efficient code for tallying entries in very large lists $\endgroup$
    – garej
    May 5, 2017 at 16:41
  • $\begingroup$ Also see 18813 $\endgroup$
    – garej
    May 5, 2017 at 16:45

1 Answer 1

3
$\begingroup$

It is hard to understand why you want to proceed with an awkward procedural approach when Mathematica has elegant built-in functional methods accomplish your goal. The built-in function Tally can do the job of both your producer and reducer on a list of words of arbitrary length. For example,

Tally[{"foo", "foo", "quux", "labs", "foo", "bar", "quux"}]

{{"foo", 3}, {"quux", 2}, {"labs", 1}, {"bar", 1}}

It is simple to write a function so the result is a dictionary.

toDictionary[words : {_String ..}] := AssociationThread @@ Transpose[Tally[words]]

toDictionary[{"foo", "foo", "quux", "labs", "foo", "bar", "quux"}]

<|"foo" -> 3, "quux" -> 2, "labs" -> 1, "bar" -> 1|>

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.