0
$\begingroup$

I want to make a toy mapreduce demonstration in Mathematica using iterator. Iterator allows to simulate real case when we get input row by row, not nowing in advance how many rows are there in source. Consider the simple job known as Words Count. Each row is represented by a word in the text. We may simulate a text as list = {foo, foo, quux, labs, foo, bar, quux}.

To simulate row-by-row throwing of words the following iterator is fine [1]:

ClearAll[mIterator];
mIterator[fname_Symbol, a_List] :=
  With[{indices = a},
   With[{len  = Length[indices]},
    Module[{i = 0},
      ClearAll[fname];
      fname[] := With[{ind = ++i}, indices[[ind]] /; ind <= len];
      fname[] := Null;
 ]]];

We write:

mIterator[next, Thread[{{foo, foo, quux, labs, foo, bar, quux}, {1,1,1,1,1,1,1}}]]

Each next[] will return {"word", 1}, thus we simulate a mapper job. What I need is a reducer. (To be certain sorter and reducer at once as output of mIterator is not sorted.)

How to make a reducer function that takes the output of mIterator and returns (prints or saves to a dictionary):

bar     1
foo     3
labs    1
quux    2

? Limitation. Reducer does not know how may rows mapper will produce. I now realize that the question concerns procedural style, but even in "Mathematica Gets Big Data with HadoopLink" (blog.wolfram.com/2013/07/31/mathematica-gets-bigdata-with-hadooplink/) procedural style is used.

$\endgroup$
  • 4
    $\begingroup$ Check Tally and Counts. $\endgroup$ – Anjan Kumar May 4 '17 at 7:13
  • $\begingroup$ @Kuba, is it better now? $\endgroup$ – iot May 4 '17 at 16:07
  • 1
    $\begingroup$ @io_tuta Don't know if enough to get answers you look for but better, thanks! $\endgroup$ – Kuba May 4 '17 at 21:14
  • 1
    $\begingroup$ Possible duplicate of Efficient code for tallying entries in very large lists $\endgroup$ – garej May 5 '17 at 16:41
  • $\begingroup$ Also see 18813 $\endgroup$ – garej May 5 '17 at 16:45
3
$\begingroup$

It is hard to understand why you want to proceed with an awkward procedural approach when Mathematica has elegant built-in functional methods accomplish your goal. The built-in function Tally can do the job of both your producer and reducer on a list of words of arbitrary length. For example,

Tally[{"foo", "foo", "quux", "labs", "foo", "bar", "quux"}]

{{"foo", 3}, {"quux", 2}, {"labs", 1}, {"bar", 1}}

It is simple to write a function so the result is a dictionary.

toDictionary[words : {_String ..}] := AssociationThread @@ Transpose[Tally[words]]

toDictionary[{"foo", "foo", "quux", "labs", "foo", "bar", "quux"}]

<|"foo" -> 3, "quux" -> 2, "labs" -> 1, "bar" -> 1|>

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.