Return to Revisions

As Leonid mentions in one of his answers one of the methods of managing complexity is using Domain Specific Languages (DSL's). In this answer I will provide links to documents, packages, blog posts, and discussions of creating and utilizing DSL's in Mathematica.

For a 2.5 minutes introduction see this video between 25:00 and 27:30.

When to apply DSL's

Here are some cases for applying DSL.

1. When designing conversational engines.

2. When there are too many tuning options and usage scenarios of the developed algorithms.

• For example, we have a bunch of search, recommendation, and interaction algorithms for a dating site. A different department designs interactive user interfaces for these algorithms. We make a natural language DSL that invokes the different algorithms according to specified outcomes.

3. When designing an API for a collection of algorithms.

• Just designing a DSL can bring clarity of what signatures should be in the API.

• NIntegrate's Method option was designed and implemented using a DSL. See this video between 25:00 and 27:30.

Designing DSL

1. Decide what kind of sentences the DSL is going to have.

• Are natural language sentences going to be used?

• Are the sentences mathematical in nature?

2. Prepare, create, or accumulate sentences.

• In some cases using Morphological Analysis can greatly help for coming up with use cases and the corresponding sentences.

3. Create a context free grammar that describes the sentences. (Or a large subset of them.)

• At this stage I use exclusively Extended Backus-Naur Form (EBNF).

4. Program parser(s) for the grammar.

• I use most of the time functional parsers.

• The package FunctionalParsers.m provides a Mathematica implementation of this kind of parsing.

• The package can automatically generate parsers from a grammar given in EBNF. (See the coding example below.)

5. Program an interpreter for the parsed sentences.

• At this stage the parsed sentences are hooked to the algorithms of the problem domain.

• In Mathematica this can be done fairly easy using Case to pick grammar rule wrappers.

6. Test the parsing and interpretation.

See the example below illustrating steps 3-6.

Introduction to using DSL's in Mathematica

1. This blog post of mine "Natural language processing with functional parsers".

2. This detailed slide-show presentation "Functional parsers for an integration requests language grammar" shows how to use the package FunctionalParsers.m over a small grammar.

Advanced example

The blog post "Simple time series conversational engine" discusses the creation (design and programming) of a simple conversational engine for time series analysis (data loading, finding outliers and trends.)

Here is movie demonstrating that conversation engine: http://youtu.be/wlZ5ANglVI4.

Other discussions

1. Small part from 17:30 to 21:00 of the WTC 2012 "Spatial Access Methods and Route Finding" presentation shows a DSL for points of interest queries.

2. The answer of the MSE question "CSS Selectors for Symbolic XML" uses FunctionalParsers.m.

Coding example

This example is for the steps 3-6 of the second section.

Load the package:

 Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/FunctionalParsers.m"]

Give a EBNF description of a DSL for food craviings:

ebnfCode = "
  <lovefood> = <subject> , <loveverb> , <object-spec> <@ \
LoveFood[Flatten[#]]& ;
  <loveverb> = ( 'love' | 'crave' | 'demand' ) <@ LoveType ;
  <object-spec> = ( <object-list> | <object> | <objects> | <objects-mult> ) \
<@ LoveObjects[Flatten[{#}]]& ;
  <subject> = 'i' | 'we' | 'you' <@ Who ; 
  <object> = 'sushi' | [ 'a' ] , 'chocolate' | 'milk' | [ 'an' ] , 'ice' , \
'cream' | 'a' , 'tangerine' ;
  <objects> = 'sushi' | 'chocolates' | 'milks' | 'ice' , 'creams' | \
'ice-creams' | 'tangerines' ; 
  <objects-mult> = 'Range[2,100]' , <objects> <@ Mult ;
  <object-list> = ( <object> | <objects> | <objects-mult> ) , { 'and' \
\[RightTriangle] ( <object> | <objects> | <objects-mult> ) } ; ";

Generate parses from EBNF string:

GenerateParsersFromEBNF[ToTokens@ebnfCode];

Test the parsers with a list of sentences:

sentences = {"I love milk", "We demand 2 ice creams", 
  "I crave 2 ice creams and 5 chocolates", 
  "You crave chocolate and milk"}; ParsingTestTable[pLOVEFOOD, 
 ToLowerCase@sentences, "Layout" -> "Horizontal"]

Next we implement interpreters. I am using WolframAlpha to get the calories. I gave up figuring out using EntityValue["Food",___]. (Since using is WolframAlpha it can be overridden inside Block.)

Clear[LoveObjectsCalories]
LoveObjectsCalories[parsed_] :=
  Block[{res, wares(*, WolframAlpha={}&*)},
   res = (StringJoin @@ 
      Flatten[Riffle[parsed, " and "] /. 
        Mult[{x_, y_}] :> (StringJoin @@ 
           Riffle[Flatten[{ToString[x], y}], " "])]);
   wares = WolframAlpha[res <> " calories", "DataRules"];
   {{"Result", 1}, "ComputableData"} /. 
     wares /. {{"Result", 1}, "ComputableData"} -> 
     Quantity[RandomInteger[{20, 1200}], "LargeCalories"]
  ];
LoveFoodCalories[parsed_] :=
  Block[{who, type},
   who = Cases[parsed, Who[id_] :> id, \[Infinity]][[1]];
   type = Cases[parsed, LoveType[id_] :> id, \[Infinity]][[1]];
   Which[
    who == "you",
    Row[{"No, I do not. I am a machine."}],
    type == "love",
    Row[{"you gain ", Sqrt[1*10.] parsed[[-1]], " per day"}],
    True,
    Row[{"you will gain ", parsed[[-1]]}]
   ]
  ];

Here the parsing tests are done by changing the definitions of the wrapping symbols LoveFood and LoveObjects.

Block[{LoveFood = LoveFoodCalories, LoveObjects = LoveObjectsCalories},
 ParsingTestTable[pLOVEFOOD, ToLowerCase@sentences, "Layout" -> "Vertical"]
]