As Leonid mentions in one of his answers one of the methods of managing complexity is using Domain Specific Languages (DSL's). In this answer I will provide links to documents, packages, blog posts, and discussions of creating and utilizing DSL's in Mathematica.
For a 2.5 minutes introduction see this video between 25:00 and 27:30.
When to apply DSL's
Here are some cases for applying DSL.
When designing conversational engines.
When there are too many tuning options and usage scenarios of the developed algorithms.
- For example, we have a bunch of search, recommendation, and interaction algorithms for a dating site. A different department designs interactive user interfaces for these algorithms. We make a natural language DSL that invokes the different algorithms according to specified outcomes.
- When designing an API for a collection of algorithms.
Just designing a DSL can bring clarity of what signatures should be in the API.
NIntegrate'sMethodoption was designed and implemented using a DSL. See this video between 25:00 and 27:30.
Designing DSL
- Decide what kind of sentences the DSL is going to have.
Are natural language sentences going to be used?
Are the sentences mathematical in nature?
- Prepare, create, or accumulate sentences.
- In some cases using Morphological Analysis can greatly help for coming up with use cases and the corresponding sentences.
- Create a context free grammar that describes the sentences. (Or a large subset of them.)
- At this stage I use exclusively Extended Backus-Naur Form (EBNF).
- Program parser(s) for the grammar.
I use most of the time functional parsers.
The package FunctionalParsers.m provides a Mathematica implementation of this kind of parsing.
The package can automatically generate parsers from a grammar given in EBNF. (See the coding example below.)
- Program an interpreter for the parsed sentences.
At this stage the parsed sentences are hooked to the algorithms of the problem domain.
In Mathematica this can be done fairly easy using
Caseto pick grammar rule wrappers.
- Test the parsing and interpretation.
See the example below illustrating steps 3-6.
Introduction to using DSL's in Mathematica
This blog post of mine "Natural language processing with functional parsers".
This detailed slide-show presentation "Functional parsers for an integration requests language grammar" shows how to use the package FunctionalParsers.m over a small grammar.
Advanced example
The blog post "Simple time series conversational engine" discusses the creation (design and programming) of a simple conversational engine for time series analysis (data loading, finding outliers and trends.)
Here is movie demonstrating that conversation engine: http://youtu.be/wlZ5ANglVI4.
Other discussions
Small part from 17:30 to 21:00 of the WTC 2012 "Spatial Access Methods and Route Finding" presentation shows a DSL for points of interest queries.
The answer of the MSE question "CSS Selectors for Symbolic XML" uses FunctionalParsers.m.
Coding example
This example is for the steps 3-6 of the second section.
