Ideal workflow for developing and running "pipelines"

Question

tl;dr What are some good workflows for developing and running data processing pipelines with Mathematica?

---

I sometimes develop data processing pipelines with Mathematica. I load some data, transform it, and derive some summary results. I tend to experiment quite a bit when doing this, checking the output after each step. The notebook environment is very convenient for this.

When I'm done, I sometimes need to run this pipeline on multiple datasets. The notebook is not convenient for this, unfortunately. It's better to wrap up the sequence of operations into a function and call that functions with several pieces of data.

The problems with this approach are:

• Collecting all the steps into a function takes some work, and is error-prone.
• If I need to modify the pipeline later, it is very inconvenient to work with a single large function. It does not make it easy to look at partial results.

How do people deal with this situation? What is the workflow you found most convenient?

Answer 1

I have frequently encountered a need to do the same thing. An example is a complex notebook which loads an image and analyzes it as scientific data. I want to run the same analysis on a large number of images to obtain a result for each. But the notebook is long and complex. Trying to merge it into a single cell for a function definition makes it almost unreadable and extremely difficult to debug when something breaks the analysis method.

My solution is to leave the analysis algorithm in the original notebook and call that notebook from a controlling notebook.

Here is how I do that:

The analysis notebook leads with a definition that defines the file name of the image to be analyzed. This notebook can be evaluated as a standalone notebook in development or later in debugging.

When I am satisfied that the analysis notebook is working correctly, I save a version in which the image file name is not defined, by just commenting out the leading definition.

I then create a controlling notebook which is usually very simple. It often runs a loop (or uses Map) to analyze a large number of images and saves the result for each. To do this it opens the analysis notebook and obtains a handle to it using, for example:

nb = NotebookOpen["AnalysisNotebook.nb"]

It now relies on the fact that the controlling notebook and the newly opened analysis notebook share the same kernel and therefore the same symbol definitions. It can run a loop like this:

1. Define an image file name using the symbol name used in the analysis notebook.

2. Use NotebookEvaluate[nb] to run the analysis.

3. Save the results which have been produced as symbol definitions by the analysis notebook. Often saved in a Dataset or just appended to a list.

4. Define a new image filename and do it again until done.

I wrote this as though it was done in a loop, but all of this can just be done by a function in the controlling notebook that is mapped onto a list of image file names.

I find this works really well. If some image breaks the analysis algorithm. I just open the analysis notebook, uncomment the leading definition and revise it to point to the offending image. Then execute the notebook a cell at a time in the usual way to locate the problem.

Kind regards, David

Answer 2

Disclaimer: I have faced similar issues as well, but I have not "field-tested" the solution I present below, so I can't speak about its issues and limitations.

The idea is the following: We use AutoGeneratedPackage to convert the pipeline notebook into a package file. We then do minor post-processing on the package file, and package it into a function. Compared to trying to run the notebook itself via NotebookEvaluate or similar, this approach should be significantly more robust and performant.

There are two files involved:

• The pipeline notebook, here PipelineDemo.nb (this file has AutoGeneratedPackage->True set):

  input = FindFile@"ExampleData/turtle.jpg"
  (* "D:\\Program Files\\Wolfram \
  Research\\Mathematica\\12.3\\Documentation\\English\\System\\\
  ExampleData\\turtle.jpg" *)
  
  data = Import@input
  
  output1 = ColorNegate@data
  
  output2 = Colorize@data
  
  (* this will be the final output of the pipeline *)
  {output1, output2}
  

  

  Note how everything except the input parameters is marked as initialization cell, such that it is exported to the package file. It is of course easy to exclude tests/notes from being exported simply by not marking the relevant cells as initialization cell. The last line will be what's returned from the pipeline function.

• The file calling the pipeline, PipelineUsageDemp.nb. This could also be an actual package file to nicely wrap up the pipeline, but for simplicity, I just put the code in a notebook:

  BeginPackage["PipelineDemo`"];
  pipeline;
  Begin["`Private`"];
  
  (* new cell *)
  Join @@ Import[FileNameJoin@{NotebookDirectory[],"PipelineDemo.m"}, "HeldExpressions"] /. 
   HoldComplete[expr__] :> (
     SetDelayed @@ Hold[pipeline[input_], CompoundExpression[expr]]
     )
  
  End[];
  EndPackage[];
  
  pipeline[FindFile@"ExampleData/ocelot.jpg"]
  

  

  As you can see, we import the auto-generated PipelineDemo.m file as "HeldExpressions", and post-process it into the body of the pipeline function. The symbol input (whose corresponding line was not exported to the package file) is now a parameter of the pipeline function.

Alternative

An even simpler (but less clean) approach would be to have the same PipelineDemo.nb file, and then simply set the global variable input before Getting the file.