What is a good way to implement a versatile folder structure checking system?

Question

I'm writing an 'application' (set of *.wl and *.wls files) that requires a given folder structure to function; I am not sure how this structure will eventually look like, because it changes as I implement additional functionality.

I have a general idea about the dimensions of the folder structure-though nothing is written in stone, as I said earlier-namely it is fairly shallow-no more than 3 or 4 levels deep-and probably 2 or 3 folders wide at its widest level (I will provide an instance of the current folder structure below for reference purposes).

I have chosen to implement the folder structure as a nested Association. The top-level folder resides at root, where root holds the (path) value returned from FileNameDrop[$ScriptCommandLine[[-1]], -1]:

dirs = Association[

  "root" -> root,
  "folders" -> Association[

    "logs" -> "log",
    "files" -> Association[

      "root" -> "fls",
      "folders" -> Association[

        "auxiliary" -> "aux",
        "other" -> "msc"
       ]
     ],

    "data" -> Association[

      "root" -> "dta",
      "folders" -> Association[

        "downloaded" -> "dld",
        "transformed" -> "tsf",
        "series" -> Association[

          "root" -> "ser",
          "folders" -> Association[

            "primitives" -> "prm",
            "derived" -> "sec"
          ]
        ]
      ]
    ]
  ] 
]

I'll digress a little bit on how I intend to use this structure.

For the purposes of analysis, the 'application' needs to download, save and repeatedly process series of data.

After writing and using an alpha version for a while I noticed it was cumbersome to remember where every result should be saved, so I figured I should automate the process.

Instead of plugging in manually the relevant file path, I intend to do something like eg fpath=searchFor[dirs,"derived"] and get back the path to where all 'derived' series (ie partially transformed data) should be eventually saved, for subsequent processing.

I have already implemented a version of searchFor:

(* will use 'patt' to identify nodes *)
With[{patt = {"root", "folders"}},

  (* returns list(s) of paths to 'goal' *)
  (* inputs: a path structure and a folder to search for*)
  searchFor[structure_, goal_] := Module[{nodes, curr, path, paths = {}},

    path = {structure["root"]};
    nodes = {structure["folders"]};
    While[
      Length[nodes] > 0,

      {{curr}, nodes} = TakeDrop[nodes, 1];
      With[{fs = curr["folders"], match = MatchQ[Keys[curr], patt]},

        Which[
          match && Length[Select[fs, AssociationQ]] > 0, 
          path = Join[path, {curr["root"]}],

          match && MemberQ[Keys[fs], goal], (
            path = Join[path, {curr["root"], fs[goal]}];
            paths = Join[paths, {path}];
            path = Drop[path, -2]
          )
        ]

      ];
    nodes = Join[nodes, List @@ Select[curr, AssociationQ]]

   ];

  paths

 ]

]

For an example evaluation, consider eg searchFor[dirs, "derived"] which returns:

{{root, "dta", "ser", "sec"}}

The output can be used to retrieve the desired path "root\data\series\derived" by evaluating FileNameJoin[{root, "dta", "ser", "sec"}] or StringJoin[Riffle[{root, "dta", "ser", "sec"}, "\\"]].

In a nutshell, when I'm coding an extra feature that needs to save some data at some appropriate place (namely, the leaves of the dirs structure above), I don't want to have to look up the appropriate path but use searchFor instead.

In that way, the folder structure I have in my mind (the "root\data\series\derived") can be easily mapped into something like "root\dta\ser\sec" and even if I have to relocate something in the future (ie change the folder structure), the whole transition will be as smooth as possible.

Back to the question

I am not sure if searchFor is the right tool for the task (I have implemented a random folder structure generator I used to test searchFor, which I can provide upon request) and I was wondering if there are any suggestions.

What is a tested way to implement a versatile folder structure system for a Mathematica application without too much hassle?

Answer 1

I'd reverse the structure a little, in order to make simpler earchFor version.

The idea is to use Position, it won't handle labels located in keys so we need them somewhere as values:

dirs=<|"rootPath"-><|
  "label"->"rootLabel",
  "folders"-><|
    "log"-><|"label"->"logs"|>,
    "fls"-><|
      "label"->"files",
      "folders"-><|
        "aux"-><|"label"->"auxiliary"|>,
        "msc"-><|"label"->"other"|>
      |>
    |>,
    "dta"-><|
      "label"->"data",
      "folders"-><|
        "dla"-><|"label"->"downloaded"|>,
        "tsf"-><|"label"->"transformed"|>,
        "ser"-><|
          "label"->"series",
          "folders"-><|
             "prm" -> <|"label"->"primitives"|>,
             "sec" -> <|"label"->"derived"|>
            |>
          |>
        |>
      |>
    |>
  |>
|>

Now it easy, we just need to delete Key wrappers:

Position[dirs, KeyValuePattern["label" -> "derived"]] /. 
  {Key["folders"] -> Nothing, Key[k_] :> k} //
  Map[FileNameJoin]

{"rootPath\\dta\\ser\\sec"}