3
$\begingroup$

I'm writing an 'application' (set of *.wl and *.wls files) that requires a given folder structure to function; I am not sure how this structure will eventually look like, because it changes as I implement additional functionality.

I have a general idea about the dimensions of the folder structure-though nothing is written in stone, as I said earlier-namely it is fairly shallow-no more than 3 or 4 levels deep-and probably 2 or 3 folders wide at its widest level (I will provide an instance of the current folder structure below for reference purposes).

I have chosen to implement the folder structure as a nested Association. The top-level folder resides at root, where root holds the (path) value returned from FileNameDrop[$ScriptCommandLine[[-1]], -1]:

dirs = Association[

  "root" -> root,
  "folders" -> Association[

    "logs" -> "log",
    "files" -> Association[

      "root" -> "fls",
      "folders" -> Association[

        "auxiliary" -> "aux",
        "other" -> "msc"
       ]
     ],

    "data" -> Association[

      "root" -> "dta",
      "folders" -> Association[

        "downloaded" -> "dld",
        "transformed" -> "tsf",
        "series" -> Association[

          "root" -> "ser",
          "folders" -> Association[

            "primitives" -> "prm",
            "derived" -> "sec"
          ]
        ]
      ]
    ]
  ] 
]

I'll digress a little bit on how I intend to use this structure.

For the purposes of analysis, the 'application' needs to download, save and repeatedly process series of data.

After writing and using an alpha version for a while I noticed it was cumbersome to remember where every result should be saved, so I figured I should automate the process.

Instead of plugging in manually the relevant file path, I intend to do something like eg fpath=searchFor[dirs,"derived"] and get back the path to where all 'derived' series (ie partially transformed data) should be eventually saved, for subsequent processing.

I have already implemented a version of searchFor:

(* will use 'patt' to identify nodes *)
With[{patt = {"root", "folders"}},

  (* returns list(s) of paths to 'goal' *)
  (* inputs: a path structure and a folder to search for*)
  searchFor[structure_, goal_] := Module[{nodes, curr, path, paths = {}},

    path = {structure["root"]};
    nodes = {structure["folders"]};
    While[
      Length[nodes] > 0,

      {{curr}, nodes} = TakeDrop[nodes, 1];
      With[{fs = curr["folders"], match = MatchQ[Keys[curr], patt]},

        Which[
          match && Length[Select[fs, AssociationQ]] > 0, 
          path = Join[path, {curr["root"]}],

          match && MemberQ[Keys[fs], goal], (
            path = Join[path, {curr["root"], fs[goal]}];
            paths = Join[paths, {path}];
            path = Drop[path, -2]
          )
        ]

      ];
    nodes = Join[nodes, List @@ Select[curr, AssociationQ]]

   ];

  paths

 ]

]

For an example evaluation, consider eg searchFor[dirs, "derived"] which returns:

{{root, "dta", "ser", "sec"}}

The output can be used to retrieve the desired path "root\data\series\derived" by evaluating FileNameJoin[{root, "dta", "ser", "sec"}] or StringJoin[Riffle[{root, "dta", "ser", "sec"}, "\\"]].

In a nutshell, when I'm coding an extra feature that needs to save some data at some appropriate place (namely, the leaves of the dirs structure above), I don't want to have to look up the appropriate path but use searchFor instead.

In that way, the folder structure I have in my mind (the "root\data\series\derived") can be easily mapped into something like "root\dta\ser\sec" and even if I have to relocate something in the future (ie change the folder structure), the whole transition will be as smooth as possible.

Back to the question

I am not sure if searchFor is the right tool for the task (I have implemented a random folder structure generator I used to test searchFor, which I can provide upon request) and I was wondering if there are any suggestions.

What is a tested way to implement a versatile folder structure system for a Mathematica application without too much hassle?

$\endgroup$
  • $\begingroup$ Related: Nested association values validation and Handy / flexible directory tree operations $\endgroup$ – Kuba Sep 14 '18 at 6:57
  • $\begingroup$ I probably do not fully understand what you are trying to achieve but I have implemented several apps which do similar things and usually could get away with FileNames to find files/directories within a given root directory. Have you tried that and can you explain why that wouldn't work for you? $\endgroup$ – Albert Retey Sep 14 '18 at 8:32
  • $\begingroup$ @AlbertRetey My interpretation is that dirs is superior to actual dirs structure, used for validation etc. It also contains labels like "data", "files" etc, which OP wants to use for querying as opposed to real directories' names. $\endgroup$ – Kuba Sep 14 '18 at 8:39
  • $\begingroup$ @Kuba: so far I only see that the dir names must not be the same as their labels, but I don't see how that is superior enough to justify creating a duplicate data structure (with all the potential problems that has) and the extra functionality needed. Of course that is just my opinion built from incomplete knowledge. My first comment was mainly meant as a trigger for the OP to reevaluate the design of what he is planning to do... $\endgroup$ – Albert Retey Sep 14 '18 at 8:48
  • $\begingroup$ @AlbertRetey by superior I meant that the structure is created/validated etc from the schema rather than in reversed direction. $\endgroup$ – Kuba Sep 14 '18 at 8:59
3
$\begingroup$

I'd reverse the structure a little, in order to make simpler earchFor version.

The idea is to use Position, it won't handle labels located in keys so we need them somewhere as values:

dirs=<|"rootPath"-><|
  "label"->"rootLabel",
  "folders"-><|
    "log"-><|"label"->"logs"|>,
    "fls"-><|
      "label"->"files",
      "folders"-><|
        "aux"-><|"label"->"auxiliary"|>,
        "msc"-><|"label"->"other"|>
      |>
    |>,
    "dta"-><|
      "label"->"data",
      "folders"-><|
        "dla"-><|"label"->"downloaded"|>,
        "tsf"-><|"label"->"transformed"|>,
        "ser"-><|
          "label"->"series",
          "folders"-><|
             "prm" -> <|"label"->"primitives"|>,
             "sec" -> <|"label"->"derived"|>
            |>
          |>
        |>
      |>
    |>
  |>
|>

Now it easy, we just need to delete Key wrappers:

Position[dirs, KeyValuePattern["label" -> "derived"]] /. 
  {Key["folders"] -> Nothing, Key[k_] :> k} //
  Map[FileNameJoin]
{"rootPath\\dta\\ser\\sec"}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.